---
title: "The Exam Is a Hypothesis"
date: "2026-10-12T20:00:00+08:00"
categories:
- talks
talk_url: "https://talks.s-anand.net/2026-10-12-tds-ideation/"
description: I ran a TDS ideation session on designing exams that teach skills even when students use AI agents and collaborate. I share six question checks and how to measure learning transfer.
tags: [assessment-design, ai-in-education, ai-agents, learning-design]
---

I conducted an ideation session for my [TDS](https://tds.s-anand.net/) course. It was an open session along with students, TAs and instructors. It roughly explained how we conduct TDS - and in particular, how we design questions that teach something even when students can (and are encouraged to) work with AI agents as well as each other.

[Here's the link to the session](https://talks.s-anand.net/2026-10-12-tds-ideation/)

Here's my summary of the session:

1. **Pick what to teach**. My [AI Advice](https://www.s-anand.net/blog/ai-advice/) page lists AI-era skills. Yours may vary.
2. **Evaluate questions against six tests**:
   1. **Learning test**. Is the step the student must do themselves the skill we want - and not just pointless difficulty?
   2. **Forward test**. Does one student's submission score for another?
   3. **Solver test**. Would one shared tool score for everyone?
   4. **Agent test**. Could Codex solve the question with the student's logged-in browser / terminal?
   5. **Faking test**. Is faking costlier than doing it honestly, and does faking still teach?
   6. **Oracle test**. Is grading server‑side, robust (behavior, not text-matching), and rate‑limited or revealed after the test?
3. **Measure learning transfer**. How do we know the actually learnt the skill?

(Interestingly, just passing these principles to Claude / ChatGPT gives me pretty good AI and collaboration proof questions on many topics.)

And a couple of meta principles:

- **Agents improve fast**. What we teach, how to teach, _and_ how we evaluate, needs to evolve rapidly.
- **Test and fix fast**. Being able to experiment (e.g. test questions) make it easy to try new ideas. Agents speed this up.
- **Design in the open**. Maybe students (and others) learn from the process - and shape it with ideas.

[![Comic](https://talks.s-anand.net/2026-10-12-tds-ideation/comic-page.avif)](https://talks.s-anand.net/2026-10-12-tds-ideation/)

**Links**:

- [Video](https://media.s-anand.net/2026-10-12-tds-course-ideation.webm) (69 min)
- [Transcript](https://talks.s-anand.net/2026-10-12-tds-ideation/transcript.md)
- [Audio](https://github.com/sanand0/talks/releases/download/talks/2026-10-12-tds-ideation.opus) (69 min)