---
name: designing-assessments
description: Read this before generating any quiz, test, task or item — assessment written item-first drifts to recall and produces a score nobody can act on. Use it when asked to build a quiz, test, exam paper, question set, retrieval practice or knowledge check, to write or fix items and distractors, or to judge whether an existing assessment measures what it claims. It produces a blueprint, items whose wrong answers are diagnostic, and a guide to what each result means.
---

# Designing assessments that elicit evidence of learning

Wiliam's second key strategy. An assessment is a machine for producing an
inference about what learners know. Design it backwards from the inference.

## The first question, always

**What decision will this inform, and what would make you decide differently?**

If the answer is "it tells me who's doing well", stop — a ranking supports almost
no instructional decision. Push for a real one:

| Decision | Shape |
|---|---|
| Move on or reteach, now | One hinge question → `hinge-questions` |
| Which of several misconceptions is in the room | Diagnostic MCQ, distractors mapped to misconceptions |
| Are the prerequisites there before I start | 5–8 items, a week ahead |
| Who needs which intervention | Item-level diagnostic, clean subscales |
| Has it stuck since last term | Spaced retrieval, no new content |
| A grade for reporting | Summative — say so plainly, design for reliability |

Never blur the last row into the others. The same test can be used either way, but
the design trade-offs differ: formative wants diagnostic richness per item,
summative wants reliability across the whole. Optimise for both, get neither.

## What done looks like

1. **Blueprint** — outcomes × cognitive demand × item count. Written before any item.
2. **The items.**
3. **Mark scheme**, with acceptable alternatives spelled out.
4. **Interpretation guide** — per item: distractor → misconception → action. This
   is what separates a formative instrument from a score generator.
5. **Timing**, and where it sits in the sequence.
6. **What this assessment cannot tell you** — one honest paragraph, always included.
   It's what stops the scores being over-read.

## Constraints

- **Blueprint first.** Items written first drift to recall, because recall items
  are the easy ones to write. The grid exposes it immediately.
- **Every distractor has a named misconception behind it.** Write the misconception
  in plain language, work the problem as a learner holding it, and the answer they
  get is the distractor. If you can't name it, you haven't designed the item.
- **No item is answerable correctly without the target knowledge**, or incorrectly
  because of something irrelevant to it.
- **The interpretation guide exists.** No guide, not shipped.
- **Item count is defensible for the claims made.** Six items cannot support an
  individual placement decision. Say so before the scores exist.

## Validity is a property of inferences, not of tests

Correct this whenever someone asks "is this a valid test?". State the inference,
then attack it on three fronts:

- **Construct under-representation** — does it cover the domain, or the
  easily-testable corner? A "scientific enquiry" test made of recall items doesn't.
- **Construct-irrelevant variance** — does anything *else* move the scores? Reading
  demand in a maths problem, cultural assumptions in a comprehension text, time
  pressure on a test that isn't about fluency. These are design faults, not learner
  deficits. Run `references/fairness-check.md` over every item.
- **Reliability** — same score tomorrow, or with a different marker, or on a
  parallel form? Short assessments are unreliable by construction.

## Choosing the response format

Per outcome, not per assessment; mixed-format is normal. Need to know *which* wrong
idea → diagnostic MCQ. Need to see reasoning → short constructed response. Need
procedural fluency → several similar items. Need transfer → novel context, same
structure. Need a performance → task plus rubric, and accept the reliability cost.

`references/item-types.md` has the full comparison, including **two-tier items**
(answer + reason), which are the only cheap way to catch right-answer-wrong-reason
— the failure mode conventional marking hides.

## Your specific failure modes

- **Distractor padding.** Three plausible-sounding wrong options with no diagnostic
  value. Name the misconception or replace the option.
- **All-recall drift.** Recall is cheapest for you to generate. Check the grid.
- **Answer leakage.** Longest option is the key; the key is grammatically
  consistent with the stem and the others aren't; "all of the above" as filler;
  key clustered on B and C. Check the distribution.
- **Context creep.** Elaborate scenarios that raise reading load without adding
  construct relevance.
- **Fabricated stimulus.** Never invent data, sources, quotations or historical
  detail and present them as real. Mark invented stimulus as invented.
- **Curriculum overconfidence.** Specifications differ by jurisdiction, board and
  year. Ask which; if you can't, state the assumption at the top.

Say plainly that AI-generated items contributing to a grade need checking against
the specification first.

## References

- `references/item-writing.md` — stems, options, distractor construction, item
  types that reveal reasoning, retrieval-practice design.
- `references/item-types.md` — response format selection; two-tier items.
- `references/fairness-check.md` — construct-irrelevant variance, item by item.