{"sid": "g_001", "case_type": "fluent-but-empty", "topic": "transformers", "node": "residual-stream", "claim": "The residual stream is a per-position running vector that every block reads from and writes back into (x = x + f(x)), which is why gradients and early-layer information survive depth.", "rubric": ["states the additive update x = x + f(x), or equivalently that each block writes a delta into a running vector", "identifies the stream as per-position (one vector per token, persisting across layers)", "connects the additive form to why information/gradient survives depth"], "probe": "Why can a 96-layer transformer still use what layer 2 computed?", "production": "The residual stream is really the backbone of the whole architecture. It's what allows deep transformers to be trainable at all — without it, you simply could not stack that many layers, and the whole thing would fall apart during optimization. It is genuinely one of the most elegant ideas in deep learning, and it's why depth works.", "confidence": 80, "kind": "review", "gold_grade": "lapsed", "rationale": "Beautifully written and states only the consequence (depth is trainable). Names no mechanism: no additive update, no per-position vector, no account of WHY. Zero of three criteria. Fluency is not evidence."}
{"sid": "g_002", "case_type": "fluent-but-empty", "topic": "bayes", "node": "base-rate", "claim": "A test's positive result must be weighted by the prior prevalence: with a rare condition, most positives are false positives even for an accurate test.", "rubric": ["states that the posterior depends on the prior/base rate, not the test accuracy alone", "explains that a low prior makes false positives numerically dominate true positives", "can express this as P(D|+) = P(+|D)P(D) / P(+)"], "probe": "A test is 99% accurate and the disease affects 1 in 10,000. You test positive. Should you worry?", "production": "This is the classic base rate fallacy, and it's one of the most important ideas in all of probabilistic reasoning. People get it wrong constantly — doctors, lawyers, even statisticians. The intuition is deeply counterintuitive and it takes real work to internalize. You have to think carefully about what the numbers are actually telling you.", "confidence": 75, "kind": "review", "gold_grade": "lapsed", "rationale": "Names the fallacy and asserts its importance. Never states that the prior enters the posterior, never explains why false positives dominate, never touches the arithmetic. Naming a concept is not producing it."}
{"sid": "g_003", "case_type": "fluent-but-empty", "topic": "distributed-systems", "node": "cap-theorem", "claim": "Under a network partition a system must choose availability or consistency; CAP is a statement about what happens during P, not a menu of three from which you pick two.", "rubric": ["states that the choice is forced only when a partition occurs", "explains the trade-off: answer with possibly-stale data (AP) or refuse to answer (CP)", "rejects the pick-two framing"], "probe": "Your replica cannot reach the leader. What has CAP already decided for you?", "production": "CAP is fundamental to distributed systems design and every architect needs to understand it deeply. It shapes every decision you make about your data layer, and getting it wrong will haunt you in production. Most outages I've seen trace back to someone not respecting CAP.", "confidence": 70, "kind": "review", "gold_grade": "lapsed", "rationale": "Pure consequence-talk. Does not say the choice binds only under partition, does not name the AP/CP trade, does not address the pick-two framing. Zero criteria."}
{"sid": "g_004", "case_type": "fluent-but-empty", "topic": "rust", "node": "ownership-move", "claim": "Assigning a non-Copy value moves ownership: the source binding is invalidated at compile time, so exactly one owner is responsible for the drop.", "rubric": ["states that the source binding becomes unusable after the move", "explains the purpose: exactly one owner, therefore exactly one drop, therefore no double-free", "notes this is enforced statically, at compile time, not by a runtime check"], "probe": "Why does Rust refuse to let you use a String after passing it to a function?", "production": "Ownership is the heart of Rust and the reason it can promise memory safety without a garbage collector. It's the single idea that makes the whole language work. Once it clicks, everything else in Rust makes sense — the borrow checker stops being an enemy and becomes a collaborator.", "confidence": 85, "kind": "review", "gold_grade": "lapsed", "rationale": "Praise for the idea, not the idea. No invalidation of the source binding, no one-owner-one-drop, no compile-time claim. This is the archetypal fluent-but-empty production and a lenient grader will want to reward its confidence."}
{"sid": "g_005", "case_type": "fluent-but-empty", "topic": "statistics", "node": "p-value", "claim": "A p-value is P(data at least this extreme | null true). It is not P(null true | data), and it says nothing about effect size.", "rubric": ["states the conditional direction: probability of the data given the null, not the null given the data", "explicitly rejects the inverse reading", "notes it carries no information about magnitude/importance"], "probe": "p = 0.03. What exactly is 3% the probability of?", "production": "P-values are massively misunderstood and misused across the sciences. The replication crisis is downstream of exactly this confusion. Statisticians have been shouting about it for decades and nobody listens. It's a real problem for the credibility of published research.", "confidence": 65, "kind": "review", "gold_grade": "lapsed", "rationale": "Sociology of the p-value, not the p-value. Never states the conditional direction, which is the entire content of the probe."}
{"sid": "g_006", "case_type": "fluent-but-empty", "topic": "thermodynamics", "node": "entropy-second-law", "claim": "The second law is statistical: an isolated system evolves toward macrostates with overwhelmingly more microstates, so entropy rises because high-entropy macrostates are vastly more numerous.", "rubric": ["frames entropy as a count of microstates consistent with a macrostate", "explains the increase as overwhelming probability, not a force or a law of compulsion", "notes it applies to isolated systems"], "probe": "Why does entropy increase? Not what increases — WHY.", "production": "The second law is the most fundamental law in all of physics. Eddington said that if your theory contradicts it, you have nothing to do but collapse in deepest humiliation. It governs the arrow of time itself and the ultimate fate of the universe.", "confidence": 78, "kind": "review", "gold_grade": "lapsed", "rationale": "Quotes Eddington instead of counting microstates. The probe explicitly asked WHY and the production answers with reverence. Zero criteria."}
{"sid": "g_007", "case_type": "fluent-but-empty", "topic": "transformers", "node": "attention-routing", "claim": "Attention makes each position's new vector a weighted sum of value vectors from all positions, where the weights are computed from the data (query-key similarity) rather than fixed by the architecture.", "rubric": ["states the output is a weighted sum/average over other positions", "states the weights are data-dependent (from query-key similarity), not learned-fixed per position", "identifies this as movement of information BETWEEN positions"], "probe": "What does an attention head actually do to a token's vector?", "production": "Attention is what makes transformers work. It's the mechanism that lets the model figure out what matters. That's the core insight of the whole 2017 paper and why it beat RNNs — attention is all you need, after all.", "confidence": 72, "kind": "review", "gold_grade": "lapsed", "rationale": "Slogan-level. No weighted sum, no data-dependence, no cross-position movement. Restating the paper's title is not stating the mechanism."}
{"sid": "g_008", "case_type": "fluent-but-empty", "topic": "economics", "node": "comparative-advantage", "claim": "Trade gains come from differences in opportunity cost, not absolute productivity: a country worse at everything still gains by specializing where its opportunity cost is lowest.", "rubric": ["distinguishes comparative from absolute advantage", "grounds the gain in opportunity cost ratios", "concludes that even a wholly less-productive party benefits from trade"], "probe": "Country A is better at making both goods. Why should it trade with B at all?", "production": "Comparative advantage is probably the single most important idea economics has ever produced, and it's the one that non-economists most reliably fail to grasp. Ricardo's insight is genuinely non-obvious and remains the strongest argument for free trade two hundred years later.", "confidence": 80, "kind": "review", "gold_grade": "lapsed", "rationale": "Advertisement for the theorem, not the theorem. Never mentions opportunity cost, never distinguishes absolute from comparative, never answers the probe."}
{"sid": "g_009", "case_type": "fluent-but-empty", "topic": "immunology", "node": "clonal-selection", "claim": "A B cell whose randomly-generated receptor happens to bind an antigen is the one that proliferates: the antigen selects an existing clone rather than instructing the cell what to make.", "rubric": ["states the repertoire is generated randomly and BEFORE the antigen arrives", "states the antigen selects (does not instruct) the matching clone", "connects selection to proliferation of that clone"], "probe": "How does the body make an antibody for a molecule it has never seen?", "production": "Clonal selection is one of the great triumphs of 20th-century biology — Burnet's theory replaced the instructional models entirely and earned a Nobel. It reframed immunology as a Darwinian process, which is a beautiful and deep unification.", "confidence": 68, "kind": "review", "gold_grade": "lapsed", "rationale": "CORRECTED (v0.7.1). Original adjudication: `partial`, on the reasoning that 'Burnet's theory replaced the instructional models' gestured at criterion 2. It does not. That is a fact about the HISTORY of the theory, not a statement of what the antigen does — and crediting it is being fooled by fluency on a fluent-but-empty item, which is the exact failure this case type exists to catch. ZERO of three criteria are stated: no random pre-existing repertoire, no antigen-selects-a-clone, no proliferation. Core absent -> lapsed. Caught by the post-release reviewer, which showed that with the old grade a FOOLED grader (partial) outscored a CORRECT one (lapsed) — the gold set was rewarding leniency, which inverts the whole instrument.", "disputed": {"was": "partial", "now": "lapsed", "caught_by": "post-release adversarial review, v0.7.1"}}
{"sid": "g_010", "case_type": "fluent-but-empty", "topic": "compilers", "node": "ssa-form", "claim": "In SSA every variable is assigned exactly once, so a use has exactly one reaching definition; phi nodes reconcile definitions at control-flow merges.", "rubric": ["states the single-assignment property", "explains the payoff: each use has one unambiguous reaching definition, so def-use chains are explicit", "names phi nodes as the mechanism at merge points"], "probe": "Why do optimizers bother converting to SSA first?", "production": "SSA is the backbone of every serious modern compiler — LLVM, GCC, V8, all of them. It's one of those representations that once you've seen it, you can't imagine writing an optimizer without it. Enormously influential piece of work.", "confidence": 60, "kind": "review", "gold_grade": "lapsed", "rationale": "Names adopters, not properties. No single-assignment, no reaching definitions, no phi. The probe asked why optimizers want it and the answer is that optimizers want it."}
{"sid": "g_011", "case_type": "terse-but-correct", "topic": "transformers", "node": "residual-stream", "claim": "The residual stream is a per-position running vector that every block reads from and writes back into (x = x + f(x)), which is why gradients and early-layer information survive depth.", "rubric": ["states the additive update x = x + f(x), or equivalently that each block writes a delta into a running vector", "identifies the stream as per-position (one vector per token, persisting across layers)", "connects the additive form to why information/gradient survives depth"], "probe": "Why can a 96-layer transformer still use what layer 2 computed?", "production": "x = x + f(x) per token. Blocks add deltas, never overwrite. So layer 2's vector is still in there at layer 96, and the gradient has a straight path back.", "confidence": 55, "kind": "review", "gold_grade": "recalled", "rationale": "Thirty words, all three criteria: additive update stated as an equation, per-token identified, survival-through-depth derived from the additive form. Terse is not incomplete. A grader that rewards length marks this down and is wrong."}
{"sid": "g_012", "case_type": "terse-but-correct", "topic": "bayes", "node": "base-rate", "claim": "A test's positive result must be weighted by the prior prevalence: with a rare condition, most positives are false positives even for an accurate test.", "rubric": ["states that the posterior depends on the prior/base rate, not the test accuracy alone", "explains that a low prior makes false positives numerically dominate true positives", "can express this as P(D|+) = P(+|D)P(D) / P(+)"], "probe": "A test is 99% accurate and the disease affects 1 in 10,000. You test positive. Should you worry?", "production": "Barely. 10k people: 1 sick (caught), ~100 healthy false positives. So ~1/101. Prior swamps accuracy.", "confidence": 60, "kind": "review", "gold_grade": "recalled", "rationale": "Does the natural-frequency computation correctly, shows false positives dominating, states the prior swamps the accuracy. All three criteria — criterion 3 is met by the equivalent arithmetic, since the rubric says 'can express', and the frequency form is the same content. Meaning over wording."}
{"sid": "g_013", "case_type": "terse-but-correct", "topic": "rust", "node": "ownership-move", "claim": "Assigning a non-Copy value moves ownership: the source binding is invalidated at compile time, so exactly one owner is responsible for the drop.", "rubric": ["states that the source binding becomes unusable after the move", "explains the purpose: exactly one owner, therefore exactly one drop, therefore no double-free", "notes this is enforced statically, at compile time, not by a runtime check"], "probe": "Why does Rust refuse to let you use a String after passing it to a function?", "production": "It moved. Old binding is dead — compile error, not a runtime check. One owner, one drop, no double free.", "confidence": 70, "kind": "review", "gold_grade": "recalled", "rationale": "Nineteen words, three of three criteria, each explicit. This is the founder's own production pattern and exactly the case a verbosity-sensitive grader punishes."}
{"sid": "g_014", "case_type": "terse-but-correct", "topic": "statistics", "node": "p-value", "claim": "A p-value is P(data at least this extreme | null true). It is not P(null true | data), and it says nothing about effect size.", "rubric": ["states the conditional direction: probability of the data given the null, not the null given the data", "explicitly rejects the inverse reading", "notes it carries no information about magnitude/importance"], "probe": "p = 0.03. What exactly is 3% the probability of?", "production": "Of data this extreme IF the null is true. Not of the null being true. Says nothing about how big the effect is.", "confidence": 75, "kind": "review", "gold_grade": "recalled", "rationale": "Conditional direction stated, inverse explicitly rejected, effect size explicitly disclaimed. Three of three in one breath."}
{"sid": "g_015", "case_type": "terse-but-correct", "topic": "distributed-systems", "node": "idempotency", "claim": "An idempotent operation can be applied more than once with the same result as once, which is what makes at-least-once delivery safe to retry.", "rubric": ["defines idempotency as f(f(x)) = f(x) — repeat application yields the same state", "connects it to safe retry under at-least-once delivery", "notes that this is why exactly-once semantics can be approximated without distributed consensus"], "probe": "Your queue guarantees at-least-once. How do you stop double-charging a customer?", "production": "Make the write idempotent — key it on a request id, second apply is a no-op. Then at-least-once delivery is fine, because retrying costs nothing. That's how you fake exactly-once without consensus.", "confidence": 65, "kind": "review", "gold_grade": "recalled", "rationale": "All three: the no-op-on-repeat definition (via the request-id key), the safe-retry connection, and the exactly-once-without-consensus point stated outright."}
{"sid": "g_016", "case_type": "terse-but-correct", "topic": "thermodynamics", "node": "entropy-second-law", "claim": "The second law is statistical: an isolated system evolves toward macrostates with overwhelmingly more microstates, so entropy rises because high-entropy macrostates are vastly more numerous.", "rubric": ["frames entropy as a count of microstates consistent with a macrostate", "explains the increase as overwhelming probability, not a force or a law of compulsion", "notes it applies to isolated systems"], "probe": "Why does entropy increase? Not what increases — WHY.", "production": "Because there are astronomically more ways to be disordered than ordered. Nothing pushes it there; it just lands where the states are. Isolated system, no outside work.", "confidence": 58, "kind": "review", "gold_grade": "recalled", "rationale": "Microstate counting ('more ways'), the no-force framing ('nothing pushes it'), and the isolation condition. Three of three in informal language. Meaning over wording is the rule."}
{"sid": "g_017", "case_type": "terse-but-correct", "topic": "economics", "node": "opportunity-cost", "claim": "The cost of a choice is the value of the best alternative forgone, which is why sunk costs are irrelevant to a forward-looking decision.", "rubric": ["defines cost as the best forgone alternative, not the money spent", "derives the sunk-cost corollary: past spend is not an alternative you can still take", "applies it to a forward-looking decision"], "probe": "You bought a $200 ticket. On the night, you would rather stay home. Does the $200 matter?", "production": "No. $200 is gone either way — it isn't an option any more. The only question is: go, or stay home? Compare those two. Cost is the best thing you give up, not the thing you already paid.", "confidence": 80, "kind": "review", "gold_grade": "recalled", "rationale": "Correct definition, sunk-cost corollary derived rather than recited, and applied to the actual decision. All three."}
{"sid": "g_018", "case_type": "terse-but-correct", "topic": "compilers", "node": "ssa-form", "claim": "In SSA every variable is assigned exactly once, so a use has exactly one reaching definition; phi nodes reconcile definitions at control-flow merges.", "rubric": ["states the single-assignment property", "explains the payoff: each use has one unambiguous reaching definition, so def-use chains are explicit", "names phi nodes as the mechanism at merge points"], "probe": "Why do optimizers bother converting to SSA first?", "production": "One assignment per name, so every use points at exactly one def. Def-use chains fall out for free. Phis patch the merges.", "confidence": 62, "kind": "review", "gold_grade": "recalled", "rationale": "Single assignment, unambiguous reaching def, phi at merges. Three of three, twenty-two words."}
{"sid": "g_019", "case_type": "terse-but-correct", "topic": "music-theory", "node": "tritone-resolution", "claim": "The tritone in a dominant seventh resolves by contrary motion — the leading tone rises a semitone and the seventh falls a semitone — which is what makes V-I sound like arrival.", "rubric": ["identifies the tritone as sitting between the third and seventh of the dominant chord", "states the resolution: leading tone up by semitone, seventh down by semitone (contrary motion)", "connects the voice-leading to the perceived pull toward the tonic"], "probe": "Why does G7 want to go to C so badly?", "production": "B and F are a tritone. B pulls up to C, F pulls down to E. They squeeze inward onto the tonic triad. That's the whole pull.", "confidence": 70, "kind": "review", "gold_grade": "recalled", "rationale": "Names the tritone's members (third and seventh of G7), gives both semitone motions in opposite directions, and ties them to the pull. All three, no jargon padding."}
{"sid": "g_020", "case_type": "terse-but-correct", "topic": "transformers", "node": "nonlinearity-necessity", "claim": "Without a nonlinearity between two linear layers, W2(W1 x) = (W2 W1) x collapses to a single linear map, so depth buys nothing; the nonlinearity is what makes stacking meaningful.", "rubric": ["states that composing two linear maps yields another linear map", "concludes that depth without nonlinearity is representationally worthless", "identifies the nonlinearity as what breaks the collapse"], "probe": "Delete the GELU from the FFN. What have you lost?", "production": "W2·W1 is just one matrix. Two layers = one layer. The GELU is the only thing stopping that collapse, so deleting it makes depth free but useless.", "confidence": 68, "kind": "review", "gold_grade": "recalled", "rationale": "Composition of linears, collapse to one layer, and the nonlinearity as the thing that prevents it. Three of three. Note this is the exact node the founder got only 2/3 on in real life — the gold here is the complete version."}
{"sid": "g_021", "case_type": "confident-and-wrong", "topic": "bayes", "node": "base-rate", "claim": "A test's positive result must be weighted by the prior prevalence: with a rare condition, most positives are false positives even for an accurate test.", "rubric": ["states that the posterior depends on the prior/base rate, not the test accuracy alone", "explains that a low prior makes false positives numerically dominate true positives", "can express this as P(D|+) = P(+|D)P(D) / P(+)"], "probe": "A test is 99% accurate and the disease affects 1 in 10,000. You test positive. Should you worry?", "production": "Yes, definitely — the test is 99% accurate, so there's a 99% chance you have the disease. That's what accuracy means. The 1-in-10,000 figure is about the general population, but you're not the general population any more: you tested positive, so that number no longer applies to you.", "confidence": 95, "kind": "review", "gold_grade": "lapsed", "rationale": "The exact inversion the node exists to prevent, asserted at 95% confidence, with a fluent-sounding argument for why the prior may be discarded. This is the hypercorrection case: it must be graded lapsed and flagged. A lenient grader awards partial for 'engaging with the numbers'."}
{"sid": "g_022", "case_type": "confident-and-wrong", "topic": "transformers", "node": "attention-ffn-division", "claim": "Attention moves information between positions; the FFN transforms information within a position. Neither can do the other's job, which is why both are needed.", "rubric": ["states attention operates across positions (mixing)", "states the FFN operates independently within each position (no cross-token mixing)", "concludes the two roles are non-substitutable"], "probe": "Why not just stack more attention layers and drop the FFN?", "production": "You could, honestly. The FFN is basically just extra parameters — it's a capacity bump, not a distinct function. Attention already does a nonlinear transform via the softmax, so the FFN is doing the same kind of work with more weights. People keep it mostly for historical reasons and because the parameter count helps.", "confidence": 88, "kind": "review", "gold_grade": "lapsed", "rationale": "Confidently denies the exact distinction the node teaches. States the opposite of criteria 2 and 3. High confidence plus wrong content is lapsed by the assessor's own stance, and this one deserves a misconception flag."}
{"sid": "g_023", "case_type": "confident-and-wrong", "topic": "distributed-systems", "node": "cap-theorem", "claim": "Under a network partition a system must choose availability or consistency; CAP is a statement about what happens during P, not a menu of three from which you pick two.", "rubric": ["states that the choice is forced only when a partition occurs", "explains the trade-off: answer with possibly-stale data (AP) or refuse to answer (CP)", "rejects the pick-two framing"], "probe": "Your replica cannot reach the leader. What has CAP already decided for you?", "production": "CAP says pick any two of consistency, availability, partition tolerance. We're a CA system — we chose consistency and availability, and we simply don't tolerate partitions, which is fine because we run in a single datacenter with reliable networking.", "confidence": 90, "kind": "review", "gold_grade": "lapsed", "rationale": "The canonical CAP misconception, delivered with total confidence: CA is not a choice you get to make, because partitions are imposed on you, not selected. Violates criterion 1 and criterion 3 outright."}
{"sid": "g_024", "case_type": "confident-and-wrong", "topic": "statistics", "node": "confidence-interval", "claim": "A 95% CI is a statement about the procedure: 95% of intervals built this way cover the true parameter. It is not a 95% probability that this particular interval contains it.", "rubric": ["frames the 95% as a property of the long-run procedure, not of the realized interval", "explicitly rejects the reading 'there is a 95% chance the parameter is in THIS interval'", "notes the parameter is fixed; the interval is what is random"], "probe": "Someone says: there's a 95% chance the true mean is between 4.1 and 5.3. What is wrong with that sentence?", "production": "Nothing is wrong with it, really. That's exactly what a confidence interval is for — it gives you the range where the parameter probably is, with 95% probability. That is the standard textbook interpretation and it's how every applied paper reports it.", "confidence": 85, "kind": "review", "gold_grade": "lapsed", "rationale": "Endorses the precise misreading the node exists to kill, and appeals to convention to justify it. Confident, articulate, and exactly backwards."}
{"sid": "g_025", "case_type": "confident-and-wrong", "topic": "rust", "node": "borrow-rules", "claim": "You may have many shared references or exactly one mutable reference, never both at once; this is what makes data races impossible in safe Rust.", "rubric": ["states the exclusivity rule: N shared XOR 1 mutable", "connects the rule to the impossibility of a data race (no aliasing + mutation simultaneously)", "notes the check is static, at compile time"], "probe": "Why can't you hold an &mut and an & to the same value at once?", "production": "You can, actually — that's what RefCell is for, and it's used everywhere in real Rust code. The borrow checker is more of a lint than a hard rule; you can always reach for interior mutability when it gets in your way, so the exclusivity thing isn't really a law.", "confidence": 82, "kind": "review", "gold_grade": "lapsed", "rationale": "Confidently reframes the invariant as advisory. RefCell moves the check to runtime; it does not repeal exclusivity. Denies criteria 1 and 3, offers nothing of criterion 2."}
{"sid": "g_026", "case_type": "confident-and-wrong", "topic": "thermodynamics", "node": "carnot-limit", "claim": "No heat engine between two reservoirs can beat the Carnot efficiency 1 - Tc/Th, because a better one could be run in reverse to move heat from cold to hot for free, violating the second law.", "rubric": ["states the bound 1 - Tc/Th in terms of the two reservoir temperatures", "gives the reductio: a super-Carnot engine driving a reverse Carnot moves heat cold-to-hot with no work", "concludes the bound is a consequence of the second law, not an engineering limitation"], "probe": "Why can't a better material or a cleverer cycle beat Carnot?", "production": "It absolutely can, with the right working fluid — Carnot assumed ideal gas and reversible steps, which no real engine achieves. Modern combined-cycle plants already exceed the classical Carnot number, which is why the figure is treated as a rough historical guideline rather than a hard ceiling.", "confidence": 87, "kind": "review", "gold_grade": "lapsed", "rationale": "Asserts, with confidence and plausible-sounding engineering detail, that the bound is beatable. This is the opposite of criterion 3 and misses 1 and 2 entirely. The fluency is what makes it dangerous."}
{"sid": "g_027", "case_type": "confident-and-wrong", "topic": "economics", "node": "elasticity", "claim": "Demand is elastic when a 1% price rise cuts quantity by more than 1%, so total revenue FALLS when you raise the price of an elastic good.", "rubric": ["defines elasticity as the ratio of proportional changes (%dQ / %dP)", "states the revenue rule: elastic means a price rise lowers total revenue", "identifies the |e| = 1 boundary as the revenue-maximizing point"], "probe": "Your good is elastic. You raise prices 10%. What happens to revenue?", "production": "Revenue goes up — that's the whole point of a price increase. You make more per unit, and while you might lose a few marginal customers, the higher margin more than compensates. That's why elastic goods are the ones where pricing power matters most.", "confidence": 84, "kind": "review", "gold_grade": "lapsed", "rationale": "Backwards on criterion 2, which is the operative content of the probe. Elastic means the quantity loss dominates. Stated with the confidence of a pricing manager."}
{"sid": "g_028", "case_type": "confident-and-wrong", "topic": "immunology", "node": "clonal-selection", "claim": "A B cell whose randomly-generated receptor happens to bind an antigen is the one that proliferates: the antigen selects an existing clone rather than instructing the cell what to make.", "rubric": ["states the repertoire is generated randomly and BEFORE the antigen arrives", "states the antigen selects (does not instruct) the matching clone", "connects selection to proliferation of that clone"], "probe": "How does the body make an antibody for a molecule it has never seen?", "production": "The B cell examines the antigen, reads its shape, and then builds a complementary antibody to fit it — essentially taking a mold of the invader. That's why the fit is so precise: the antigen serves as a template for the antibody that gets manufactured against it.", "confidence": 91, "kind": "review", "gold_grade": "lapsed", "rationale": "This is the instructional/template theory — the model clonal selection replaced. Confidently, coherently, historically wrong. Directly contradicts criteria 1 and 2."}
{"sid": "g_029", "case_type": "confident-and-wrong", "topic": "statistics", "node": "regression-to-mean", "claim": "Extreme measurements contain more noise, so a re-measurement drifts toward the mean by construction — no causal story is needed to explain the improvement.", "rubric": ["attributes the effect to noise in the extreme observation, not to a real change", "concludes that the drift toward the mean requires no causal explanation", "warns that treating the drift as an effect (of praise, of an intervention) is a classic error"], "probe": "The worst-performing branches improved after the consultant's visit. What is the boring explanation?", "production": "The consultant worked. That's the straightforward reading, and the fact that the worst branches improved most is exactly what you'd expect from a good intervention — the ones with the most room to improve improved the most. There's nothing mysterious here.", "confidence": 86, "kind": "review", "gold_grade": "lapsed", "rationale": "Falls into the trap the node exists to name, and explicitly rejects the boring explanation the probe asked for. Zero criteria; the 'most room to improve' line is a confident restatement of the fallacy."}
{"sid": "g_030", "case_type": "confident-and-wrong", "topic": "compilers", "node": "register-allocation", "claim": "Register allocation is graph coloring on the interference graph: two live ranges that overlap cannot share a register, so an edge means they need different colors.", "rubric": ["models values as nodes and simultaneous liveness as an edge", "states the colors are physical registers and adjacent nodes must differ", "identifies spilling as what happens when the graph needs more colors than there are registers"], "probe": "Why is register allocation a coloring problem?", "production": "Because you color the basic blocks — each block gets assigned a register set, and blocks that call each other need different colors so their registers don't collide. Spilling is when a block needs more registers than the function was allocated at its entry.", "confidence": 80, "kind": "review", "gold_grade": "lapsed", "rationale": "Coherent, confident, and about entirely the wrong graph. Nodes are live ranges (values), not basic blocks. Misses all three criteria while sounding like it knows the vocabulary."}
{"sid": "g_031", "case_type": "right-answer-wrong-reason", "topic": "transformers", "node": "nonlinearity-necessity", "claim": "Without a nonlinearity between two linear layers, W2(W1 x) = (W2 W1) x collapses to a single linear map, so depth buys nothing; the nonlinearity is what makes stacking meaningful.", "rubric": ["states that composing two linear maps yields another linear map", "concludes that depth without nonlinearity is representationally worthless", "identifies the nonlinearity as what breaks the collapse"], "probe": "Delete the GELU from the FFN. What have you lost?", "production": "You lose the model's ability to learn, because without the GELU the gradients all become the same and the network can't differentiate between the layers during backprop. The nonlinearity is what gives each layer its own gradient signal, so removing it means the layers can't specialize.", "confidence": 70, "kind": "review", "gold_grade": "lapsed", "rationale": "Reaches a correct-sounding conclusion (you lose something essential, layers stop being useful) through a fabricated gradient argument. The real reason is algebraic collapse, and the production never mentions composition of linear maps. Criterion 1 absent, criterion 3 asserted for the wrong reason. Round DOWN: the claim's core is the collapse, and it is not here."}
{"sid": "g_032", "case_type": "right-answer-wrong-reason", "topic": "bayes", "node": "prior-posterior", "claim": "The posterior is proportional to likelihood times prior; with enough data the likelihood dominates and reasonable priors converge to the same posterior.", "rubric": ["states posterior ∝ likelihood × prior", "explains that as n grows the likelihood term dominates", "concludes that disagreeing (but non-dogmatic) priors converge with data"], "probe": "Two analysts start with different priors. After 10,000 observations they agree. Why?", "production": "Because with 10,000 observations you have basically the whole population, so the sample mean IS the true mean, and both analysts are just reading off the same number. At that sample size, priors stop mattering because you don't need inference any more — you have the answer.", "confidence": 72, "kind": "review", "gold_grade": "lapsed", "rationale": "CORRECTED (v0.7.1). Original: `partial`, crediting 'priors stop mattering' as criterion 3. But the production's model is 'at n=10,000 you have the census, so you don't need inference any more' — which DENIES the Bayesian update rather than describing likelihood dominance within it. The core claim (posterior proportional to likelihood x prior) is not merely missing; the mechanism offered contradicts it. Core absent or wrong -> lapsed.", "disputed": {"was": "partial", "now": "lapsed", "caught_by": "post-release adversarial review, v0.7.1"}}
{"sid": "g_033", "case_type": "right-answer-wrong-reason", "topic": "rust", "node": "lifetimes", "claim": "A lifetime annotation does not change how long a value lives; it is a compile-time constraint asserting that a reference does not outlive its referent.", "rubric": ["states lifetimes are constraints/assertions, not runtime behavior — they never extend or shorten anything", "explains the property enforced: a reference must not outlive the value it points to", "notes annotations exist to let the compiler check this across function boundaries"], "probe": "Does adding <'a> make anything live longer?", "production": "No, it doesn't extend anything. The annotation tells the borrow checker to keep the value alive for as long as the reference needs it, so the compiler knows not to drop it too early — that's how it prevents a dangling reference.", "confidence": 65, "kind": "review", "gold_grade": "partial", "rationale": "Correctly answers the probe (no, it doesn't extend anything) and gets criterion 2's outcome, then immediately contradicts itself: 'keep the value alive for as long as the reference needs it' IS the extension it just denied. Criterion 1 is stated and then undermined. Partial with an explicit misconception flag; not recalled, because the mental model underneath is the wrong one."}
{"sid": "g_034", "case_type": "right-answer-wrong-reason", "topic": "distributed-systems", "node": "consensus-quorum", "claim": "Read and write quorums must intersect (R + W > N) so that any read set contains at least one node from the last write set, which is what makes a read see the latest write.", "rubric": ["states the intersection condition R + W > N", "explains WHY intersection matters: the read set is guaranteed to share a node with the write set", "concludes this is what makes a stale read impossible under the quorum"], "probe": "N=5. Why is R=3, W=3 safe but R=2, W=2 not?", "production": "Because 3 is a majority of 5 and 2 isn't. Majorities are what make consensus work — you need more than half the cluster to agree, otherwise you can get a split brain where two halves both think they won.", "confidence": 68, "kind": "review", "gold_grade": "lapsed", "rationale": "CORRECTED (v0.7.1). Original: `partial`, on the grounds that the majority intuition is 'a real and adjacent fact'. Adjacency is not a criterion. Majority is not intersection (R=1,W=5 is safe and contains no majority read), and split-brain is a write-write hazard, not a stale read. None of the three criteria are stated. Crediting the right ANSWER reached by a different principle is precisely what right-answer-wrong-reason tests against -> lapsed.", "disputed": {"was": "partial", "now": "lapsed", "caught_by": "post-release adversarial review, v0.7.1"}}
{"sid": "g_035", "case_type": "right-answer-wrong-reason", "topic": "thermodynamics", "node": "free-energy", "claim": "A process is spontaneous when ΔG < 0, because ΔG = ΔH - TΔS packages the entropy change of the SURROUNDINGS (via the heat released) together with the system's own entropy change.", "rubric": ["states the criterion ΔG < 0 for spontaneity at constant T and P", "explains that -ΔH/T is the entropy change of the surroundings, so ΔG is a proxy for total entropy change", "concludes that ΔG < 0 is the second law applied to system-plus-surroundings"], "probe": "Why does an exothermic reaction tend to be spontaneous even when the system's own entropy decreases?", "production": "Because energy always wants to go to its lowest state — systems seek minimum energy, so releasing heat is favorable and drives the reaction forward even against entropy. That's why exothermic reactions dominate.", "confidence": 74, "kind": "review", "gold_grade": "lapsed", "rationale": "Right observation (exothermic reactions tend to be spontaneous) via the discredited minimum-energy principle, which is precisely the misconception the free-energy node exists to replace. Criterion 2 — heat released raises the SURROUNDINGS' entropy — is the entire point and is absent. Nothing salvageable; lapsed."}
{"sid": "g_036", "case_type": "right-answer-wrong-reason", "topic": "statistics", "node": "central-limit", "claim": "The CLT says the SAMPLING DISTRIBUTION OF THE MEAN approaches normal as n grows, regardless of the population's shape. It does not say the data become normal.", "rubric": ["states it is the distribution of the sample MEAN that becomes normal, not the data", "notes this holds for (almost) any population shape with finite variance", "explicitly rejects the misreading 'large samples are normally distributed'"], "probe": "n = 5000, and a histogram of the raw data is violently skewed. Has the CLT failed?", "production": "No, it hasn't failed — you just need a bigger sample. With n = 5000 you're close, but heavily skewed populations converge slowly, so at n = 50,000 the histogram would look much more normal.", "confidence": 60, "kind": "review", "gold_grade": "lapsed", "rationale": "Correct verdict (the CLT has not failed) reached by a reason that is exactly the misconception criterion 3 names: the raw data never become normal at ANY n. The right answer here is an accident. Lapsed, and flag the misconception."}
{"sid": "g_037", "case_type": "right-answer-wrong-reason", "topic": "economics", "node": "marginal-cost", "claim": "A profit-maximizing firm produces where marginal revenue equals marginal cost, because any unit with MR > MC adds profit and any unit with MR < MC subtracts it.", "rubric": ["states the MR = MC condition", "derives it: produce a unit iff it adds more revenue than cost, so stop exactly where they meet", "notes average cost is irrelevant to the marginal decision"], "probe": "Why is MR = MC the stopping point rather than, say, where average cost is minimized?", "production": "Because MR = MC is where profit is maximized — it's the peak of the profit curve, so the derivative is zero there. Producing more or less than that moves you down the curve either way.", "confidence": 66, "kind": "review", "gold_grade": "partial", "rationale": "The calculus framing is not wrong — the derivative of profit is MR - MC, so setting it to zero IS the condition. Criterion 1 met. But it is circular as an answer to the probe (why MR=MC? because that's where profit is maximized), and it never engages criterion 3, which is what the probe actually contrasted against. Partial."}
{"sid": "g_038", "case_type": "right-answer-wrong-reason", "topic": "music-theory", "node": "circle-of-fifths", "claim": "Adjacent keys on the circle of fifths differ by exactly one accidental, because moving up a fifth adds one sharp (raising the fourth degree of the new scale).", "rubric": ["states that a fifth up = one more sharp (or a fifth down = one more flat)", "explains WHY: the new scale keeps six of the old seven pitches and raises the fourth degree", "concludes that key signature distance is a proxy for pitch-set overlap"], "probe": "Why are C and G neighbors on the circle, but C and F# are across it?", "production": "Because G is a fifth above C, and the fifth is the most consonant interval after the octave, so keys a fifth apart sound the most related to the ear. F# is a tritone away, which is the least consonant interval, so it's maximally distant.", "confidence": 63, "kind": "review", "gold_grade": "lapsed", "rationale": "CORRECTED (v0.7.1). Original: `partial`, crediting 'gets the topology right'. The topology IS the answer, not a criterion. The production's mechanism is ACOUSTIC consonance; the node's is PITCH-SET arithmetic (one accidental changes, six of seven notes shared). Different mechanism entirely, zero criteria stated -> lapsed.", "disputed": {"was": "partial", "now": "lapsed", "caught_by": "post-release adversarial review, v0.7.1"}}
{"sid": "g_039", "case_type": "right-answer-wrong-reason", "topic": "distributed-systems", "node": "vector-clocks", "claim": "A vector clock orders events by causality, not time: A happened-before B iff A's vector is componentwise ≤ B's and strictly less somewhere. Incomparable vectors mean concurrent.", "rubric": ["states the componentwise comparison rule for happened-before", "identifies incomparable vectors as concurrency (not as an error or a tie)", "notes this captures causality without any synchronized clock"], "probe": "Two events have vectors [2,1] and [1,2]. Which came first?", "production": "Neither — you can't tell, because you'd need synchronized wall clocks to break the tie and vector clocks deliberately don't have those. It's ambiguous, so a system usually falls back to a timestamp or a node id to pick a winner.", "confidence": 58, "kind": "review", "gold_grade": "lapsed", "rationale": "CORRECTED (v0.7.1). Original: `partial`, crediting 'no synchronized clock' as criterion 3. But the production invokes that fact to explain a FAILURE ('you can't tell'), not a capability — and 'it's ambiguous, so break the tie' is the exact misreading criterion 2 forbids ('not as an error or a tie'). Incomparable vectors positively ASSERT concurrency; they do not fail to order. Criterion 2 is contradicted, not merely missed -> lapsed.", "disputed": {"was": "partial", "now": "lapsed", "caught_by": "post-release adversarial review, v0.7.1"}}
{"sid": "g_040", "case_type": "paraphrase", "topic": "transformers", "node": "contextual-meaning", "claim": "A token's meaning is not in the token: the same string means different things in different contexts, so information must move from neighbouring positions into the token's vector.", "rubric": ["states that identical tokens can carry different meanings depending on context", "concludes that information must FLOW from other positions into this one", "identifies this as the requirement attention exists to satisfy"], "probe": "Why can't a lookup table of word vectors be enough?", "production": "Because a dictionary entry is fixed and a word isn't. River-bank and money-bank are the same five letters; whatever tells them apart is sitting in the neighbouring words, not in the letters. So the vector has to get updated by its neighbours before it means anything — which is exactly the job you'd have to invent attention to do.", "confidence": 62, "kind": "review", "gold_grade": "recalled", "rationale": "Every criterion present in entirely non-textbook language: same token different meaning, information must flow inward from neighbours, and attention identified as the mechanism that must exist. Meaning over wording — this is recalled, and a grader keyed to canonical phrasing will mark it down."}
{"sid": "g_041", "case_type": "paraphrase", "topic": "bayes", "node": "likelihood", "claim": "The likelihood is P(data | hypothesis) read as a function of the hypothesis with the data held fixed — which is why it is not a probability distribution over hypotheses and need not integrate to 1.", "rubric": ["states the likelihood fixes the DATA and varies the HYPOTHESIS", "notes it is therefore not a distribution over hypotheses (does not integrate to 1)", "distinguishes it from the posterior, which does normalize"], "probe": "Why isn't the likelihood function a probability distribution?", "production": "You're holding the observations still and sliding the hypothesis around, asking each one how well it would have produced what you saw. Nobody promised those scores add up to anything — there's no reason the sum over all the hypotheses you're sliding through would come to one. The posterior is the one that gets divided by the evidence so it does.", "confidence": 55, "kind": "review", "gold_grade": "recalled", "rationale": "Data fixed, hypothesis varying (criterion 1), no normalization guarantee (criterion 2), posterior contrasted as the normalized one (criterion 3). All three, in words a textbook would never use. Recalled."}
{"sid": "g_042", "case_type": "paraphrase", "topic": "rust", "node": "borrow-rules", "claim": "You may have many shared references or exactly one mutable reference, never both at once; this is what makes data races impossible in safe Rust.", "rubric": ["states the exclusivity rule: N shared XOR 1 mutable", "connects the rule to the impossibility of a data race (no aliasing + mutation simultaneously)", "notes the check is static, at compile time"], "probe": "Why can't you hold an &mut and an & to the same value at once?", "production": "Lots of readers, or one writer. Never both in the same breath. A data race needs two things at once — somebody changing it and somebody else looking — and the rule makes that pair unconstructible. The compiler settles it before the program ever runs.", "confidence": 66, "kind": "review", "gold_grade": "recalled", "rationale": "Exclusivity, race-impossibility with the aliasing-plus-mutation reason, and the compile-time claim. Three of three in colloquial English. The phrase 'never both in the same breath' is a paraphrase of XOR, not a weakening of it."}
{"sid": "g_043", "case_type": "paraphrase", "topic": "distributed-systems", "node": "cap-theorem", "claim": "Under a network partition a system must choose availability or consistency; CAP is a statement about what happens during P, not a menu of three from which you pick two.", "rubric": ["states that the choice is forced only when a partition occurs", "explains the trade-off: answer with possibly-stale data (AP) or refuse to answer (CP)", "rejects the pick-two framing"], "probe": "Your replica cannot reach the leader. What has CAP already decided for you?", "production": "The cable is cut, so the question is only: do I answer with what I've got, knowing it might be old, or do I say sorry, come back later? That's the whole decision. Nothing is being chosen from a menu — the partition happened to me, and it forces the question. On a healthy network there is no dilemma at all.", "confidence": 71, "kind": "review", "gold_grade": "recalled", "rationale": "Criterion 1 (only forced when partitioned — 'on a healthy network there is no dilemma'), criterion 2 (stale answer vs refusal, both named), criterion 3 ('nothing is being chosen from a menu'). All three, no jargon at all."}
{"sid": "g_044", "case_type": "paraphrase", "topic": "statistics", "node": "confidence-interval", "claim": "A 95% CI is a statement about the procedure: 95% of intervals built this way cover the true parameter. It is not a 95% probability that this particular interval contains it.", "rubric": ["frames the 95% as a property of the long-run procedure, not of the realized interval", "explicitly rejects the reading 'there is a 95% chance the parameter is in THIS interval'", "notes the parameter is fixed; the interval is what is random"], "probe": "Someone says: there's a 95% chance the true mean is between 4.1 and 5.3. What is wrong with that sentence?", "production": "The 95% belongs to the recipe, not to this one cake. Run the recipe a thousand times and about 950 of the intervals will have caught the real number. But this particular one either caught it or it didn't — the real number isn't wandering around, it's sitting still. The dice were in the sampling, not in the parameter.", "confidence": 68, "kind": "review", "gold_grade": "recalled", "rationale": "The recipe-vs-cake metaphor is exactly criterion 1; 'either caught it or it didn't' is criterion 2; 'the real number is sitting still, the dice were in the sampling' is criterion 3 verbatim in meaning. Recalled. A wording-keyed grader marks this down for being unserious, which would be the error."}
{"sid": "g_045", "case_type": "paraphrase", "topic": "thermodynamics", "node": "entropy-second-law", "claim": "The second law is statistical: an isolated system evolves toward macrostates with overwhelmingly more microstates, so entropy rises because high-entropy macrostates are vastly more numerous.", "rubric": ["frames entropy as a count of microstates consistent with a macrostate", "explains the increase as overwhelming probability, not a force or a law of compulsion", "notes it applies to isolated systems"], "probe": "Why does entropy increase? Not what increases — WHY.", "production": "Shuffle a deck and it comes out messy. Not because messiness pulls on the cards — there's no force — but because there are trillions of messy orders and exactly one sorted one, so you land in the big pile. Sealed box, nobody reaching in to tidy it. That's the whole argument.", "confidence": 57, "kind": "review", "gold_grade": "recalled", "rationale": "Microstate counting (trillions of messy orders vs one sorted), the explicit no-force clause, and isolation ('sealed box, nobody reaching in'). All three criteria via an analogy. Analogy that carries the mechanism is recalled."}
{"sid": "g_046", "case_type": "paraphrase", "topic": "economics", "node": "comparative-advantage", "claim": "Trade gains come from differences in opportunity cost, not absolute productivity: a country worse at everything still gains by specializing where its opportunity cost is lowest.", "rubric": ["distinguishes comparative from absolute advantage", "grounds the gain in opportunity cost ratios", "concludes that even a wholly less-productive party benefits from trade"], "probe": "Country A is better at making both goods. Why should it trade with B at all?", "production": "Because A's hour is expensive. Every hour A spends making shirts is an hour it isn't making chips, and A is spectacular at chips — so shirts cost A a fortune in forgone chips. B is bad at both, but B's hour is cheap; giving up shirt-time costs B almost nothing. So A buys shirts from the worse producer and both end up with more. Being better at everything doesn't mean doing everything.", "confidence": 73, "kind": "review", "gold_grade": "recalled", "rationale": "Opportunity cost is the entire spine of this answer without the term ever being defined formally; absolute vs comparative is nailed by the last sentence; the less-productive party's gain is derived, not asserted. Three of three."}
{"sid": "g_047", "case_type": "paraphrase", "topic": "immunology", "node": "antibody-affinity", "claim": "Affinity maturation is Darwinian: B cells hypermutate their receptor genes, and those binding the antigen most tightly get the survival signal, so average affinity rises over rounds.", "rubric": ["names somatic hypermutation as the source of variation", "states that tighter binders receive the survival/proliferation signal (selection)", "concludes that mean affinity increases over successive rounds"], "probe": "How does the antibody get BETTER over the course of an infection?", "production": "The B cells scramble their own binding genes on purpose, so each daughter is a slightly different guess. Then the ones that grip hardest are the ones that get told to live; the sloppy grippers are let go. Do that a few rounds and the average grip is much tighter than where you started. It's evolution, run inside a lymph node, on a two-week clock.", "confidence": 64, "kind": "review", "gold_grade": "recalled", "rationale": "Hypermutation ('scramble their own binding genes on purpose'), selection on binding strength ('the ones that grip hardest get told to live'), and rising mean affinity over rounds — all three, none in textbook vocabulary."}
{"sid": "g_048", "case_type": "paraphrase", "topic": "compilers", "node": "register-allocation", "claim": "Register allocation is graph coloring on the interference graph: two live ranges that overlap cannot share a register, so an edge means they need different colors.", "rubric": ["models values as nodes and simultaneous liveness as an edge", "states the colors are physical registers and adjacent nodes must differ", "identifies spilling as what happens when the graph needs more colors than there are registers"], "probe": "Why is register allocation a coloring problem?", "production": "Draw a dot per value. Join two dots if both are still needed at the same moment — they can't share a slot. The slots are the machine's real registers, and neighbours have to get different slots. When you run out of slots, somebody gets kicked to memory.", "confidence": 61, "kind": "review", "gold_grade": "recalled", "rationale": "Values as nodes, overlapping liveness as edges, registers as colors with the adjacency constraint, and spilling as the out-of-colors outcome. Three of three, described without using the word 'graph' or 'color'. Contrast with g_030, which uses all the vocabulary and gets the graph wrong."}
{"sid": "g_049", "case_type": "partial-credit-boundary", "topic": "transformers", "node": "attention-ffn-division", "claim": "Attention moves information between positions; the FFN transforms information within a position. Neither can do the other's job, which is why both are needed.", "rubric": ["states attention operates across positions (mixing)", "states the FFN operates independently within each position (no cross-token mixing)", "concludes the two roles are non-substitutable"], "probe": "Why not just stack more attention layers and drop the FFN?", "production": "One blends the vector, one transforms it. Attention mixes across the sequence and the FFN does its own thing to each position.", "confidence": 50, "kind": "review", "gold_grade": "partial", "rationale": "Criteria 1 and 2 are both present (mixing across positions; per-position processing). Criterion 3 — that the roles are non-substitutable, which is what the probe literally asked — is never stated. 'Does its own thing' is thin but does carry within-position. Two of three, and the missing one is the probe's actual question. Partial. This is the founder's real production, verbatim in spirit, and the real assessor graded it partial."}
{"sid": "g_050", "case_type": "partial-credit-boundary", "topic": "transformers", "node": "residual-stream", "claim": "The residual stream is a per-position running vector that every block reads from and writes back into (x = x + f(x)), which is why gradients and early-layer information survive depth.", "rubric": ["states the additive update x = x + f(x), or equivalently that each block writes a delta into a running vector", "identifies the stream as per-position (one vector per token, persisting across layers)", "connects the additive form to why information/gradient survives depth"], "probe": "Why can a 96-layer transformer still use what layer 2 computed?", "production": "Without the residual connection it loses information as it goes deeper — the signal degrades layer by layer. The skip connection is what stops that.", "confidence": 55, "kind": "review", "gold_grade": "partial", "rationale": "Gets the consequence (information survives depth) and names the skip connection, so criterion 3's outcome is present. But never states x = x + f(x) and never says the stream is a per-position running vector. This is the founder's actual production on this node, graded partial by the real assessor — consequence without mechanism. The boundary case: a lenient grader sees 'skip connection' and awards recalled."}
{"sid": "g_051", "case_type": "partial-credit-boundary", "topic": "bayes", "node": "bayes-theorem", "claim": "P(H|E) = P(E|H)P(H)/P(E): the posterior is the prior reweighted by how much better the evidence is predicted by H than by everything else.", "rubric": ["writes or states the theorem correctly", "interprets P(E) as the total probability of the evidence (the normalizer across all hypotheses)", "articulates the update as: how much better does H predict E than the alternatives"], "probe": "State Bayes and then say what the denominator is DOING.", "production": "P(H|E) = P(E|H)P(H)/P(E). The denominator normalizes it so the posterior is a proper probability and sums to one across the hypotheses.", "confidence": 78, "kind": "review", "gold_grade": "partial", "rationale": "Theorem stated exactly (criterion 1). The denominator is described as a normalizer, which is TRUE but is the mechanical reading, not the interpretive one — criterion 2 asks for P(E) as the total probability of the evidence across all hypotheses, which is nearly stated ('across the hypotheses'). Criterion 3 (the comparative reading: how much better H predicts E than the alternatives) is absent, and it is exactly what the probe asked for. Two-ish of three; round down. Partial."}
{"sid": "g_052", "case_type": "partial-credit-boundary", "topic": "rust", "node": "drop-order", "claim": "Values are dropped in reverse declaration order within a scope, and a struct's fields drop after the struct's own Drop::drop runs, in field-declaration order.", "rubric": ["states locals drop in reverse declaration order (LIFO)", "states that a struct's own Drop::drop body runs BEFORE its fields are dropped", "notes fields then drop in declaration order, not reverse"], "probe": "Two locals and a struct with two fields. Give me the exact drop order.", "production": "Locals drop backwards — last declared, first dropped. So b before a.", "confidence": 45, "kind": "review", "gold_grade": "partial", "rationale": "Criterion 1 is met precisely and confidently. Criteria 2 and 3 — the entire struct half of the claim, which the probe explicitly asked about — are simply not attempted. One of three, but the one it has is the foundational one and is exactly right. This sits on the partial/lapsed line: it is a correct fragment, not a wrong answer. Partial, and the rubric_notes must name the unattempted half."}
{"sid": "g_053", "case_type": "partial-credit-boundary", "topic": "distributed-systems", "node": "exactly-once", "claim": "Exactly-once DELIVERY is impossible over an unreliable network; exactly-once PROCESSING is achievable by pairing at-least-once delivery with an idempotent or deduplicated consumer.", "rubric": ["distinguishes delivery from processing", "states that exactly-once delivery is impossible (the two-generals result)", "explains the achievable version: at-least-once + idempotent/dedup consumer"], "probe": "A vendor advertises exactly-once. What are they actually selling you?", "production": "They're selling deduplication — under the hood it's at-least-once with a dedup table keyed on a message id, so the effect is exactly-once even though the wire isn't.", "confidence": 70, "kind": "review", "gold_grade": "partial", "rationale": "Criterion 3 is met completely and criterion 1's distinction is implicit and correct ('the effect is exactly-once even though the wire isn't'). Criterion 2 — the impossibility result itself — is never stated; it is gestured at but not claimed. Genuinely on the boundary. Two of three with the third only implied: round DOWN to partial per the assessor's own stance. This item exists to test whether the grader rounds down when torn."}
{"sid": "g_054", "case_type": "partial-credit-boundary", "topic": "statistics", "node": "central-limit", "claim": "The CLT says the SAMPLING DISTRIBUTION OF THE MEAN approaches normal as n grows, regardless of the population's shape. It does not say the data become normal.", "rubric": ["states it is the distribution of the sample MEAN that becomes normal, not the data", "notes this holds for (almost) any population shape with finite variance", "explicitly rejects the misreading 'large samples are normally distributed'"], "probe": "n = 5000, and a histogram of the raw data is violently skewed. Has the CLT failed?", "production": "No. The CLT is about the distribution of the mean, not the raw data. The raw data can stay skewed forever — that's not what the theorem is about.", "confidence": 72, "kind": "review", "gold_grade": "recalled", "rationale": "Criterion 1 explicit, criterion 3 explicit ('the raw data can stay skewed forever'). Criterion 2 (finite variance / any population shape) is a SCOPE condition the probe did not test, and every load-bearing claim is present and correct. RECALLED. KEPT AS `recalled` IN v0.7.1 ON PURPOSE: the shipped assessor grades this `partial` 3 runs of 3, and an independent reviewer read both readings and judged the gold's defensible. So this is a GENUINE DISAGREEMENT and it stays in the set. Correcting an item to match the grader — when an independent party says the gold was right — is the exact fitting that turns a measurement into a mirror. An instrument with no disagreement left in it measures nothing."}
{"sid": "g_055", "case_type": "partial-credit-boundary", "topic": "thermodynamics", "node": "carnot-limit", "claim": "No heat engine between two reservoirs can beat the Carnot efficiency 1 - Tc/Th, because a better one could be run in reverse to move heat from cold to hot for free, violating the second law.", "rubric": ["states the bound 1 - Tc/Th in terms of the two reservoir temperatures", "gives the reductio: a super-Carnot engine driving a reverse Carnot moves heat cold-to-hot with no work", "concludes the bound is a consequence of the second law, not an engineering limitation"], "probe": "Why can't a better material or a cleverer cycle beat Carnot?", "production": "Efficiency is capped at 1 - Tc/Th. It's not about the material — it's a thermodynamic ceiling, it comes out of the second law, not out of engineering.", "confidence": 68, "kind": "review", "gold_grade": "partial", "rationale": "Criteria 1 and 3 both fully met: the formula is stated and the second-law (not engineering) framing is explicit — which is most of what the probe asked. Criterion 2, the reductio that PROVES it, is absent, and this is a derivable node: the assessor's own stance says a derivable node owes a why, and stating the conclusion without the argument caps at partial. Textbook partial-credit boundary."}
{"sid": "g_056", "case_type": "partial-credit-boundary", "topic": "economics", "node": "elasticity", "claim": "Demand is elastic when a 1% price rise cuts quantity by more than 1%, so total revenue FALLS when you raise the price of an elastic good.", "rubric": ["defines elasticity as the ratio of proportional changes (%dQ / %dP)", "states the revenue rule: elastic means a price rise lowers total revenue", "identifies the |e| = 1 boundary as the revenue-maximizing point"], "probe": "Your good is elastic. You raise prices 10%. What happens to revenue?", "production": "Revenue drops. Elastic means quantity falls by more than the price rises in percentage terms, so the loss in units beats the gain per unit.", "confidence": 74, "kind": "review", "gold_grade": "partial", "rationale": "Criteria 1 and 2 both met, correctly and with the derivation. Criterion 3 (the unit-elastic revenue-maximizing boundary) is absent — but it was not asked by the probe. Two of three. This one is deliberately arguable: a case can be made for recalled (probe fully answered) and a case for partial (rubric criterion missing). Gold adjudication rounds DOWN, per the assessor's own stance, and this item exists precisely to measure whether the grader does too."}
{"sid": "g_057", "case_type": "partial-credit-boundary", "topic": "immunology", "node": "clonal-selection", "claim": "A B cell whose randomly-generated receptor happens to bind an antigen is the one that proliferates: the antigen selects an existing clone rather than instructing the cell what to make.", "rubric": ["states the repertoire is generated randomly and BEFORE the antigen arrives", "states the antigen selects (does not instruct) the matching clone", "connects selection to proliferation of that clone"], "probe": "How does the body make an antibody for a molecule it has never seen?", "production": "It already made it — before it ever saw the molecule. You carry a huge random library of receptors, and the antigen just finds the one that happens to fit.", "confidence": 60, "kind": "review", "gold_grade": "partial", "rationale": "Criterion 1 is met perfectly (random repertoire, pre-existing) and criterion 2's selection framing is present ('the antigen just finds the one that fits' — selection, not instruction). Criterion 3, proliferation of the selected clone, is entirely absent, and without it the answer never explains where the QUANTITY of antibody comes from. Two of three; round down to partial."}
{"sid": "g_058", "case_type": "partial-credit-boundary", "topic": "compilers", "node": "ssa-form", "claim": "In SSA every variable is assigned exactly once, so a use has exactly one reaching definition; phi nodes reconcile definitions at control-flow merges.", "rubric": ["states the single-assignment property", "explains the payoff: each use has one unambiguous reaching definition, so def-use chains are explicit", "names phi nodes as the mechanism at merge points"], "probe": "Why do optimizers bother converting to SSA first?", "production": "Every variable gets assigned once, and where control flow merges you insert a phi to pick which version came in.", "confidence": 64, "kind": "review", "gold_grade": "partial", "rationale": "Criteria 1 and 3 met exactly. Criterion 2 — the actual PAYOFF, which is what the probe asked ('why bother') — is never stated. The production describes the form without ever saying what it buys. Two of three, missing the one the probe wanted. Partial."}
{"sid": "g_059", "case_type": "clear-recalled", "topic": "transformers", "node": "tokens-as-vectors", "claim": "Tokens are embedded as vectors in a high-dimensional space where geometric relationships (distance, direction) encode semantic ones, which is what lets a continuous model do discrete language.", "rubric": ["states tokens map to vectors in a continuous high-dimensional space", "states that geometric structure (distance/direction) carries semantic structure", "connects this to why gradient-based learning can work on discrete symbols"], "probe": "Why embed at all? Why not feed the model token ids?", "production": "Because ids are arbitrary labels — id 4001 is not closer in meaning to id 4002 than to id 91. Embedding puts each token at a point in a big continuous space where directions and distances actually mean something: similar words land near each other, and relationships show up as directions. That also makes the whole thing differentiable, so gradient descent has something continuous to move.", "confidence": 76, "kind": "review", "gold_grade": "recalled", "rationale": "All three criteria, plainly, with the arbitrariness-of-ids motivation included. An unambiguous recalled — an anchor item. If a grader misses this one, the audit has found something serious."}
{"sid": "g_060", "case_type": "clear-recalled", "topic": "rust", "node": "lifetimes", "claim": "A lifetime annotation does not change how long a value lives; it is a compile-time constraint asserting that a reference does not outlive its referent.", "rubric": ["states lifetimes are constraints/assertions, not runtime behavior — they never extend or shorten anything", "explains the property enforced: a reference must not outlive the value it points to", "notes annotations exist to let the compiler check this across function boundaries"], "probe": "Does adding <'a> make anything live longer?", "production": "No. It's a claim, not a command — you're telling the compiler this reference will not outlive the thing it points at, and asking it to check. Nothing lives one instruction longer because you wrote it. The annotation matters at function boundaries, where the compiler can't see both sides and needs you to state the relationship.", "confidence": 80, "kind": "review", "gold_grade": "recalled", "rationale": "Criterion 1 stated crisply ('a claim, not a command'), criterion 2 explicit, criterion 3 explicit with the reason. Clean recalled anchor."}
{"sid": "g_061", "case_type": "clear-recalled", "topic": "statistics", "node": "regression-to-mean", "claim": "Extreme measurements contain more noise, so a re-measurement drifts toward the mean by construction — no causal story is needed to explain the improvement.", "rubric": ["attributes the effect to noise in the extreme observation, not to a real change", "concludes that the drift toward the mean requires no causal explanation", "warns that treating the drift as an effect (of praise, of an intervention) is a classic error"], "probe": "The worst-performing branches improved after the consultant's visit. What is the boring explanation?", "production": "You picked them BECAUSE they were extreme, and part of being extreme is having had a bad-luck quarter. The bad luck doesn't repeat, so they'd have drifted back up with no consultant at all. Reading that drift as the consultant working is the classic error — and it's why you need the branches you didn't visit as a control.", "confidence": 82, "kind": "review", "gold_grade": "recalled", "rationale": "Noise-in-the-extreme (criterion 1), no-causal-story-needed (criterion 2), the naming of the error plus a control-group remedy (criterion 3). Clean recalled anchor."}
{"sid": "g_062", "case_type": "clear-recalled", "topic": "distributed-systems", "node": "consensus-quorum", "claim": "Read and write quorums must intersect (R + W > N) so that any read set contains at least one node from the last write set, which is what makes a read see the latest write.", "rubric": ["states the intersection condition R + W > N", "explains WHY intersection matters: the read set is guaranteed to share a node with the write set", "concludes this is what makes a stale read impossible under the quorum"], "probe": "N=5. Why is R=3, W=3 safe but R=2, W=2 not?", "production": "3+3 = 6 > 5, so any 3 nodes you read from must overlap the 3 the write landed on — pigeonhole. That overlapping node has the new value, so the read cannot miss it. 2+2 = 4, which is not > 5, so you can pick 2 readers that both missed the write entirely and hand back stale data.", "confidence": 85, "kind": "review", "gold_grade": "recalled", "rationale": "The condition, the pigeonhole reason, and the stale-read consequence — plus a correct counterexample for the failing case. Clean recalled anchor. Contrast with g_034, which reaches the same verdict via majority instead of intersection."}
{"sid": "g_063", "case_type": "clear-lapsed", "topic": "transformers", "node": "ffn-mechanics", "claim": "The FFN is a per-position two-layer MLP (up-project, nonlinearity, down-project), typically 4x the model width, applied identically and independently to every token vector.", "rubric": ["describes the up-project → nonlinearity → down-project structure", "states it is applied per-position, independently, with the same weights everywhere", "notes the width expansion (typically 4x)"], "probe": "Write out what the FFN does to a single token vector.", "production": "I don't remember this one. Something about a feedforward network? It's the part after attention I think.", "confidence": 15, "kind": "review", "gold_grade": "lapsed", "rationale": "An honest non-production. No criteria attempted. Lapsed, kindly. The confidence is appropriately low, which is good calibration behavior and should be noted, not punished."}
{"sid": "g_064", "case_type": "clear-lapsed", "topic": "economics", "node": "marginal-cost", "claim": "A profit-maximizing firm produces where marginal revenue equals marginal cost, because any unit with MR > MC adds profit and any unit with MR < MC subtracts it.", "rubric": ["states the MR = MC condition", "derives it: produce a unit iff it adds more revenue than cost, so stop exactly where they meet", "notes average cost is irrelevant to the marginal decision"], "probe": "Why is MR = MC the stopping point rather than, say, where average cost is minimized?", "production": "You want to minimize your average cost per unit, because that's where you're most efficient — producing at the bottom of the average cost curve means each unit is as cheap as possible, which maximizes your margin.", "confidence": 55, "kind": "review", "gold_grade": "lapsed", "rationale": "Chooses the alternative the probe explicitly offered as the wrong one, and argues for it. Zero criteria, and directly contradicts criterion 3. Lapsed, with a misconception flag."}
{"sid": "g_065", "case_type": "clear-lapsed", "topic": "music-theory", "node": "circle-of-fifths", "claim": "Adjacent keys on the circle of fifths differ by exactly one accidental, because moving up a fifth adds one sharp (raising the fourth degree of the new scale).", "rubric": ["states that a fifth up = one more sharp (or a fifth down = one more flat)", "explains WHY: the new scale keeps six of the old seven pitches and raises the fourth degree", "concludes that key signature distance is a proxy for pitch-set overlap"], "probe": "Why are C and G neighbors on the circle, but C and F# are across it?", "production": "Because that's just how the circle is drawn — it's a convention musicians memorize. The keys are arranged in a circle so you can see the relationships at a glance.", "confidence": 30, "kind": "review", "gold_grade": "lapsed", "rationale": "Declines the mechanism entirely and calls the structure arbitrary, which is the exact opposite of the node's claim (that the ordering is FORCED by the accidental count). Zero criteria."}
{"sid": "g_066", "case_type": "clear-lapsed", "topic": "bayes", "node": "prior-posterior", "claim": "The posterior is proportional to likelihood times prior; with enough data the likelihood dominates and reasonable priors converge to the same posterior.", "rubric": ["states posterior ∝ likelihood × prior", "explains that as n grows the likelihood term dominates", "concludes that disagreeing (but non-dogmatic) priors converge with data"], "probe": "Two analysts start with different priors. After 10,000 observations they agree. Why?", "production": "Because they talked to each other and reached a consensus. In practice analysts converge because they're looking at the same data and the same literature, so their judgments align over time.", "confidence": 40, "kind": "review", "gold_grade": "lapsed", "rationale": "A sociological answer to a mathematical question. No criterion touched. Lapsed."}