A machine marked 480 answer sheets. 156 of those grades do not match the teacher key, and the pipeline is perfectly happy about all of them — the scan was clean, the scorer ran, a number came out. This is the layer that decides which grades are supported well enough to reach a student, and it can name the criterion, the reason, and the words on the page for every single one it holds.
The scan comes back at 0.94 and looks clean, so the batch goes out. But the mark for the first criterion rested on four words — less dense than water — and the reader was 0.44 sure of one of them. It guessed clense. The phrase stopped matching, the student silently lost two marks they had earned, and nothing anywhere was unhappy. It reaches a report card, and the only person who could have caught it is the one person who got skipped.
A genuinely messy page — 0.73 overall — pulled for review even though every phrase the rubric actually cares about was read at 0.94 or better. The teacher opens it, agrees with the machine, closes it. That is not caution. That is a page-level number being asked a question about a span.
light is quicker than sound earns full marks from a keyword matcher and partial marks from a teacher. maybe it is less dense than water earns full marks too. A fluent, correct, cleanly-read answer written in words the rubric does not list earns zero. In all three the reader did its job perfectly.
| released | wrong grades released | released accuracy | held | of those, held for nothing | |
|---|---|---|---|---|---|
| page confidence ≥ 0.85 | 378 | 128 | 66.14% | 102 | 74 |
| rubric-grading-gate | 326 | 6 | 98.16% | 154 | 4 |
The threshold catches 28 of the 156 bad grades. The gate catches 150. The two queues are not nested: they overlap on 28 sheets, the gate adds 126 and drops 74 — and of the 126 it adds, 122 are a grade that was actually wrong.
score.py never returns a bare score. Per criterion it returns which phrasing matched, which token indices it matched on, whether the rubric flags that phrasing as arguable, and whether a hedge or a negation sits on it. Everything else is built on those spans.
When a criterion fails to match, text.find_near_span looks for a run of the answer that differs from an accepted phrasing in exactly one token. If that token is also the one the reader was least sure of, the zero is more likely the reader's than the student's. This is the check that is easy to leave out — you audit the marks you gave, not the marks something silently deleted.
| flag | held | wrong grades released if dropped |
|---|---|---|
hedged_evidence — a hedge or negation in the 4 tokens before the matched claim | 66 | 6 → 64 |
off_rubric — a substantive answer matched no accepted phrasing | 44 | 6 → 37 |
unread_evidence — a zero whose one breaking word was guessed | 39 | 6 → 27 |
rubric_borderline — marks from a phrasing the rubric calls disputable | 26 | 6 → 27 |
low_evidence_confidence — an awarded mark rests on badly-read words | strict | — |
unread_span — some word on the page was never read at all | strict | — |
Turning both strict flags on adds 49 sheets to the teacher's queue and catches zero grades the four defaults miss — the same six escape either way. Queue precision falls from 97.4% to 73.9%, and released accuracy goes down (98.16% → 97.83%), because every extra hold is a correct grade.
The flag in the strict set is the one anyone would build first. "Was the evidence for this mark read well?" sounds like the whole problem — and on this batch it finds nothing on its own, because a phrase that was read badly but still matched has usually still produced the right mark. The value is almost entirely on the other side of the ledger: on the marks that were not awarded. Same signal, opposite direction, and only one of the two directions is worth a teacher's evening.
Keeping all six would have made the queue look thorough. The ablation is pinned by test_every_default_flag_pays_for_itself and test_the_two_excluded_flags_catch_nothing_the_other_four_miss.
baseline PAGE_FLOOR | 0.75 | 0.80 | 0.85 | 0.90 | 0.95 |
|---|---|---|---|---|---|
| wrong grades released | 153 | 147 | 128 | 33 | 1 |
| sheets held | 28 | 45 | 102 | 365 | 477 |
gate EVIDENCE_FLOOR | 0.55 | 0.65 | 0.75 | 0.85 | 0.95 |
|---|---|---|---|---|---|
| wrong grades released | 16 | 9 | 6 | 4 | 0 |
| sheets held | 144 | 151 | 154 | 156 | 160 |
The baseline has no good setting: between 0.85 and 0.90 it goes from releasing 128 wrong grades to holding three quarters of the class. The gate moves 16 sheets across its entire sweep. Both tables are printed by make sweep on the same data — and 0.75 is not the optimum, it is just the default.
It does not correct the transcription, annotate the image, or decide the disputed grades — it names them and hands them to the person whose job that is. Handwriting recognition is not in here and is not claimed: the input is text with per-token confidences.
The scorer is a phrase matcher deliberately. It stands in for whatever actually marks the paper. The claim is about the layer above it, and that layer needs only one thing from the marker: evidence spans alongside the score. A marker that returns a bare number cannot be gated this way at all — which is the main integration constraint, and worth knowing before you pick one.
make demo # the page-confidence threshold vs this gate, side by side make gate # the classroom batch (exits non-zero = held for review) make gate-clean # the no-false-alarms case: a clean batch must exit 0 make gate-strict # the same batch with the two flags the ablation rejected make sweep # regenerate the threshold-sensitivity tables above make test # 25 tests
No dependencies beyond the standard library — no model, no keys, no network. Fixtures are generated deterministically from seed 20260806; CI regenerates them and fails on any diff, and runs the gate both ways: a clean batch must pass, the classroom batch must be blocked. A gate that waves a bad batch through fails its own build.
· Synthetic data: 480 generated responses over 12 questions, one rubric, one reader. A demonstrator of the method. Not a benchmark, and not a report on a deployment. · ⚠️ The fixture injects damage in the same vocabulary the flags are written in — a hedge goes in, `hedged_evidence` comes out. So "150 of 156 caught" is partly a property of the generator and should not be quoted as a detection rate. What the fixture can honestly settle is narrower: that a page-confidence threshold does not see these failures, that two of the six flags cost more than they return, and where the design is blind. Those three survive the objection. · 6 wrong grades are still released, and they are all the same thing: the reader misread a word and rated it above the flag's floor. Raising the floor to 0.95 catches all six here — but only because the generator caps misread confidence at 0.94, so that is a fact about the fixture, not a safe setting. · `off_rubric` requires the WHOLE answer to have matched nothing. A student who phrases one criterion in their own words and the rest in the rubric's loses those marks silently and the sheet is released. This is the largest known hole in the design; the fixture cannot produce the case, so it is pinned by a test instead. · `hedged_evidence` looks 4 tokens back and never forward. Two mangled words in the same phrase leave no near miss, so nothing looks at their confidence at all. · `rubric_borderline` only knows a phrasing is arguable because the rubric says so. Writing that list is teacher work. The gate makes the disputes visible; it does not discover them.
Each of the four admitted failure modes above has a test that fails if it quietly gets worse. An admitted limitation that nothing holds in place is just a sentence.