screenshot-to-sheet
Screenshots become spreadsheet rows only after per-field confidence gates, type checks, and a totals reconciliation.
Public · synthetic demo · stdlib onlyA missing row is visible. A wrong row is not.
OCR misreads things. That is the premise, not the failure. When a row goes missing, someone notices; when a digit is misread, the number lands in the sheet, gets summed, and nobody looks at it again. So the question worth asking of an OCR pipeline is not how accurate the engine is — it is how many of the engine's mistakes were stopped before they reached the clean data, and how many got through anyway.
Twelve fictional order-confirmation screenshots
Drawn by the repo itself and committed as PNGs, so a fresh clone runs with no dependencies. Every field is checked against a ground-truth file that the pipeline is structurally forbidden to read.
| Capture condition | Count | What it does to the OCR |
|---|---|---|
| Clean, high contrast | 4 | Occasional single-character slips |
| Low contrast (grey on grey) | 3 | Confidence drops; more slips |
| Downscaled capture | 3 | Text edges lost; most confidence-gate failures |
| Cropped before scrolling | 2 | A label is genuinely off-frame — no value exists |
Fields per screenshot: order_id, order_date, customer, item_count, subtotal, tax, total. No real company, customer, or order appears anywhere in the repository.
Both halves of the truth, from the committed run
Screenshots read: 12 Rows on the clean sheet: 3 Rows held for review: 9 Review entries written: 25 Wrong values in total: 8 caught, rule named it: 6 caught, row held anyway: 1 reached the clean sheet: 1 <- silent errors Fields with no value at all: 8 (cropped screenshots)
Exit code is 1 whenever anything was held, because a run that needs a person is not a run that succeeded.
Six per-field rules, and one that earns its keep
| Rule | Catches |
|---|---|
required | the label is missing entirely — cropped off, renamed, or misread |
conf_gate | the engine itself was unsure (confidence below 0.75) |
type_parse | 84.5O is not an amount; 2O26-07-09 is not a date |
pattern | ORD-l017 is not an order id |
range | an item count of 400, an amount of 0 |
reconcile | a confidently misread digit that passes all five above |
subtotal 245.01 + tax 18.60 = 263.61 but total reads 264.61
19.60 was misread as 18.60. Two decimal places. Positive. Inside every plausible range. Returned with high confidence. Every per-field rule passes it. Only the cross-field identity contradicts it.
Values are located by label anchor, not by position index: find the printed label, take the tokens sharing its line to the right. A label that cannot be found returns nothing — and nothing is a real answer. On a cropped capture the label genuinely is off-frame, and grabbing a nearby number would be worse than saying so.
The client drops screenshots in a folder and reads two tabs
The repo ships a Google Apps Script implementation of the same six rules and the same reconciliation, calling Cloud Vision DOCUMENT_TEXT_DETECTION. It authenticates with the script's own OAuth token rather than an API key, so the Cloud project, the quota and the bill all stay in the client's account — and no key is ever pasted into the code.
One row per screenshot that passed every check, with a link back to the source image.
Which field, what the OCR read, its confidence, which rule failed, and a sentence in plain English saying why — plus the image link, so a person settles it in about ten seconds.
A file is marked done only after it is processed, so a transient Vision error is retried on the next tick instead of being lost. Six-step setup in deploy/README.md.
What this demo does not prove
The one that got through. On order_007.png, Copperfield Supply reached the clean sheet as Copperfield Supp1y — a lowercase l read as a 1 inside a free-text field. No identity constrains a customer name, so nothing in this repo can catch it. It is in the report, by name.
The OCR backend is synthetic. It replays recorded word boxes and applies a fixed, seeded set of character corruptions. Crucially, not every corruption lowers the confidence score — some stay high on purpose, because a confidently wrong digit is the entire reason a totals check has to exist. If corrupted tokens always arrived with low confidence, the confidence gate alone would look perfect and this page would be a lie. A real --backend tesseract is included and optional. The verification layer is the transferable part; the OCR is swappable.
Twelve screenshots is a small sample, from one layout. Field anchoring is the part most likely to break on a real client's captures, and that cannot be assessed without them.
The reconciliation rule is layout-specific. subtotal + tax = total holds here. Discounts, shipping, rounding rules and multiple currencies all break that identity, and it has to be rewritten to match the client's data before it means anything at all.
Nothing can see the answers. The ground-truth file is used for scoring only; no module in the pipeline reads it, and the run loads it only after every routing decision is made. A test fails the build if that ever stops being true.