5:day
Employees may carry over up to 5 unused PTO days into the following calendar year.
RAG · cross-check · conflict gate
rag-multidoc-crosscheck
Fictional policy Q&A retrieves across every source, exposes contradictory claims, and refuses to manufacture a winner.
Public · fictional corpusThe problem
The top chunk can hide a stale or regional contradiction.
In policy and compliance, silently selecting one source is often worse than refusing. A safe system must show both claims and route disagreement to a person.
Input
One question across two fictional policy versions
Q: How many unused PTO days can I carry over into the next year? Sources: pto_policy_2023.md + pto_policy_2024.md
The money shot
Both claims survive retrieval; the conflict becomes the answer
Verdict accuracy1.012 questions
Citation coverage1.0Non-refused questions
Conflict recall1.03 labelled conflicts
8:day
Employees may carry over up to 8 unused PTO days into the following calendar year.
| Verdict | CONFLICT |
|---|---|
| Decision | needs_review |
| Reason | Comparable values differ; no source is silently preferred. |
How it's verified
A labelled harness tests every verdict class.
The 12-question offline evaluation covers CONSENSUS, CONFLICT, SINGLE_SOURCE, and INSUFFICIENT. It separately scores verdict accuracy, all-source citation coverage, and recall on the three conflicts because a missed contradiction is the costly failure.
Honest limitations
- All five documents and claims are fictional.
- Offline claim extraction uses simple number-and-unit regex; production needs unit conversion and richer normalization.
- The small labelled set does not establish production accuracy on unseen policies.
- Conflict detection does not decide which source is authoritative; a human resolves the discrepancy.