RAG · citation · refusal · evaluation

rag-grounded-qa

Question answering over fictional company policies that cites its evidence, refuses unsupported questions, and measures both behaviors.

Public · fictional corpus
The problem

A fluent answer without evidence is an expensive guess.

Naïve document Q&A answers even when retrieval found nothing relevant. A trustworthy system needs a citable source, a refusal threshold, and a labelled evaluation set that catches regressions before deployment.

Input

Questions against four fictional policy documents

In scope:  "How many PTO days can I carry over?"
Out of scope: "What is the office Wi-Fi password?"

Corpus: remote work, expenses, security, and PTO policies.
Evaluation set: 11 answerable + 3 out-of-scope questions.
The money shot

Committed evaluation result

Retrieval recall@31.011 answerable questions
Refusal accuracy1.014 total questions
Grounded rate1.0Every answer cites a source
Grounded answer

Q: How many PTO days can I carry over?

A: Up to 5 unused PTO days may be carried over into the following calendar year. [pto_policy#1]

Retrieved sources: pto_policy#1, pto_policy#5
Refusal

Q: What is the office Wi-Fi password?

A: I don't have enough information in the provided documents to answer that.

Refused: top score 0.262 was below the confidence threshold.
How it's verified

The evaluation harness is the moat.

The labelled set measures whether the expected source appears in the top three results, whether in-scope questions are answered and out-of-scope questions refused, and whether every accepted answer includes a citation. The bundled harness exposed two silent retrieval failures during development and turned them into regression guards.

Honest limitations
  • The corpus is four fictional policy documents and the evaluation set contains only 14 questions; scores do not generalize to an unseen production corpus.
  • Offline retrieval uses lexical TF-IDF. Paraphrases and domain terminology can require embeddings and a larger, representative eval set.
  • The in-memory implementation is a demonstration; larger corpora need a persistent vector store and operational monitoring.
  • Citations and refusal reduce risk but do not replace human review for consequential policy decisions.