RAG · hybrid retrieval · evaluation
rag-hybrid-rerank
BM25 and a semantic proxy are fused with Reciprocal Rank Fusion, reranked, and compared side by side on a labelled fictional corpus.
Public · fictional corpusThe problem
Recall alone can hide a weak ranking.
On a small corpus every retriever may place the right document somewhere in the top three. MRR and nDCG reveal whether useful evidence appears first or is buried behind irrelevant chunks.
Input
Fifteen labelled questions over six fictional handbook documents
Exact: "How many paid leave days ...?" Paraphrase: "Can I skip the commute and just work from home ...?" Vocabulary mismatch: "laptop" vs "computer" Offline pair: BM25 + character-trigram semantic proxy
The money shot
Four configurations, measured on the same eval set
| Configuration | Recall@3 | MRR | nDCG@3 |
|---|---|---|---|
| BM25 only | 1.000 | 0.933 | 0.701 |
| SemanticProxy only | 1.000 | 0.922 | 0.714 |
| Hybrid (RRF) | 1.000 | 0.967 | 0.729 |
| Hybrid + Rerank | 1.000 | 1.000 | 0.788 |
MRR lift vs BM25+0.067
nDCG@3 lift vs BM25+0.087
Recall@31.000Saturated for all four
The honest result is not “recall improved”—it did not. Fusion and reranking improved where the relevant chunks appeared.
How it's verified
Retrieval quality is a table, not a claim.
The offline harness re-runs every labelled question through BM25, character-trigram SemanticProxy, Hybrid RRF, and Hybrid + Rerank, then computes recall@3, reciprocal rank, and rank-discounted nDCG@3 from retrieved source labels.
Honest limitations
- The corpus is six fictional documents and roughly 25 chunks; recall@3 saturates.
- Character-trigram cosine catches morphology and substrings, not true synonymy.
- Production should replace the proxy with learned embeddings and the feature reranker with a cross-encoder, then re-evaluate.
- These metrics describe the bundled 15-question set and do not guarantee unseen-corpus performance.