AI / retrieval systems
Citation-Grounded RAG
A retrieval-augmented answer engine where every claim is grounded in a specific source — hybrid retrieval, verifiable citations, an abstain path, all measured by an eval harness.
- 266
- passages, public-domain
- 0.91
- recall@5 (hybrid)
- 1.00
- faithfulness (Claude, 22-q)
- 3/3
- out-of-corpus abstained
The domain
The public demo answers over the U.S. founding documents — the Constitution, the Bill of Rights, and the Federalist Papers (public domain) — so every answer is externally checkable. The same engine now runs in production on the Shia Library — a ~255k-passage classical corpus where citation and abstention matter most.
01 · Problem
The problem
Most 'chat with your docs' demos paste chunks into a prompt and hope. On high-stakes text that's dangerous: the model produces fluent, confident prose with invented '[1]' markers, and there's no way to tell a grounded answer from a hallucinated one — or to know when the honest answer is 'the sources don't say.'
I wanted the opposite defaults: every claim tied to a verifiable span, a system that declines rather than guesses, and — critically — numbers that prove it works, so changes are measurable instead of vibes.
02 · Approach
Approach & key decisions
Hybrid retrieval: BM25 + embeddings, fused with RRF
Lexical BM25 catches exact terms, names, and citation numbers ('Article I, Section 8'); dense embeddings catch paraphrase and concepts. Reciprocal Rank Fusion combines the two ranked lists without needing their scores to be comparable. The corpus sits behind a small interface, so the same pipeline runs in-process for the demo and over Postgres/pgvector in production.
Citations that can't be hallucinated
Each source is sent as an Anthropic document block with citations enabled, so the model returns text spans tied to a specific source with exact character offsets — not prompt-engineered '[1]'s. The UI renders those as inline chips; clicking one opens the retrieved passage with the cited span highlighted in context (with a hover preview on desktop), so a reader can check any sentence in one click.
Abstention as a feature
The prompt instructs the model to return a sentinel when the retrieved sources don't support an answer, and the UI shows 'no supported answer' instead of guessing. A retrieval-confidence gate (top cosine ~0.65 for answerable vs ~0.15 for unanswerable questions) can pre-abstain before even spending a generation call.
Streaming with inline citations
Answers stream token-by-token over newline-delimited JSON; each Anthropic content block becomes a segment, so citation chips attach inline exactly where they belong as they arrive, rather than all bunched at the end.
Measured, not asserted — an eval harness
A golden set scores retrieval (recall@k, MRR) and answer faithfulness — judged by two independent models (Claude and gpt-4o-mini), both shown the full cited passages — and the recall gate runs in CI on the Claude judge so a regression fails the build. The harness doubles as an ablation: it's how I know dense, hybrid, rerank, query-rewrite, and a per-source diversity cap each help — and by how much.
03 · Architecture
How it fits together
Hybrid retrieval
Evaluation (CI-gated)
04 · Results
Results
- On a 22-question golden set (k=5): hybrid retrieval reaches recall@5 0.91 / MRR 0.81; query rewrite or an LLM reranker lifts recall to 0.96. Faithfulness is judged by two independent models — Claude 1.00 (the headline) and gpt-4o-mini 0.96, combined 0.98, with gpt-4o-mini scoring 0.00 on one clearly-grounded answer (the cheap judge is the unreliable one). Out-of-corpus questions abstain 3/3.
- Native citations mean every claim links to a verifiable span — clicking a chip opens the passage with the cited text highlighted in context, with a hover preview on desktop.
- Repeat questions replay from a 24-hour cache instead of re-spending on generation, and each grounded answer ends with fail-soft suggested follow-ups.
- Runs three ways from one codebase: fully offline (BM25 + a deterministic answer, no keys), dense + hybrid with an embeddings key, and live streamed cited answers with Claude.
- CI fails the build if recall@5 regresses below the gate, so retrieval quality is protected, not assumed.
05 · Tradeoffs
Honest limitations
- The honest ablation result: on this concept-heavy corpus, dense-alone (0.96) actually edged out naïve RRF hybrid (0.91) — unweighted fusion can dilute a strong retriever. Hybrid earns its keep on keyword, name, and citation-heavy queries (and on the real Arabic + English corpus); the clean fix for the dilution is the reranker.
- Reranking and query rewriting each add an LLM call, so they're quality/latency tradeoffs I left as measured, opt-in stages rather than always-on.
- I backported the per-source diversity cap that doubled grounding on the production corpus — where one giant, irrelevant book flooded retrieval — but measured it here too: on the founding documents it lowered recall@5 from 0.91 to 0.86, because the flooding source is usually the relevant Federalist essay. It ships measured and off by default. Retrieval wins are corpus-dependent — the point of the eval harness is to catch exactly this.
- The demo corpus is ~266 passages — chosen to be public and checkable, not to prove scale. The scale story lives in the Shia Library, where this same engine now runs over ~255,000 passages in production.
06 · Next
What I'd do next
- Deep-link each citation chip straight to its source page in the reader, now that the production corpus path is live.
- Promote the cross-encoder reranker and the retrieval-confidence pre-abstain gate into the default pipeline, behind a latency budget.