Search & retrieval platform
Shia Library
Multilingual search over a quarter-million classical passages — now with a grounded "Ask" that answers only from cited sources, or says the sources are silent.
- ~255k
- hadith indexed
- 91%
- answers grounded
- 100%
- abstained out-of-corpus
- 0.94
- faithfulness (Claude judge)
The domain
The corpus is classical Twelver Shia scholarship (hadith and related texts) in Arabic with English translations — a domain where search relevance is hard (Arabic orthography) and correctness is high-stakes. The Ask feature is deliberately descriptive: it reports what the cited sources say, or their silence — no verdicts of its own — and prioritises the words of the Maʿsumeen over later commentary.
01 · Problem
The problem
Searching classical Arabic text is deceptively hard. The same word appears with or without diacritics (harakat), with variant letterforms (alef and ya variants), and with elongation marks (tatweel) — so a naïve query for a term silently misses most of its real occurrences.
Keyword search alone can't answer conceptual questions, and pure semantic search loses exact matches on names, citations, and hadith numbers that scholars depend on. The content is also sensitive: some books are access-restricted, and the full corpus must not be trivially scrapeable.
It has to be fast and cheap to serve a read-mostly corpus that has grown to ~255,000 passages across thousands of pre-rendered pages.
And answering questions over this corpus raises the stakes again: a fluent, confident wrong answer is worse than no answer. An "Ask" here has to ground every claim in a real passage, abstain when the sources don't support one, and never drift into giving verdicts of its own — it reports what the cited sources say, or their silence.
02 · Approach
Approach & key decisions
Diacritic-insensitive Arabic search at the database layer
An IMMUTABLE `normalize_arabic()` SQL function strips harakat and tatweel and folds letter variants, backed by a GIN trigram index so normalized matching stays fast as the corpus grows. A client-side mirror of the same normalization drives result highlighting, while the original glyphs are preserved verbatim for display. SQL-level normalization (indexed, not per-query) keeps search fast at a quarter-million rows.
Hybrid retrieval: keyword + semantic, not either/or
Postgres `tsvector` full-text search handles exact terms and phrases; a semantic path calls an OpenAI embedding (text-embedding-3-small) for conceptual queries. Pure vector missed exact names and hadith numbers, and pure keyword missed paraphrase — so both exist, fused with Reciprocal Rank Fusion rather than forcing one to win.
A chapter-title retrieval arm for entity recall
The corpus names its subjects in chapter titles ('Miracles of Imam al-Hasan al-Askari') but the hadith bodies rarely repeat the subject — so body-only embeddings simply can't find them. A third retrieval arm queries chapter titles with the distinctive words from the user's question (no hardcoded name lists), plus an alias module that folds transliteration variants (Hussain / Husayn / Husain). It cost nothing — no migration — and was one of the two changes that actually moved recall.
Per-book diversity cap — the cost-neutral win that doubled grounding
One enormous book was flooding the top-k slots and starving the answer of corroborating sources. Capping passages at three per book — free, one parameter — doubled the grounded rate from 26% to 52% and lifted faithfulness from 0.48 to 0.65, measured end-to-end. The durable lesson: recall@k was a poor proxy here; only end-to-end eval told the truth, and corpus coverage, not the algorithm, was the ceiling.
Citations that can't be hallucinated, and an Akhbari abstain discipline
Passages are sent to Claude as document blocks with citations enabled, so the model returns validated character-offset spans — not prompt-engineered [1]s — which the UI renders as inline chips that open the cited passage with the span highlighted. Switching from a tool-use JSON approach to native citations took grounding from 52% to 91%. A trigram grounding verifier drops any citation it can't match back to source; if too little of the answer is grounded, the answer is withheld with an explicit insufficiency reason. The system prompt is descriptive-only: it reports what the sources say or their silence and never issues verdicts of its own.
Graceful degradation under a latency budget
The classify, rewrite, and rerank LLM calls each run under a timeout (8 / 9 / 15s) and fall back to sensible defaults instead of failing the request, which cut end-to-end latency from ~60–80s to ~27–32s. A 24-hour LRU cache replays a cached answer as a stream, and a persisted segments array (interleaved text + citation) means the inline [n] markers survive both a page refresh and a cache hit.
03 · Architecture
How it fits together
Retrieval
Ask · grounded answers (beta)
Admin lane
04 · Results
Results
- ~255,000 hadith are searchable across Arabic (diacritic-insensitive), English full-text, and semantic vector search — grown from ~66k via a daily GitHub Action embedding-backfill (COPY + bulk UPDATE).
- The grounded "Ask" scores, on a 30-question golden set judged by two independent LLMs: 91% of answer claims grounded, 100% correct abstention on out-of-corpus questions, and faithfulness 0.94 (Claude judge) / 0.87 combined, at ~19–30s end-to-end.
- Pages are statically pre-rendered and served from Vercel's edge with multi-layer caching and request de-duplication; an in-app command-palette CMS edits passages in place with full history and tag-based cache invalidation.
- Row-level security separates anon / auth / admin and revokes anonymous bulk SELECT to deter scraping; ~280 unit and component tests plus 17 Playwright e2e run in CI, alongside secret-scanning and a hardened Content-Security-Policy.
05 · Tradeoffs
Honest limitations
- gpt-4o-mini is an unreliable faithfulness judge on this material — it threw 0.00 outliers on answers that were clearly grounded. I keep it for cheap signal but trust the Claude judge for the headline number, and I'd weight any future eval the same way.
- Coverage is the ceiling, not the cleverness: I tested the popular cost-neutral retrieval tricks — MMR, RAG-fusion, HyDE, multi-query — and rejected all of them on this sparse corpus. The per-book cap and the chapter-title arm were the only changes that moved the numbers. Empirics beat priors here.
- One large book (Bihar al-Anwar) is currently excluded from embeddings pending re-ingestion, so semantic recall over it is a known gap; keyword search still covers it.
- The Ask feature ships behind a beta gate and is 404-invisible in production — it isn't publicly reachable yet, so the externally checkable proof of the technique lives in the public citation-rag demo, which runs the same engine over public-domain text.
- The platform is JavaScript with JSDoc rather than TypeScript; migrating the data layer to TS for compile-time safety on Supabase queries is the obvious next step.
06 · Next
What I'd do next
- Finish the corpus: re-ingest and embed the excluded large book so semantic recall covers the whole library, not most of it.
- Grow the golden set well past 30 questions and add a retrieval eval arm, so "coverage is the ceiling" is a tracked metric and not just a finding.
- Graduate the Ask feature out of beta to authenticated users, with per-user rate limits and the same ≤200-char restricted-quote previews the reader already enforces.