Search & retrieval platform
Shia Library
Multilingual search over 315,000 ingested classical passages — one backend serving a live web platform and iOS + Android apps.
- ~315k
- hadith records ingested
- 91%
- claims grounded (30-q eval)
- 1,000+
- web tests in CI
- 0.94
- faithfulness (Claude judge)
The domain
The corpus is classical Twelver Shia scholarship (hadith and related texts) in Arabic with English translations — a domain where search relevance is hard (Arabic orthography) and correctness is high-stakes. The Ask feature is deliberately descriptive: it reports what the cited sources say, or their silence — no verdicts of its own — and prioritises the words of the Maʿsumeen over later commentary. Since mid-2026 the platform has a second surface: offline-first iOS and Android apps — live on the App Store and Google Play — served by this same backend, with their engineering told in a companion case study.
01 · Problem
The problem
Searching classical Arabic text is deceptively hard. The same word appears with or without diacritics (harakat), with variant letterforms (alef and ya variants), and with elongation marks (tatweel) — so a naïve query for a term silently misses most of its real occurrences.
Keyword search alone can't answer conceptual questions, and pure semantic search loses exact matches on names, citations, and hadith numbers that scholars depend on. The content is also sensitive: some books are access-restricted, and the full corpus must not be trivially scrapeable.
It has to be fast and cheap to serve a read-mostly corpus that has grown to over 315,000 ingested records across thousands of pre-rendered pages — and the same backend now has to serve a second client, an offline-first mobile app, without either surface compromising the other.
And answering questions over this corpus raises the stakes again: a fluent, confident wrong answer is worse than no answer. An "Ask" here has to ground every claim in a real passage, abstain when the sources don't support one, and never drift into giving verdicts of its own — it reports what the cited sources say, or their silence.
02 · Approach
Approach & key decisions
Diacritic-insensitive Arabic search at the database layer
An IMMUTABLE `normalize_arabic()` SQL function strips harakat and tatweel and folds letter variants, backed by a GIN trigram index so normalized matching stays fast as the corpus grows. A client-side mirror of the same normalization drives result highlighting, while the original glyphs are preserved verbatim for display. SQL-level normalization (indexed, not per-query) keeps search fast at over 300,000 rows.
Hybrid retrieval: keyword + semantic, not either/or
Postgres `tsvector` full-text search handles exact terms and phrases; a semantic path calls an OpenAI embedding (text-embedding-3-small) for conceptual queries. Pure vector missed exact names and hadith numbers, and pure keyword missed paraphrase — so both exist, fused with Reciprocal Rank Fusion rather than forcing one to win.
A chapter-title retrieval arm for entity recall
The corpus names its subjects in chapter titles ('Miracles of Imam al-Hasan al-Askari') but the hadith bodies rarely repeat the subject — so body-only embeddings simply can't find them. A third retrieval arm queries chapter titles with the distinctive words from the user's question (no hardcoded name lists), plus an alias module that folds transliteration variants (Hussain / Husayn / Husain). It cost nothing — no migration — and was one of the two changes that actually moved recall.
Per-book diversity cap — the cost-neutral win that doubled grounding
One enormous book was flooding the top-k slots and starving the answer of corroborating sources. Capping passages at three per book — free, one parameter — doubled the grounded rate from 26% to 52% and lifted faithfulness from 0.48 to 0.65, measured end-to-end. The durable lesson: recall@k was a poor proxy here; only end-to-end eval told the truth, and corpus coverage, not the algorithm, was the ceiling.
Citations that can't be hallucinated, and a descriptive-only abstain discipline
Passages are sent to Claude as document blocks with citations enabled, so the model returns validated character-offset spans — not prompt-engineered [1]s — which the UI renders as inline chips that open the cited passage with the span highlighted. Switching from a tool-use JSON approach to native citations took grounding from 52% to 91%. A trigram grounding verifier drops any citation it can't match back to source; if too little of the answer is grounded, the answer is withheld with an explicit insufficiency reason. The system prompt is descriptive-only: it reports what the sources say or their silence and never issues verdicts of its own.
Graceful degradation under a latency budget
The classify, rewrite, and rerank LLM calls each run under a timeout (8 / 9 / 15s) and fall back to sensible defaults instead of failing the request, which cut end-to-end latency from ~60–80s to ~27–32s. A 24-hour LRU cache replays a cached answer as a stream, and a persisted segments array (interleaved text + citation) means the inline [n] markers survive both a page refresh and a cache hit.
03 · Architecture
How it fits together
Clients — one backend
Retrieval
Ask · grounded answers (beta)
Admin lane
04 · Results
Results
- Over 315,000 hadith records ingested; 41 books (~188,000 hadith across ~21,000 chapters) published live and statically pre-rendered per build — searchable in diacritic-insensitive Arabic, English full-text, and semantic vector search. Publication is controlled independently from ingestion: every book clears a validation sweep before it goes public.
- On a documented 30-question golden set judged by two independent LLMs, the grounded "Ask" scores 91% of answer claims grounded, 100% correct abstention on out-of-corpus questions, and faithfulness 0.94 (Claude judge) / 0.87 combined, at ~19–30s end-to-end on the eval run (~27–32s in typical interactive use).
- One Postgres backend serves two clients: 75 versioned migrations, row-level security on every public table separating anon / auth / admin — with anonymous bulk SELECT on the corpus revoked to deter scraping — and durable mutation receipts plus a versioned change feed built for the mobile app's delta sync.
- Pages are statically pre-rendered and served from Vercel's edge with multi-layer caching and request de-duplication; an in-app command-palette CMS edits passages in place with full history and tag-based cache invalidation.
- 1,000+ automated tests run in CI — unit, component, and accessibility suites, 58 Playwright end-to-end cases across desktop and mobile browsers, and SQL contract tests replayed against an ephemeral Postgres 17 — plus secret scanning in CI and a hardened Content-Security-Policy verified on the live site.
- Deploys are deliberately gated: auto-deploy is off, and production ships through a gated multi-stage release workflow that verifies a CI-green commit and a completed backup, applies migrations, and probes live SQL contracts before and after the deploy.
05 · Tradeoffs
Honest limitations
- gpt-4o-mini is an unreliable faithfulness judge on this material — it threw 0.00 outliers on answers that were clearly grounded. I keep it for cheap signal but trust the Claude judge for the headline number, and I'd weight any future eval the same way.
- Coverage is the ceiling, not the cleverness: I tested the popular cost-neutral retrieval tricks — MMR, RAG-fusion, HyDE, multi-query — and rejected all of them on this sparse corpus. The per-book cap and the chapter-title arm were the only changes that moved the numbers. Empirics beat priors here.
- One large book (Bihar al-Anwar) is currently excluded from embeddings pending re-ingestion, so semantic recall over it is a known gap; keyword search still covers it.
- The Ask feature still ships behind a beta gate and is not publicly reachable in production — the externally checkable proof of the technique lives in the public citation-rag demo, which runs the same engine over public-domain text.
06 · Next
What I'd do next
- Finish the corpus: work the book-by-book validation queue that gates publication, and re-ingest + embed the excluded large book so semantic recall covers the whole library.
- Grow the golden set well past 30 questions and add a retrieval eval arm, so "coverage is the ceiling" is a tracked metric and not just a finding.
- Graduate the Ask feature out of beta to authenticated users, with per-user rate limits and the same ≤200-char restricted-quote previews the reader already enforces.