AI evaluation & methodology
Judgelab
A reproducible lab that measures how reliable LLM-as-a-judge systems actually are — chance-corrected agreement with confidence intervals and a keyless, license-clean benchmark, not a single agreement score.
- 1,814
- aligned judge–human comparisons
- κ 0.767
- GPT-4 vs human (ties excluded)
- α 0.485
- human vs human ceiling
- 127
- tests · keyless CI
The domain
This one has no corpus and no domain — it's about the tools everyone now trusts blindly. LLM-as-a-judge decides which model ships and what a reinforcement-learning policy is rewarded for, yet its reliability is usually reported as one number. Judgelab reproduces the canonical GPT-4-vs-human study on MT-Bench (a public, neutral chatbot benchmark) and asks the harder question honestly. Python-first — a deliberate change of stack from the rest of this portfolio.
01 · Problem
The problem
LLM-as-a-judge is load-bearing now: it decides which model ships and what an RL policy is rewarded for. But its reliability is almost always reported as a single "agrees with humans 88% of the time" — a number that ignores how often two raters would agree by chance, and says nothing about bias, self-consistency, or whether the humans even agree with each other.
Raw agreement systematically overstates reliability. Chance-corrected agreement (Cohen's kappa) tells a different story, and a point estimate with no confidence interval hides how uncertain it is. The field's own gold standard — human labels — is itself only moderately consistent, so a judge that looks impressive against humans may simply be sitting at the human ceiling rather than beating it.
And auditing a judge is no longer a novel idea in 2026 — research harnesses already exist. So the bar isn't the concept; it's doing the statistics correctly, making the whole result reproducible with no API keys, and being honest about exactly what the numbers do and don't show.
02 · Approach
Approach & key decisions
A committed, license-clean benchmark you can re-run for free
The MT-Bench human judgments (CC-BY-4.0) ship both the human votes and the published GPT-4 verdicts, snapshotted into the repo with provenance and per-file sha256 — after a primary-source license review, and dual-licensed (MIT code, CC-BY-4.0 data with attribution). The benchmark runs from that committed snapshot with no API keys and no network, so anyone can reproduce it for $0.
Order-invariant alignment, honest about position bias
Human and GPT-4 verdicts use different presentation orders for the same pair, so each verdict is reduced to a model-identity preference in a canonical ordering — they align even when the rows are swapped. The 1-to-7 human annotators per comparison are combined by majority vote, and the GPT-4 "tie (inconsistent)" value — a verdict that flipped when the two answers were swapped — is surfaced directly as a position-inconsistency rate.
Chance-corrected statistics, with intervals
Raw agreement is reported next to Cohen's kappa so the gap between them is explicit, with seeded percentile bootstrap 95% confidence intervals on every figure. Each statistic is pinned by a hand-computed fixture and cross-checked against an independent implementation — scikit-learn, SciPy, and the krippendorff package — because a statistics library is only worth its cross-checks.
The human-human ceiling, so the judge is read in context
Krippendorff's alpha — which handles the variable, missing annotators that Cohen's kappa cannot — measures how much the humans agree with each other. That turns the judge-human number from a figure in a vacuum into one read against a ceiling: a judge cannot be more reliable than the reference it is graded against.
Reproducibility as a feature, not a footnote
Content-addressed experiment fingerprints, a deterministic seeded runner, and an append-only store that never overwrites a result underpin the engine. The reliability card is a committed artifact that CI re-derives byte-for-byte on every push — a keyless drift gate — so no number on the page can ever silently diverge from the data or the code that produced it.
A Python engine that runs and tests keyless
Judges sit behind a provider interface with a deterministic fake judge, so the whole pipeline runs and is tested with no paid calls. Strict mypy, Ruff lint + format, 127 tests, and a fully keyless GitHub Actions CI (with secret scanning) hold the bar — the first Python-scientific-stack project in this portfolio, chosen deliberately over reusing the TypeScript one.
03 · Architecture
How it fits together
Committed snapshot · MT-Bench (CC-BY-4.0)
Chance-corrected statistics
04 · Results
Results
- Reproduced the canonical MT-Bench GPT-4-vs-human agreement over 1,814 aligned comparisons (from 3,355 human votes and 2,400 GPT-4 verdicts): raw agreement 0.884 on decisive cases, in line with the published ~85%.
- Added what the original left out — chance-corrected Cohen's kappa with bootstrap 95% CIs: kappa 0.767 [0.726, 0.804] on decisive cases, dropping to 0.505 [0.472, 0.538] once ties are included. The raw-vs-kappa gap is the headline: raw agreement overstates reliability, and kappa is what should be reported.
- Measured the human-human ceiling: Krippendorff's alpha 0.485 over the 961 multiply-annotated comparisons. So with ties, the GPT-4 judge (kappa 0.505) agrees with humans about as well as humans agree with each other — it is sitting at the human reliability ceiling, and a high raw score does not mean it beats them.
- Quantified position bias straight from the data: the GPT-4 judge flips its verdict on 16% of comparisons when the two answers are swapped.
- The whole benchmark is keyless and free — computed from the committed CC-BY-4.0 snapshot with one command (`judgelab report`) and re-derived byte-for-byte in CI as a drift gate. 127 tests, public and MIT-licensed, every statistic cross-checked against a trusted independent implementation.
05 · Tradeoffs
Honest limitations
- It ships one benchmark today: agreement on MT-Bench. The full bias battery it is designed for — position, verbosity, self-preference, and test-retest probes, calibration (ECE), and prompt-injection robustness — is in the architecture and the README's roadmap, but not yet built; those need live judge calls, which stay owner-gated behind cost caps. The README marks that built-vs-roadmap split explicitly rather than implying more is done.
- Auditing a judge is not a novel idea in 2026 — research harnesses like RAND's Judge Reliability Harness already exist. Judgelab's value is doing the statistics correctly (chance-corrected, with CIs), adding the human-human ceiling, and full keyless reproducibility — rigor and integration, not first-of-kind. I would rather say that than oversell it.
- The headline replays published GPT-4 verdicts; running a live judge on fresh orderings is the paid path that is not wired up yet. The deterministic "fake" judge in the repo exercises the pipeline and the tests but is not a measurement of any real model.
- Comparing the judge's Cohen's kappa to the humans' Krippendorff alpha is fair in magnitude but not a single unified statistic — both are chance-corrected nominal agreement on the same three-way verdicts. It contextualises the judge; it does not collapse the two into one number.
06 · Next
What I'd do next
- Wire the owner-gated live-judge path (OpenAI / Anthropic adapters behind cost caps) and run the bias battery on fresh orderings: position, verbosity, self-preference, and test-retest probes.
- Add calibration (ECE + reliability diagrams) and a prompt-injection attack-success-rate, then emit a per-judge reliability card.