Machine learning & evaluation
Polarity
A 184M-parameter cache-equivalence model, and the scientific debugging trail that established both where it works and where it fails.
- 184M
- trainable parameters
- 2/44
- Veritas false accepts
- 0.640
- MRPC ROC-AUC
- 416
- tests in hosted CI
The domain
Polarity predicts whether two queries are equivalent enough to reuse the same cached answer. It is deliberately narrower than general semantic similarity or paraphrase detection. The public release contains source, documentation, manifests, score artifacts, and reproducibility gates; model weights, raw MRPC text, and private run history are not shipped.
01 · Problem
The problem
A semantic cache saves latency and cost only when a hit is correct. A false positive is the dangerous error: a query with a changed date, quantity, entity, or negation can receive a fluent cached answer to a different question.
In the historical 75-pair Veritas evaluation, one meaning-changing pair reached cosine similarity 0.9590 and outranked every paraphrase. The public repository includes the pair data and aggregate evidence, but not the original embedding vectors, so this is historical motivation rather than a metric reproducible from the public source alone.
The task therefore needed more than a better cosine threshold: leakage-safe pair data, evaluation at low false-positive operating points, and a classifier trained to separate cache equivalence from surface similarity without treating a validation budget as a deployment guarantee.
02 · Approach
Approach & key decisions
Define the decision around the costly error
The positive class is cache-equivalent, so every evaluation names false accepts explicitly. Thresholds are selected on validation and then frozen; realized false-positive rates are reported separately on internal test, Veritas, and MRPC instead of being presented as guarantees.
Build a leakage-safe, task-specific recipe
The primary recipe combines PAWS-Wiki with CANNOT, capped deterministically and split by source family so near-duplicate families cannot straddle train and evaluation. The data audit excluded MNLI from the final recipe and kept Veritas out of training and threshold selection.
Falsify no-training alternatives first
Five preregistered baselines tested exact matching, normalization, embedding thresholds, an adaptive threshold, and an off-the-shelf NLI cross-encoder. The best NLI baseline reached only 0.047 recall in a descriptive internal-test sweep at actual FPR≤1%; at its validation-selected threshold it reached 0.073 recall with 16/883 false accepts.
Correct the metric before interpreting the model
Two initial DeBERTa runs appeared to show low-FP signal until an audit found that the partial-AUC implementation over-credited degenerate curves. Re-scoring the committed predictions with the corrected metric overturned that conclusion and exposed complete constant-output collapse. The prediction artifacts were preserved; the scientific interpretation changed.
Falsify the precision hypothesis
A controlled fp32 run kept the same recipe and produced stable gradients, but collapsed exactly like the bf16 runs. That ruled out reduced precision as the dominant cause and shifted the investigation from numerical instability to the model signal path.
Isolate a Transformers-version-sensitive boundary
Author-run one-batch controls showed the same pipeline could train DistilBERT, while DeBERTa learned under Transformers 4.x and failed under 5.15.1. Committed full-run predictions support collapse under 5.15.1 and success under 4.57.6; the decisive small diagnostic logs are not machine-readable in the release, so the narrower version-sensitive finding is preserved without claiming a bisected upstream defect.
Recover, calibrate, and test the boundary
The successful fp32 run pinned Transformers 4.57.6 and used a 184,423,682-parameter model with no frozen parameters. Checkpoint selection used the corrected low-FP metric, while isotonic calibration and the operating threshold were fit on validation only. The frozen operating point was then applied to internal test, Veritas, and MRPC without rewriting the model scores.
Release evidence without overreleasing artifacts
The source-only public repository runs keyless CI on Python 3.12 and 3.13, re-derives committed reports, verifies artifact identities, and scans for secrets. The checkpoint remains private because its redistribution posture is unresolved; the release does not contain raw third-party datasets or private history.
03 · Architecture
How it fits together
Frame the decision
Diagnose the collapse
Recover, calibrate, then try to break it
04 · Results
Results
- On the same-distribution internal test, a descriptive sweep over each model's test-side curve gave custom step-pAUC[0,2%] 0.716 versus 0.048 for the NLI baseline, and recall 0.763 versus 0.047 at actual internal-test FPR≤1%. These are threshold-free curve summaries, not the frozen operating point.
- At the separate validation-selected tau=0.994833, internal-test recall was 0.903 with 21/883 false positives (2.38% realized FPR). The higher recall comes from transferring that validation-selected threshold, not from the descriptive test-side FPR≤1% point. At the same tau, equivalent-negation recall was 57/58 (0.983) and explicit-negation FPR was 1/107 (0.009).
- On the author-created, protocol-held-out Veritas target-domain set, the same validation-selected threshold recalled 31/31 equivalent pairs with 2/44 false accepts (4.55% realized FPR), including 0/22 negation false accepts.
- External MRPC transfer was weak: ROC-AUC 0.639870 and 332/578 false accepts (57.44% realized FPR) at the frozen threshold. Both classes received very high scores, so the model is not a general paraphrase detector.
- The public source release is guarded by 416 passing tests in hosted CI across Python 3.12 and 3.13, plus metric, data, leakage, artifact-identity, and secret-scanning gates. Model weights remain private.
05 · Tradeoffs
Honest limitations
- Veritas is small (N=75), author-created, and present from the project's first milestone. Protocol-before-score chronology is committed, but this is target-domain evidence rather than an independently blinded or general-OOD benchmark.
- The threshold was selected on validation to target FPR≤1%; it realized 2.38% FPR on internal test, 4.55% on Veritas, and 57.44% on MRPC. A selection budget is not a production guarantee.
- MRPC exposes weak cross-domain ranking and score saturation. A separate post-hoc probe found measurement, temporal, and numeric substitutions were weak slices, but it is too small and author-curated to prove the cause of the MRPC failure.
- Only one final model and one seed were evaluated. The exact historical Python version, platform/container, and standalone CUDA runtime were not recorded, so source and metric reproduction are stronger than byte-for-byte training reproduction.
- The weights are private and carry no public redistribution license. A fresh user can audit source and reproduce committed metrics, but cannot run the trained model from the public repository alone.
06 · Next
What I'd do next
- If weight release becomes worthwhile, retrain from a provenance-clean data recipe under a fully captured environment and review the resulting checkpoint rights separately from the code license.
- Any future domain-generalization claim should start with a preregistered, independently authored, untouched evaluation set rather than further tuning against Veritas or MRPC.