The Judge
Can a machine referee political bias without carrying it? Building a judge with the flinch removed — and finding out what actually causes the flinch.
A keyword scanner cannot detect a frame; only a reader can. The natural reader is a large language model, and the natural objection is immediate: the model has the lean too. Ask an aligned model “is this text smuggling a political frame?” and it does not evaluate the structure of the claim. It reports whether the claim is comfortable. Hand it an openly-labeled heterodox thesis — a stated argument, not a smuggle — and it flags the thesis. A detector that flags the content it is supposed to be scoring is not a detector.
The Wash treats that flinch as a measurable, removable property of the judge. It is the bias study’s judging instrument: an open-weight model with the refusal direction projected out of its weights (abliteration), gated into the panel only after it demonstrates at scale that it does not flinch. This page is the short version; the full methods-and-results writeup is linked at the bottom.
The problem is measurable
Re-scoring 160 contested-politics answers with both judge types: the abliterated spine flags 0–9% of items; stock aligned judges flag 38–90%. And the controlled flag-rate rises with how openly a stance is declared — which is the tell. It means the aligned judges are measuring their own discomfort, not detecting a smuggle. Two controlled judges built by different vendors even assign opposite mean coding directions to the framing they flag; there is no vendor-neutral aligned referee.
The central finding, CI-backed
Across a 16-template documented-criticism battery scored by the spine plus five controlled model families, the aligned judges over-flag documented institutional criticism the spine passes — the exact material a bias detector must leave alone:
| Item type | Aligned judges | Abliterated spine | Gap |
|---|---|---|---|
| Plain single documented facts | 0.16 [0.13, 0.21] | 0.00 [0.00, 0.06] | +0.16, CIs disjoint |
| Juxtaposed (a partial smuggle) | 0.55 [0.49, 0.61] | 0.31 [0.20, 0.45] | +0.24, CIs disjoint |
| Declared opinion / plain non-criticism | ~0.00 | ~0.00 | none — both pass |
| Blatant smuggles | 1.00 | 1.00 | none — both catch |
Significant at the item level (sign test p=0.002, 9/9 documented items; cluster-robust GEE OR=3.13, p=0.022), and specific — the spine passes clean material and still catches blatant smuggles at 1.00. Its low flinch is a genuine non-flinch, not blunting: on graded smuggles the spine scores ROC-AUC 0.88, d′ 2.38, beating the two most flag-happy aligned families.
The part where the study red-teamed itself
The obvious story would be “abliteration removes the flinch.” Three independent hostile reviewers went at that claim, and the follow-up runs refuted it. Within a single 9B model, stock-vs-abliterated flips sign between runs and pools to roughly equal (~0.15) — two runs straddling zero. The over-flagging is real and it is model size and family (a 9B flinches at ~0.15, a 27B at ~0.52), not the scalpel. Abliteration removes refusal; it does not remove this content-flinch.
The phenomenon stands. The practical instrument claim — use a low-flinch judge as your referee — stands. The mechanistic claim that abliteration is what cleans the judge is refuted. A study that publishes the correction against its own headline is doing the thing the whole project is about.
Where it sits in the stack
This is the reader the keyword tools are missing:
- The Capture Scanner is the keyword version — it runs in your browser and surfaces loaded words and whether they are attributed or asserted. Fast, transparent, and by construction blind to frame: it cannot tell an argument from a smuggle. That is the limit The Wash opens on.
- The Judge is the reader — a model that reads context and reports whether a structurally identical claim is treated differently, with the flinch measured and gated out.
- Tradecraft is where the two meet: its verify step runs exactly this kind of low-flinch judge as its
local/cloudbackend to confirm a cue hit isn’t a false positive. The Wash proves why an ordinary aligned model is a poor referee; Tradecraft is what consumes a good one.
Read it / run it
- The Wash — full methods & results — dose-response, decoupling, the documented-register battery, the ROC validation, and the §F adversarial review that forced the correction
- bias-study repository (v2.0.0) — the full institutional-skepticism audit this judge serves
- Reproduce it — the question set, rubric, and run protocol: point your own model at it
- The Capture Scanner · Tradecraft · Capture Leaderboard
Where it appears in print: The Ratchet and Quiet Autocomplete (Evil Robots Series) — the bias-in-the-referee problem, and why “let the AI decide what’s neutral” is the trap.