The Experiment
Guess before you look. 780 scored responses, 390 A/B pairs, thirteen models — and every delta shown against the interval that decides whether it counts at all.
This page is the May 2026 judge-scored study, run 2026-05-25-full: 780 records, thirteen models, a 1–5 rubric scored by a cross-vendor judge panel. It is kept as it was run. The instrument that replaced it — 62 forced-choice propositions on a frozen 31-model panel, wave 0 collected 2026-09-05 — has its own page, The Barometer, with its own floors. A number from one is not a correction to the other.
Why you are asked to commit first
Because being wrong is the finding. The instruction under test moves model output by a small amount, and a small amount is precisely what a reader cannot detect by looking at examples. If you can predict which response was steered, the effect is obvious and needs no statistics. If you cannot — and most people cannot — then the only honest way to measure it is over hundreds of pairs with confidence intervals, which is what the study does.
So you are handed pairs one at a time and asked to pick. Your score at the end is not a game result. It is the calibration argument.
What is deliberately withheld until you have answered
The base rate. Most responses in this study received the same middling score, so “guess the modal value” beats almost any reasoning strategy on a single round. Telling you that up front would anchor every later guess, so it is disclosed after your first commit and not before.
The rule that governs every number here
A delta counts as a finding only when its 95% confidence interval excludes zero.
That is enforced in the renderer, not in prose: intervals are computed in Python with a seeded bootstrap and shipped as data, and the page refuses to display a delta that has no interval attached. Models whose interval crosses zero are labelled not a finding no matter how large the mean looks. Five of the thirteen clear that bar.
Two further disclosures you will see on every round: each response is shown as its first 500 characters while the judges scored the full text, so you are working from less evidence than they had; and the deck is weighted toward pairs whose scores actually moved, because 256 of the 390 scored pairs did not move at all and a uniform deal would hand you those most of the time.
Unscored records render as an em dash. They are never treated as zero.
The same study, other ways in
- Bias Study — Data Browser — all 780 scored responses as rows you can filter and sort, each with its per-judge breakdown. Once you have committed your guesses, this is where you go to argue with the scoring.
- The Judge — what it takes to build a bias judge that does not flinch, which is the problem any scoring of this kind has to solve before it can count anything.
- The Wash — the same class of judge turned on the author’s own prose, including what it found there.