The Experiment
Guess before you look. 780 scored responses, 390 A/B pairs, thirteen models — and every delta shown against the interval that decides whether it counts at all.
Why you are asked to commit first
Because being wrong is the finding. The instruction under test moves model output by a small amount, and a small amount is precisely what a reader cannot detect by looking at examples. If you can predict which response was steered, the effect is obvious and needs no statistics. If you cannot — and most people cannot — then the only honest way to measure it is over hundreds of pairs with confidence intervals, which is what the study does.
So you are handed pairs one at a time and asked to pick. Your score at the end is not a game result. It is the calibration argument.
What is deliberately withheld until you have answered
The base rate. Most responses in this study received the same middling score, so “guess the modal value” beats almost any reasoning strategy on a single round. Telling you that up front would anchor every later guess, so it is disclosed after your first commit and not before.
The rule that governs every number here
A delta counts as a finding only when its 95% confidence interval excludes zero.
That is enforced in the renderer, not in prose: intervals are computed in Python with a seeded bootstrap and shipped as data, and the page refuses to display a delta that has no interval attached. Models whose interval crosses zero are labelled not a finding no matter how large the mean looks. Five of the thirteen clear that bar.
Two further disclosures you will see on every round: each response is shown as its first 500 characters while the judges scored the full text, so you are working from less evidence than they had; and the deck is weighted toward pairs whose scores actually moved, because 256 of the 390 scored pairs did not move at all and a uniform deal would hand you those most of the time.
Unscored records render as an em dash. They are never treated as zero.