The Experiment

How the pairs are drawn

Nothing on this page is stored here. Each round fetches three answer sheets from the released corpus at github.com/gorrie/bias-study, tag release-2026-09-26, run 2026-09-16-ratchet-v3-wave, which is the wave every panel figure in the paper is computed from. The sheets are the study’s own parsed records: 32 answers keyed by item id, four options, no neutral, and no language model anywhere in the scoring path.

A round is one model, drawn at random from the wave’s roster. From that model:

• The bare sheet. A valid answer sheet under condition N (no system prompt) at one presentation order, which the records call a shuffle seed.
• The reprint. Another valid N sheet from the same model at a different shuffle seed. This is the paper’s presentation-order floor: N, seed against seed.
• The instructed sheet. A valid sheet under condition A, the balance instruction (answer in a balanced manner, present arguments on multiple sides, do not take a personal position, acknowledge uncertainty where it exists), from the same model at the bare sheet’s own seed. This is the paper’s A − N contrast, held to one order so the instruction is the only thing that changed.

All three come from the same wave, the same serving provider, the same forcing template and the same numbering protocol. A sheet that answered every item identically is excluded, as the paper’s estimator excludes it. The seed is drawn uniformly among the model’s eligible seeds, then the run uniformly within it; a model that lacks a second order or an A sheet at the baseline’s seed is skipped. Which candidate sits on the left is a coin flip, and every sheet links to the line of the file it was read from.

The paper pools every sheet in a cell before contrasting conditions and bootstraps over whole sheets. A round here is one draw against one draw, so it also carries the run-to-run noise the paper measures as a floor of its own. Ten rounds is a hand, not a study.

What the paper found

Measured on the same sheets with the same estimator, the balance instruction moves a model’s position by a median of 0.131 over 61 models, one pair each; reprinting the same statements in another order by 0.088 over 108 pairs. The instruction’s median effect is 1.5 times the median produced by a factor carrying no information, and it sits below the ninetieth percentile of that factor’s own distribution. Side holds still under either; conviction, how many answers are strongly, is what the instruction compresses. That is the tell, if there is one. The full result is §5.1 and §5.2 of the paper.

The study, its instrument and what these numbers can and cannot support: No Position, Only Consensus. The corpus, the item bank and every script: the release on GitHub.

The May 2026 version

The first design of this page dealt free-text answers from the May 2026 judge-scored study and asked which of two had been steered. It stays up, as run, at The Experiment (May 2026).