#Bias-Study

Two answer sheets from one model. One was told to answer in a balanced way. The other was just reprinted in a different order. Call which is which, then see how far each moved.

Every answer sheet in the released bias study, fetched from the GitHub release as you look: one model’s 32 answers under each instruction as a row of marks, or one proposition across every model.

Observed response differences on a fixed-panel forced-choice instrument, with descriptive references, failure counts and explicit limits on inference.

Guess before you look. 780 responses, 766 scored records, 390 A/B pairs, thirteen models — and every delta shown against the interval that decides whether it counts at all.

The same 32 propositions scored two ways — a four-model judge panel reading free text, and the forced choice the released study uses instead. Where the two agree, where they part, and where the judge reports its midpoint, drawn live from the release on GitHub.

Interactive filter and inspect tool for the 780 scored model responses in the cross-vendor AI bias study. Filter by model, vendor, condition, position, score. Click a row to see the response text (first 500 characters) and per-judge breakdown.

What a political instrument measures in a language model when nothing political has changed. 32 author-written propositions, 65 models, 3,897 runs, no language model in the scoring path: the deliberate manipulation is the same size as reprinting the questions in a different order, refusal is a switch the prompt throws, and the models barely disagree. An audit of fourteen published studies finds the control that would have shown it missing from all of them. Data, scripts and paper at github.com/gorrie/bias-study.