EPOCH AI

Institutional Archetype

THE SCOREKEEPER WITH A SPONSOR — Epoch AI measures how fast AI is improving: compute trends, data trends, and benchmarks hard enough to still mean something. Its FrontierMath benchmark was used to announce one of the most-cited capability jumps of the period. Then it emerged that OpenAI had funded the benchmark and had access to much of it, and that many of the mathematicians who wrote the problems had not been told. Epoch called it a mistake. The file is open on the structure that made the mistake possible: the scorekeeper’s hardest test was paid for by one of the players.

Mandate & Origin

A nonprofit researching AI trends and producing benchmarks; per TechCrunch in January 2025, “primarily funded by Open Philanthropy, a research and grantmaking foundation” (TechCrunch, Jan 19 2025). See The EA to AI-Safety Funding Pipeline.

Funding & Backers

Open Philanthropy (now Coefficient Giving) as primary funder, per TechCrunch; OpenAI funded the creation of FrontierMath, which Epoch “revealed on December 20” 2024, around the launch of OpenAI’s o3 (TechCrunch).

Position in the Apparatus

• The disclosure. A FrontierMath contractor wrote that contributors were “unaware that OpenAI will have exclusive access to this benchmark (and others won’t)” and that “Most express they are not sure they would have contributed had they known.” Co-founder Tamay Besiroglu said the benchmark’s integrity had not been compromised but that Epoch “made a mistake,” and that it should “have made transparency with our contributors a non-negotiable part of our agreement with OpenAI” (TechCrunch).
• The safeguards, as stated. OpenAI has a “verbal agreement” with Epoch not to train on FrontierMath’s problems, and Epoch keeps a “separate holdout set” for independent verification (TechCrunch).
• Its counterweight work. Epoch’s own analysis is cited on this site for asking whether lab biorisk evaluations measure weapon-building risk at all (AI and Bioweapons Uplift).

Actions & Leadership Choices

CONDUCT: LATE DISCLOSURE — LAB-FUNDED BENCHMARK, LAB-ACCESSIBLE ANSWERS. Epoch admitted the disclosure failure in its own co-founder’s words. The safeguard it offered against the obvious risk, training on the test, was a verbal agreement. The finding is structural: a benchmark used to announce a lab’s capability gain was funded by that lab, and the protection against contamination was a promise.

Reach Assessment

Institutional: Its trend data and benchmarks are cited in lab announcements, policy papers and press. Memetic: FrontierMath scores were headline numbers; the funding story became the standard example of benchmark capture.

Sources: AI benchmarking organization criticized for waiting to disclose funding from OpenAI — TechCrunch (Jan 19, 2025).