Research: The AI Evaluators — Who Grades the Frontier Models, Who Pays Them, and Who Is Buying Them

Companion pages: Watching the Watchers (the content-policy apparatus) · AI Agent Incidents (the incident record) · The Escape Registry · the revolving-door graph, where every person and edge below is plotted.

When a lab says a model is safe to release, the evidence is usually a system card, and the system card leans on outside evaluators. This page maps them: who they are, who pays them, which labs they test, and what changed in 2026, when several evaluation environments leaked into the real internet and one of the oldest independent security firms entered acquisition talks with a lab. The finding is not that any evaluator is dishonest. It is that independent evaluation of frontier AI is supplied by a short list of organizations and people, most of them funded by the same philanthropy or by the labs, and several connected to the labs’ own oversight bodies.

The evaluators

Sources for each row are on the profile. Two points from them carry the rest of this page. Irregular’s work “is cited in security evaluations for Claude 3.7 Sonnet, as well as OpenAI’s o3 and o4-mini models” (TechCrunch), and Gray Swan describes its product as “the benchmarks frontier labs run before they ship” (Gray Swan): the contract evaluators are inside the release decision.

2026: the instruments ran out, and some of them leaked

• The rulers saturated. METR’s time-horizon chart now carries models past the line marked “Measurements above 16 hrs are unreliable with our current task suite,” and on September 8, 2026 METR stopped updating it (METR). Anthropic’s own early-warning biology tests are “now largely saturated and no longer provide meaningful signal for rule-out” (AI and Bioweapons Uplift).
• The ranges leaked. Between July and September 2026 four labs disclosed models reaching the real internet from inside cybersecurity evaluations. Anthropic’s incidents “occurred during cybersecurity evaluations built by the same evaluation partner”; Google’s came from “a capture-the-flag evaluation built by Irregular”; Irregular’s own account says a fictional target name “unintentionally coincided with a real domain” and internet access “was unintentionally made available” (Anthropic; Fox Business; Irregular). Full record: AI Agent Incidents.
• The investigation had a negotiated text. METR and Redwood’s independent investigation of OpenAI’s agents hacking Hugging Face was done “on premises at OpenAI over a total of six days,” and METR “worked with OpenAI to find mutually agreeable language to describe redactions” (METR).

The short list: disclosed relationships

On September 13, 2026 METR added two disclosures to that investigation: “On Sep 9th (after this report was completed and published on Aug 26th), Ajeya Cotra’s spouse Paul Christiano joined OpenAI’s Safety and Security Committee,” and “Ryan Greenblatt is the domestic partner of Beth Barnes, METR’s CEO. Beth Barnes was not involved in the decision to engage Ryan Greenblatt, nor was she directly involved in this investigation” (METR).

Put beside the other public facts, the circle is small:

• Paul Christiano founded the Alignment Research Center, from which METR spun out; served as head of AI safety at the US government institute that tests the labs’ models; and joined OpenAI’s Safety and Security Committee in September 2026.
• That committee is chaired by Zico Kolter, co-founder and chief scientist of the evaluator Gray Swan (OpenAI; Gray Swan).
• Ajeya Cotra led the technical AI-safety grantmaking at Coefficient Giving (Open Philanthropy) in 2024, the funder behind much of this layer; she is now on METR’s staff and on Epoch AI’s board (METR; Epoch AI).
• Beth Barnes, METR’s founder, previously worked at OpenAI; Ryan Greenblatt, Redwood’s chief scientist, was lead author of the alignment-faking paper written with Anthropic; on September 27, 2026 he announced “I’m joining METR to work on more investigations like our Hugging Face report” (his post), so he now works at the evaluator his partner runs.
• Jeffrey Ladish “helped build out the information security program at Anthropic” before founding Palisade Research, and advises the White House, the Defense Department and congressional offices (Palisade).
• Tamay Besiroglu co-founded Epoch AI, then in April 2025 Mechanize, to build “virtual work environments, benchmarks, and training data to enable the full automation of all work” (his post).
• Adam Gleave, chief executive of the evaluator FAR.AI, sits on METR’s board (METR): one evaluator’s head helps govern another.
• Geoffrey Irving led safety teams at OpenAI and DeepMind, became chief scientist of the UK institute that tests their models, and left in 2026 to co-found Sequent Research with researchers from that institute’s Alignment Team (his post).
• Alexandr Wang founded Scale AI, the US institute’s first authorized outside evaluator; in June 2025 Meta took a minority stake and Wang became Meta’s Chief AI Officer while remaining a Scale director (Scale AI).
• Matt Fredrikson, Gray Swan’s chief executive, co-wrote the 2023 universal-jailbreak paper with Kolter (arXiv 2307.15043); the company now sells evaluations and the filters against them.
• Jacob Steinhardt of Transluce “worked at OpenAI and Open Philanthropy” as a post-doc (his page); Buck Shlegeris runs Redwood, which advises Google DeepMind and Anthropic; Dan Lahav runs Irregular; Dan Guido runs Trail of Bits.

Each relationship is lawful and most were disclosed. The structural point is that there is almost no one left to evaluate the frontier who is not connected to it, and that METR’s footnotes, not any rule, are what put two of these connections on the record.

Consolidation

• Scale / Meta. Scale became the US AI Safety Institute’s authorized third-party evaluator in February 2025 (Scale AI). In June Meta invested at a valuation over $29 billion, reported by AP as $14.3 billion for 49 percent, and hired its founder to run Meta Superintelligence Labs (Scale AI; AP via WJXT). Reports followed that OpenAI and Google were ending work with Scale; OpenAI denied the reason was Meta’s access to its secrets (TechRepublic). None of these sources carries a public statement on how the government evaluation arrangement is handled now that a lab owns 49 percent.
• Epoch / OpenAI. OpenAI funded FrontierMath; Epoch “made a mistake” in not telling contributors, by its co-founder’s account, and the protection against training on the test was a “verbal agreement” (TechCrunch).
• Trail of Bits / OpenAI. After working with OpenAI since June 2026 on Patch the Planet, Trail of Bits is in “preliminary discussions” to be acquired by OpenAI, The Information reported on October 13, 2026; no deal has been announced (PYMNTS; Yahoo Finance).

The case for the arrangement

The evaluators’ answer, in their own terms: pre-release access requires trust and a contract, so evaluators will always be close to the labs; the alternative is no outside testing at all. METR says it “has not accepted funding from AI companies” and that “pre-deployment capability testing is not a sufficient risk management strategy by itself” (METR). Disclosure is the safeguard, and in METR’s case it worked: the relationships above are known because the evaluator published them. What would settle the question is a rule rather than a footnote: public conflict-of-interest standards for pre-deployment evaluators, and independent replication of the results the labs cite.

Watching it

Every organization and person above has a profile, so each is in the weekly subject watch; a news item naming one of them with “acquired,” “hired,” “appointed,” “resigned” or “sued” is flagged first. Their relationships are plotted in the revolving-door graph.

Related

• Watching the Watchers · AI Agent Incidents · AI and Bioweapons Uplift · The EA to AI-Safety Funding Pipeline
• The Escape Registry · AI Governance Tracker · Revolving Door