Research: The Evaluator Alliances — Who Tests Whom, Who Pays, and Who Gains From Which Rule

Companion pages: The AI Evaluators (who they are, who funds them, who is buying them) · Watching the Watchers · AI Agent Incidents · the revolving-door graph, where the people and organizations below are plotted.

Whether a frontier model ships, and on what terms, now rests on a small set of outside evaluators. The labs cite them in their system cards, the governments contract with them, and the bills in front of Congress and the states would make some of them mandatory. This page is the running map of those ties, built from the parties’ own documents: what each lab says about who tested its models, what each government paid or signed, what each faction asks legislators for, and what that rule would pay the faction that asks for it.

Holding an interest in a rule does not make the argument for it wrong, and nothing here says it does. Each faction’s own case is set beside its interest. The finding is structural: the parties who look like opposite ends of the arrangement, the tested and the testers, keep arguing for the same rules.

What moved

Dated events from the parties’ own documents, newest first. Each is sourced in the sections below.

• October 13, 2026. Google’s Gemini 4 Argon safety report names its outside testers for the first time in the documents read for this page: Gray Swan and Trajectory Labs as red teams, Faculty for biological and chemical risk, the UK AI Security Institute testing both a mitigated and an unmitigated model, and NIST’s renamed institute.
• September 30, 2026. METR’s president and Apollo Research’s chief executive testify to a Senate Homeland Security subcommittee, “Rogue AI: Securing the Homeland Against AI Agent Attacks.” Apollo asks Congress to require embedded evaluations and monitoring; METR says it “does not take policy positions.”
• September 2026. Anthropic’s Opus 5.5 system card carries a section on testing by the US institute, and its Fable 5.1 and Mythos 5.1 card names no government institute as a tester. Its July Opus 5 card had carried UK AISI testing sections. Anthropic’s chief executive proposes embedded third-party evaluators, “such as METR,” and asks governments to require them of other labs.
• September 9, 2026. Paul Christiano, a senior technical adviser at the US institute, joins the OpenAI Foundation board and its Safety and Security Committee, “working alongside its chair, Zico Kolter,” co-founder of the evaluator Gray Swan. The same day OpenAI asks Congress for “mandatory, capability-based national regulation.”
• The US institute has a new name. Google’s report and Anthropic’s Haiku 5.5 card both call it the “Center for Advancing Innovation and Standards for Super Intelligence (CAISSI).” Its profile here is US CAISI.
• July 24, 2026. An open-weights letter on Nvidia’s site lists Meta, OpenAI, Google and Scale AI among its signatories.
• May 5, 2026. The US institute’s testing agreements extend to Google DeepMind, Microsoft and xAI (reported).

Who tests whom: what the labs say in their own system cards

“Tested” means the lab’s document says the outside party evaluated or red-teamed the model. “Built a test” means the lab ran a benchmark or task set the party built. Dates are the documents’ own; Anthropic lists its cards by month.

What the documents say, in the labs’ words:

• OpenAI names the widest set. “Apollo evaluated GPT-5.6 Sol for strategic deception, scheming, and sabotage” (OpenAI). Its later updates name no outside evaluator: the GPT-6.1 Sol addendum, the October GPT-6 update and the August GPT-5.6 update (OpenAI; OpenAI; OpenAI). Epoch’s own account: “OpenAI commissioned Epoch AI to produce 300 advanced math problems for AI evaluation that form the core of the FrontierMath benchmark” (Epoch AI).
• Anthropic pays named red teams by the hour: “Trajectory Labs, PBC spent roughly 95 hours red teaming Claude Opus 5.5” and “10a Labs spent roughly 56 hours red teaming Claude Opus 5.5” (Anthropic). With one evaluator it shares a prize fund: “GraySwan and Anthropic are running an ongoing bug bounty in which we reward red-teamers for developing jailbreaks for our safeguards” (Anthropic). Its Opus 5 card in July carried UK AISI testing (Anthropic); its September Fable 5.1 and Mythos 5.1 card names METR as a tester and no government institute (Anthropic).
• Google DeepMind named no outside evaluator in its Gemini 3 Pro report: “External evaluators found Gemini 3 Pro exhibits a substantial propensity for strategic deception in certain limited circumstances” (Google DeepMind). For Gemini 4 Argon it names them: “we partnered with external red-teaming organizations including Gray Swan and Trajectory Labs”; “We worked with Faculty Science Limited (Faculty) to conduct an independent evaluation of biological and chemical risks”; “AISI conducted evaluations of Gemini 4 Argon, having been given both a version of the model without our full set of safety mitigations and a version which had mitigations applied” (Google DeepMind).
• Meta publishes its evaluator’s least flattering finding: “Independent third-party testing by Apollo Research found Muse Spark has the highest rate of evaluation awareness they have observed to date.” Its chemical and biological evaluations were designed with contractors: “Those engaged in this process included Deloitte, Faculty, Frontier Design, and SecureBio” (Meta). No government institute is named as a tester of Muse Spark.
• xAI names one evaluator, for biology: “Today, LatchBio published an independent analysis of Grok 4.6” (xAI). Its cyber testers are unnamed: “We additionally provided an unrestricted configuration of Grok 4.6 to third-party evaluators, who corroborated the results of our internal evaluations and testing on cyber capabilities” (xAI).

Who pays the evaluators, and who sits where

Government contracts. In January 2024 the UK Department for Science, Innovation and Technology bought evaluation work for the AI Safety Institute directly from outside firms:

Two years later three of the four are named in the labs’ system cards above, and Trail of Bits is reported to be in acquisition talks with OpenAI (The AI Evaluators).

The UK institute’s Alignment Project. Its advisory board includes Buck Shlegeris, “CEO and founder of Redwood Research,” and Zico Kolter, listed by his Carnegie Mellon post. Its funders include OpenAI, Anthropic, Microsoft, AWS and Schmidt Sciences: “The Alignment Project is supported by an international coalition of government, industry, and philanthropic funders” (Alignment Project).

The labs’ own grant fund. The Frontier Model Forum’s AI Safety Fund is backed by its founding members, “including Anthropic, Google, Microsoft, and OpenAI,” with “the David and Lucile Packard Foundation, Schmidt Sciences, and Jaan Tallinn” (Frontier Model Forum). Its December 2025 cohort paid Apollo Research (“Building black box scheming monitors for Frontier AI agents”), FAR.AI and SecureBio (Frontier Model Forum).

METR’s money. METR says it “has not accepted funding from AI companies, though we make use of significant free tokens,” and lists the UK AI Security Institute among its donors with Schmidt Sciences and Packard (METR). Its president told the Senate: “METR is not paid or funded by them,” and “AI companies provide METR with access and tokens, which we use for evaluations, research, and engineering” (testimony). FAR.AI’s chief executive Adam Gleave is listed on METR’s page as “Advisor and Board Member.”

Transluce’s money. Its policy page: “Transluce received 6% of its revenue from the personal holdings of OpenAI employees” and “32% of its revenue from the personal holdings of Anthropic employees”; “It received no revenue directly from AI developers as organizations” (Transluce). Transluce ran the pre-release audit of OpenAI’s o3.

Epoch AI’s money. Its transparency page lists Coefficient Giving (formerly Open Philanthropy) as its largest donor and names its paying partners: OpenAI, Google DeepMind, xAI, Anthropic, the UK Department for Science, Innovation and Technology, the EU AI Office and METR. Its pledge: “We are committed to disclosing any research sponsorship and data access agreements with industry” (Epoch AI). Its chief executive Jaime Sevilla on the benchmark OpenAI commissioned: “a dataset only us and OpenAI have access to,” and in April 2026, “OpenAI purchased access to the verifiers” (his post; his post).

SecureBio’s money. Its assessment of GPT-6 Astra discloses: “SecureBio received compensation from OpenAI limited to covering the costs of this assessment,” and “We are primarily funded by Coefficient Giving, the Survival and Flourishing Fund, and Sentinel Bio” (SecureBio). Its conflicts policy caps the lab side: “no more than 25% of its annual revenue is derived from services supplied to AI companies” (SecureBio).

OpenAI’s own oversight. Paul Christiano “will also join the Safety and Security Committee (SSC) of the Foundation Board, working alongside its chair, Zico Kolter. The SSC provides governance over safety and security practices across all of OpenAI.” OpenAI’s note on his government post: “As a Senior Technical Advisor, Paul will recuse himself from all OpenAI-related matters as well as all model evaluations” (OpenAI Foundation). Kolter co-founded Gray Swan, which tests OpenAI’s models.

Advising government. Palisade Research says Jeffrey Ladish “helped advise the White House, Department of Defense, and congressional offices on risks from AI,” and that it has “briefed members of Congress in the House and the Senate” (Palisade).

The US institute’s agreements. It signed testing agreements with Google DeepMind, Microsoft and xAI in May 2026 and reports “more than 40 evaluations to date,” according to trade press relaying NIST’s release (ExecutiveGov); NIST’s own release is queued for capture.

Who gains from which rule

Each row: the position in the party’s own words, what adoption would pay the party where a document shows it, and the party’s own best case.

1. Mandatory third-party evaluation: the labs and the evaluators ask for the same rule.

• Anthropic proposes that “Each frontier AI company commits to giving ongoing, employee-like access to a team of embedded third-party evaluators (such as METR),” a step “Anthropic is unilaterally committing to (and calls on governments to require other frontier companies to match).” The rule would oblige its competitors to accept what it has already accepted. Its case: embedded evaluators “can check at the level of nuts and bolts whether an AI company is actually following the training, deployment, operational, and safeguards practices they claim to be following” (Dario Amodei).
• OpenAI endorses California’s SB 813 and says “we prefer independent technical assessments to be required at the federal level” (OpenAI).
• Apollo Research asks Congress: “First, require the adoption of embedded evaluations” (testimony), and says of the evaluations it proposes, “We, at Apollo Research, are intending to conduct 3rd party Training-Run Evaluations in the future” (Apollo). Its case: “Embedded evaluations run in parallel with development and stand to identify and mitigate risks to the American public without affecting AI development.”
• Scale AI asks Washington to “Strengthen the government’s ability to evaluate frontier models before deployment,” and notes, “We have partnered with government AI evaluation bodies in the United States, the United Kingdom, Korea, and Singapore.” Its case: “Washington cannot let the labs grade their own homework” (Scale AI).
• METR is the evaluator Anthropic names, and it takes no side: “METR does not take policy positions,” while calling “better public visibility into the capabilities of frontier AI agents” valuable “under almost any policy choice” (testimony).

The lab that would be tested and the evaluator that would be hired argue for the same mandate. The lab pays under it, the evaluator is paid under it, and each describes it as a rule for the other labs.

2. Required monitoring. OpenAI: “developers should be required to monitor for these misaligned behaviors and demonstrate that effective safeguards are in place,” after “For Astra, we also introduced universal monitoring of full trajectories, including chains of thought” (OpenAI). Apollo: “Second, require the implementation of additional layers of defense such as monitoring and control”; Apollo is “building Watcher, a monitoring tool for coding agents” (Apollo). The rule matches what the lab says it already does and what the evaluator sells. Apollo’s case: “They are not a brake on innovation.”

3. Rules that bind rivals to existing practice. Anthropic on SB 53: “These requirements would formalize practices that Anthropic and many other frontier AI companies already follow,” and the bill exempts “startups and smaller companies that are less likely to develop powerful models.” Its case: “Without it, labs with increasingly powerful models could face growing incentives to dial back their own safety and disclosure programs in order to compete” (Anthropic). OpenAI’s version: requirements “should apply to the handful of well-resourced laboratories developing the most capable systems—not to startups, small developers, or researchers operating nowhere near the frontier”; its case, “Obligations should be proportionate to capabilities and risks” (OpenAI).

4. One national rulebook. OpenAI wants states to “create a de facto national baseline that Congress can ultimately codify—an approach we call reverse federalism” (OpenAI). Google asked the government to “preempt a chaotic patchwork of state-level rules on frontier AI development” (Google). Anthropic: “frontier AI safety is best addressed at the federal level instead of a patchwork of state regulations.” None of the three argues against a single national rule in the documents read.

5. Coordination among the leaders. Anthropic asks for a government antitrust waiver so the frontier companies can discuss limits on the pace of development: “Some forms of coordination that would be impactful for pacing are legally challenging, and will require government support.” Its case: “A race to the bottom, spurred by commercial incentives, can make these risks more acute” (Dario Amodei).

6. Open weights. Meta: “selling access to AI models isn’t our business model. That means openly releasing Llama doesn’t undercut our revenue”; its case, open source means “that power isn’t concentrated in the hands of a small number of companies” (Meta). The July 2026 letter signed by Meta, OpenAI, Google and Scale asks to avoid “premature restrictions on open models that stifle competition or drive innovation overseas” (letter). OpenAI: “Nor should frontier safety policy become open-weights policy by another name.” Anthropic, which is not on the letter, asks to “Crack down on unauthorized distillation by companies in authoritarian countries” (Dario Amodei).

7. The EU’s voluntary code. The Commission says signing “will reduce their administrative burden and give them more legal certainty”; OpenAI, Anthropic and Google signed, xAI signed only the safety and security chapter (European Commission), and Meta declined: “Europe is heading down the wrong path on AI” (The Register).

Overlapping seats

The people and organizations above who hold seats on more than one side of the arrangement at the same time. Each cell is a documented tie; an empty cell means none was documented in this pass, not that none exists.

The case for the arrangement

The evaluators’ answer, in their own terms: testing a model before release needs the lab’s cooperation, so the tester will always be close to the tested, and disclosure is the safeguard. METR discloses its funding and footnotes its conflicts; OpenAI publishes Christiano’s recusal; Transluce publishes its revenue split; Meta prints Apollo’s finding against its own model. Scale’s version is the strongest: the alternative to outside evaluators is that the labs “grade their own homework.” The question this page tracks is what happens when the rule that hires the evaluators is written with the evaluators and the labs both asking for it.

Related

• The AI Evaluators
• Watching the Watchers
• AI Agent Incidents
• Three-Axis Model