STANFORD CENTER FOR RESEARCH ON FOUNDATION MODELS (CRFM)

Institutional Archetype

THE ACADEMIC SCOREKEEPER — The Center for Research on Foundation Models is a university unit that builds the rulers other people use on frontier models. Its benchmark Cybench was the cyber test in the US institute’s published pre-deployment evaluations of late 2024. Its Holistic Evaluation of Language Models (HELM) framework entered maintenance mode on 1 June 2026. Its Foundation Model Transparency Index scores the companies themselves on what they disclose, and the 2025 edition gave IBM, a listed CRFM supporter, the highest score in its history. The file is open on a structure, not a fault: the grader is a university center whose published supporter list includes companies it scores and labs whose models it evaluates, and whose own pages name those relationships without attaching an amount to any of them.

Mandate & Origin

“The Center for Research on Foundation Models (CRFM) is an interdisciplinary initiative at the Stanford Institute for Human-Centered Artificial Intelligence (HAI) that makes fundamental advances in the study, development, and deployment of foundation models” (CRFM). Its home page lists four lines of work: technical foundations, beneficial applications, societal impact and “AI policy. We support governments in formulating and implementing evidence-based policy across multiple jurisdictions” (CRFM). The page says the center “brings together a multidisciplinary group of faculty, students, researchers, and engineers spanning 10+ departments at Stanford.” Its people page lists Percy Liang first, with the title “Director, Computer Science,” then 37 further faculty from fields including law, medicine, linguistics and political science, and a research engineering team of Abhinav Garg, David Hall and Yifan Mai (CRFM, People). Liang’s own page gives his post as “Professor of Computer Science (and courtesy in Statistics)” (Percy Liang).

The center’s GitHub organization was created on 26 November 2020 and holds 20 public repositories; the HELM repository dates from 29 November 2021, is licensed Apache-2.0 and carries about 2,900 stars (GitHub, stanford-crfm; GitHub, stanford-crfm/helm). The site’s footer reads “© 2021–2025.”

The people who produce its measurement work are named on its own pages. The Foundation Model Transparency Index authors are Alexander Wan, Kevin Klyman, Sayash Kapoor, Nestor Maslej, Shayne Longpre, Betty Xiong, Percy Liang and Rishi Bommasani (CRFM, FMTI); Liang’s page describes Bommasani as “now senior research scholar at Stanford HAI” (Percy Liang). Cybench’s author list on CRFM’s own post names 24 authors and then “Dan Boneh and Daniel E. Ho and Percy Liang” (CRFM, Cybench).

Funding & Backers

CRFM’s own support page is the only first-hand list of who pays. It reads: “Support our research on foundation models. To learn more about how you can support CRFM, please contact us at [email protected]. Corporations can support the Center through the HAI Membership program.” Under “Supporters” it names Google, McGovern Foundation, Schmidt Futures, Stanford Institute for Human-Centered AI (HAI) and “Stanford HAI Industrial Affiliates”: Accenture, IBM, SAP, SCBX, Wells Fargo and Arabic.AI. Under “Additional Thanks To” it lists in-kind support: API credits from AI21 Labs, Aleph Alpha, Anthropic, Cohere and OpenAI; API credits and compute from Microsoft; compute from Google and Together (CRFM, Support).

• What the page does not say. It gives no amount, year or term for any supporter, and does not say which supporter funds which project.
• The affiliate program’s price list. Stanford HAI’s industry page prices its tiers at “Strategic $1M / year,” “Impact $350K / year” and “Ecosystem $75K / year”; the Strategic tier carries “Wallet = $550K” for “faculty-specific research projects aligned with member’s priorities (research tokens, FMAs, or visiting scholar).” The same page says “Financial support and real-world insights from industry members help ensure that the institute has the resources needed to pursue innovative, interdisciplinary research,” and that “The site presentations and all information, data, and results from such visitation interactions will be shared with all members and the public” (Stanford HAI, Industry). A separate HAI Founding Members sheet describes “A $5M annual membership” that gives access to “a $1,155,000” wallet over three years and carries a quotation from Google’s Jeff Dean (Stanford HAI, Founding Members Program). Neither page says which tier any named supporter holds or that any HAI affiliate fee reaches CRFM.
• Philanthropy beside it. Percy Liang’s page lists a “Schmidt Sciences AI2050 Senior Fellowship (2022)” (Percy Liang). The open-model project Marin, which Liang and CRFM engineer David Hall created at Stanford in 2024 and which is now “primarily developed by Open Athena,” states: “This work is generously supported by the Google TPU Research Cloud program for TPUs, the Huang Foundation for GPUs, the Siegel Family Foundation, Fulcrum Science, Schmidt Sciences, and many others” (Marin). See The EA to AI-Safety Funding Pipeline for the wider funding map.
• Government money. The support page lists none; no grant to CRFM is named there.

Institutional Voice & Intent

CRFM on its own purpose: “Foundation models present novel capabilities and risks that are poorly understood. Nonetheless, they are being deployed at scale” (CRFM). On HELM, the README: an “open source Python framework created by the Center for Research on Foundation Models (CRFM) at Stanford for holistic, reproducible and transparent evaluation of foundation models” (GitHub, stanford-crfm/helm). On the Index, its page: “Since 2023, the Foundation Model Transparency Index has scored major foundation developers like Google and OpenAI for their transparency on 100 transparency indicators” (CRFM, FMTI). On the 2025 paper’s own conclusion: “As policymakers around the world increasingly mandate certain types of transparency, this work reveals the current state of transparency for foundation model developers, how it may change given newly enacted policy, and where more aggressive policy interventions are necessary to address critical information deficits” (arXiv 2512.10169). On its own tooling, the maintenance policy: “HELM entered maintenance mode on June 1, 2026” and “no new features will be added to HELM, and no new evaluations will be added to the HELM leaderboards” (HELM docs).

Stated intent: Study and build foundation models, make evaluation “holistic, reproducible and transparent,” measure what companies disclose, and “support governments in formulating and implementing evidence-based policy.”

Observed intent: Supply the measurement artifacts that institutes, labs and policymakers then use, and move on. Cybench went into the US institute’s test reports and the UK institute’s Inspect Evals; HELM’s leaderboards went out under the center’s name until June 2026; the Index puts a score on named companies each year; and the center’s people sit in the EU’s General-Purpose AI Code of Practice drafting group and on a White House advisory committee (see Position in the Apparatus). Its HELM maintenance page now lists among its alternatives a framework built by another institution: “Inspect AI Evals” (HELM docs).

Gap: The center states what it measures and who supports it by name. What the pages do not settle is how much each supporter gives and whether any supporter pays for a thing it is then scored on: the HELM Arabic leaderboards were “produced in collaboration with Arabic.AI,” a listed affiliate, and Arabic.AI’s model came first on both; the 2025 Index scores Google and IBM, both on the support page. The record shows a disclosed overlap and no changed number. For a scorekeeper, a supporter list without amounts is a disclosure that stops short of the figure a reader would need to weigh it.

Interest: CRFM argues for mandated transparency, for legal protection of independent evaluators, and for open foundation models. What each position would pay is set out from its own pages; no fee, contract or grant value is published for any of them.

• Mandated disclosure, scored by the Index. Position, the 2025 paper: “where more aggressive policy interventions are necessary to address critical information deficits” (arXiv 2512.10169). What it pays: a disclosure mandate is written in indicators like the Index’s 100, and the Index is the instrument that counts them; its vice-chair post in the EU Code’s transparency working group is held by Bommasani, then of CRFM (European Commission, chairs). No fee for scoring is published. Best case, the paper’s: “Companies are most opaque about their training data and training compute as well as the post-deployment usage and impact of their flagship models.” Who else gains: the paper finds “enterprise-focused companies” “consistently and considerably more transparent,” with IBM first at 95; a rule that rewards disclosure rewards the companies already disclosing, and IBM is on CRFM’s supporter list (CRFM, FMTI; CRFM, Support).
• Legal protection for outside evaluators. Position, CRFM’s blog post: “Today, we are calling for AI developers to invest in the needs of third-party, independent researchers, who investigate flaws in AI systems,” proposing “legal language developers can adopt to protect good-faith research” and “a centralized organization to coordinate and triage flaws” (CRFM, General-Purpose AI Needs Coordinated Flaw Reporting). What it pays: the center’s own benchmark teams and the evaluators that run its tools are the researchers the safe harbor would cover; no value is published. Best case, the post’s: “Independent AI evaluation lacks legal protections, and when flaws are found there is no infrastructure to responsibly distribute findings.” Who else gains: every independent evaluator on the AI evaluators map, among them Transluce, on whose advisory list Liang sits.
• Open foundation models. Position, Liang on his own page: “We practice a radical form of openness called open development, which goes beyond open-weight and open-source” (Percy Liang); CRFM’s policy page carries “Considerations for Governing Open Foundation Models” and two responses to the US NTIA’s 2024 request for comment on open foundation models (CRFM, Policy). What it pays: a rule that leaves open weights lawful protects the Marin and Levanter lines the center and its engineers build, and Marin’s stated supporters include Google’s TPU program and Schmidt Sciences. Best case, Liang’s own: “my goal is to enable everyone to understand, shape, and contribute to foundation model development.” Who else gains: other open-weight developers, among them IBM, Meta, Alibaba and DeepSeek, which the 2025 Index scores as open-weight flagship developers (arXiv 2512.10169).

Interest is a documented fact about each position, not a claim about motive, and not a verdict that the argument is wrong. Each best case is stated at the strength its author states it.

Position in the Apparatus

• Cybench in the US and UK government tests. The US institute’s report on the December 2024 joint test: “US AISI tested o1 on Cybench, a publicly available cybersecurity benchmark for language models developed by researchers at Stanford University,” having modified “the benchmark to use agents based on the Inspect agent framework” (NIST, US AISI and UK AISI joint pre-deployment test of OpenAI o1). Its November 2024 report on the upgraded Claude 3.5 Sonnet: “US AISI tested Sonnet 3.5 (new) on Cybench” (NIST, Upgraded Sonnet). CAISI’s September 2025 evaluation of DeepSeek models: “CAISI evaluated DeepSeek models and reference models on Cybench, a publicly available cyber benchmark developed by researchers at Stanford University” (NIST CAISI, Evaluation of DeepSeek AI Models). The UK institute’s Inspect Evals repository carries it: “Cybench is a benchmark for evaluating models’ cybersecurity capabilities” (UK AISI, inspect_evals). See US CAISI, UK AI Security Institute and Meridian Labs, which maintains Inspect with the UK institute.
• Cybench’s own account of who uses it. Its page: “The US AISI and UK AISI leveraged Cybench as the only open source cybersecurity benchmark on their Joint Pre-Deployment Test on Anthropic’s Claude 3.5 Sonnet and Joint Pre-Deployment Test on OpenAI o1”; “The UK AISI has incorporated Cybench in its Inspect Evals framework”; “xAI featured Cybench in its xAI Risk Management Framework”; “Meta featured Cybench in its Muse Spark Safety and Preparedness Report”; “Amazon featured Cybench in its evaluation suite for the Amazon Nova Premier Model Card”; “The Japan AISI and the Republic of Korea AISI leveraged Cybench”; and “The Center for AI Safety selected Cybench as a First Prize winning benchmark in the SafeBench competition” (Cybench). It also lists Anthropic’s Claude 3.7 Sonnet through Opus 4.7 system cards. The Japanese and Korean institute uses and the other lab-document uses are Cybench’s statement. See Anthropic and Center for AI Safety.
• The ruler saturated, and one port leaked. Anthropic’s Fable 5 and Mythos 5 system card: “we have not run Cybench, as it is largely `saturated’: it no longer captures changes in model capabilities” (Anthropic, system card). Transluce’s Docent write-up found, in a January 2025 Inspect fork of Cybench, a task where “the flag was accidentally exposed in the Dockerfile,” a task it reports, after correspondence with the Cybench authors, was exposed when “Cybench was ported to Inspect in November 2024,” adding that “The Cybench main repo does not contain the exposed flag” (Transluce, Introducing Docent). Cybench’s own page carries a related note: the scores of OpenAI’s o3-mini and o1-mini “are inflated because HAL likely ran on a fork of the Inspect framework that leaked the answer to a task” (Cybench).
• HELM, its leaderboards and the maintenance mode. The repository: “HELM entered maintenance mode on June 1, 2026,” with the policy that it “will continue to be maintained by volunteer maintainers on a best-effort basis,” that “the maintainers are no longer able to actively support research collaborations due to the lack of bandwidth,” and a list of alternatives: Evalchemy, Inspect AI Evals, Lighteval, LLM Evaluation Harness and Unitxt (GitHub, stanford-crfm/helm; HELM docs). Its flagship leaderboards at the time were HELM Capabilities, HELM Safety and VHELM; the last commits to the repository added the Arabic Enterprise metrics on 4 and 5 June 2026 (GitHub, stanford-crfm/helm).
• Two leaderboards made with a supporter. HELM Arabic (18 December 2025): “This leaderboard was produced in collaboration with Arabic.AI,” and “Arabic.AI LLM-X achieved the highest mean score.” HELM Arabic Enterprise (26 May 2026): “The leaderboard was developed in collaboration with Arabic.AI,” and “Arabic.AI LLM-X scored the highest mean score (0.826) of all models” (CRFM, HELM Arabic; CRFM, HELM Arabic Enterprise). The Enterprise post says “All model requests, responses, prompts, metrics, and scores are made available for inspection”; the CRFM support page lists Arabic.AI among “Stanford HAI Industrial Affiliates” (CRFM, Support).
• The Index. The 2025 edition scores 13 companies; “The mean score this year is 41: this is a 17 point decrease over the previous year”; “IBM scores a 95”; “The other two companies (xAI and Midjourney) tie for the lowest score at 14”; and “30% of the companies we contacted agreed to submit transparency reports in 2025, dropping from 74% in 2024” (CRFM, FMTI). The paper adds that the five Frontier Model Forum members it scores “end up in the middle of the Index” (arXiv 2512.10169). The Index’s advisory board is Arvind Narayanan, Daniel E. Ho, Danielle Allen, Daron Acemoglu and Rumman Chowdhury. See Frontier Model Forum and OpenAI.
• Seats in government drafting rooms. The European Commission’s chairs page lists Rishi Bommasani as “Vice Chair (Transparency)” in the first General-Purpose AI Code of Practice and describes him as “the Society Lead at the Stanford Center for Research on Models” (European Commission, chairs). The Index board’s biography of Ho, a CRFM faculty member, says he “serves on the National Artificial Intelligence Advisory Committee (NAIAC), advising the White House on AI policy, as senior advisor on Responsible AI at the U.S. Department of Labor” (CRFM, FMTI); Stanford HAI hosts his testimony of 16 May 2023 to the Senate Committee on Homeland Security and Governmental Affairs, given as director of the Regulation, Evaluation, and Governance Lab (Stanford HAI). See EU AI Office and GovAI, whose Markus Anderljung was vice-chair of a neighboring working group on the same chairs page.
• Liang and Transluce. Transluce lists Liang among its advisers (“Percy Liang, Associate Professor, Stanford University”) and says it has been “publicly endorsed by leading researchers from across the field including Ethan Perez (Anthropic), and Percy Liang (Stanford)” (Transluce, About; Transluce, 2025 fundraiser). Transluce’s Docent work on Cybench is the third-party correction recorded above.

Actions & Leadership Choices

CONDUCT: WATCH — SUPPORTER LIST OVERLAPS THE GRADED. No wrongdoing is alleged. CRFM publishes tools with open code, posts the requests and responses behind its leaderboards, discloses its supporters by name and has publicly retired a flagship project rather than let it lapse unannounced. The structural point is the one every university-housed evaluator shares: the supporter list includes companies that the center’s work scores and labs whose models it runs, the page gives no amounts, and at least two leaderboards were built with a listed affiliate whose model topped them. The record shows an overlap that was disclosed and no number that changed.

• 26 November 2020. The stanford-crfm GitHub organization is created (GitHub).
• 29 November 2021. The HELM repository is created (GitHub).
• October 2023 and May 2024. First and second editions of the Foundation Model Transparency Index (CRFM, FMTI).
• 19 August 2024. CRFM publishes Cybench (CRFM, Cybench).
• November and December 2024. The US institute runs Cybench in its Claude 3.5 Sonnet and o1 evaluations (NIST).
• 13 March 2025. CRFM publishes its coordinated flaw-reporting proposal (CRFM).
• September 2025. CAISI’s DeepSeek evaluation runs Cybench (NIST CAISI).
• 11 December 2025. The 2025 Foundation Model Transparency Index is posted to arXiv (arXiv 2512.10169).
• 18 December 2025. HELM Arabic, with Arabic.AI (CRFM).
• 26 May 2026. HELM Arabic Enterprise, with Arabic.AI (CRFM).
• 1 June 2026. HELM enters maintenance mode (HELM docs).
• 9 June 2026. Anthropic’s Fable 5 and Mythos 5 card says it did not run Cybench (Anthropic).

Alliance Map

Where ties on different axes run at the same time, the overlap is the finding. The center’s Cybench is the benchmark the US institutes ran, its Index scores the companies on its supporter list, its Arabic leaderboards were built with a supporter that came first on them, its Index author sits in the EU’s Code drafting group, and its director advises the evaluator that found the flag exposed in Cybench’s Inspect port. Each tie is disclosed on a page the party controls; they show an arrangement, not coordination.

Reach Assessment

Institutional: Cybench is in the US and UK institutes’ test pipelines and Inspect Evals, and cited on its own page by four lab or institute documents beyond them; the Index scores the named developers each year; HELM’s leaderboards stay online but add nothing. Memetic: Moderate: HELM’s maintenance page now points readers to other frameworks; the Index’s yearly headline (this year, IBM 95, xAI and Midjourney 14) reaches press and policy readers. Civilizational: If evaluation is the layer that tells governments and labs what a model does, a university center that writes the rulers and scores the disclosures sits upstream of some of what they say. Nothing in the record says CRFM has used that position for anything but publication, and its code, requests and responses are open, so a reader can recompute a score. The gap is the unattached amount: the supporters are named and none is priced.

Score basis. Atk 7 is reach: Cybench was the cyber benchmark in the US institute’s published tests of Claude 3.5 Sonnet and o1 and in CAISI’s DeepSeek evaluation, the UK institute carries it in Inspect Evals, and its own page lists lab documents from Anthropic, xAI, Meta and Amazon (NIST; UK AISI, inspect_evals; Cybench). The points under 8 are that no statute or procurement rule names it, that Anthropic’s June 2026 card says Cybench is “largely `saturated’” and was not run, and that HELM no longer adds evaluations (Anthropic, system card; HELM docs). Peers on atk 7: SecureBio and GovAI; Meridian Labs and Gray Swan hold 8, US CAISI, the UK AI Security Institute and METR hold 9. Def 6 is funding insulation: CRFM names its supporters, among them Google, six corporate affiliates and in-kind credits and compute from labs it evaluates, and HAI prices its corporate tiers at $75,000 to $1 million a year; the points off are that no amount is stated and that two HELM Arabic leaderboards were built with a listed affiliate whose model ranked first (CRFM, Support; Stanford HAI, Industry; CRFM, HELM Arabic Enterprise). Peers on def 6: Hugging Face, Gray Swan and FAR.AI; Epoch AI holds 7 because it publishes the amount of every donation of $70,000 or more. Hp 6 is entrenchment: CRFM sits inside Stanford HAI with 38 listed faculty, its Cybench port is in Inspect Evals and its Index has run three editions (CRFM, People; CRFM, FMTI). The points off are that HELM entered maintenance mode on 1 June 2026, that Cybench is saturated by Anthropic’s account and that no statute installs either (HELM docs). Peers on hp 6: FAR.AI, Palisade Research and Transluce; Epoch AI and Meridian Labs hold 7.

Sources: Stanford CRFM; CRFM — People; CRFM — Support; CRFM — Policy; CRFM — Foundation Model Transparency Index 2025; The 2025 Foundation Model Transparency Index — arXiv; CRFM — Cybench; Cybench; CRFM — General-Purpose AI Needs Coordinated Flaw Reporting; CRFM — HELM Arabic; CRFM — HELM Arabic Enterprise; HELM — GitHub; HELM — Maintenance Mode Policy; stanford-crfm — GitHub; Stanford HAI — Industry; Stanford HAI — Founding Members Program; Stanford HAI — Daniel E. Ho testimony; Percy Liang; Marin; Transluce — About; Transluce — 2025 fundraiser; Transluce — Introducing Docent; NIST — US AISI and UK AISI joint pre-deployment test of OpenAI o1; NIST — Upgraded Sonnet; Evaluation of DeepSeek AI Models — NIST CAISI; Cybench — UK AISI inspect_evals; System Card: Claude Fable 5 and Claude Mythos 5 — Anthropic; Meet the chairs — European Commission.