PRINCETON HOLISTIC AGENT LEADERBOARD

OLYMPUS opened an institutional file on Princeton’s Holistic Agent Leaderboard because it is the Forum member that most resembles a scoreboard, and a scoreboard is read by the people it scores. HAL runs the labs’ models through nine agent benchmarks, publishes the accuracy and the dollar cost, and says its evaluations are third-party. The same page thanks two of those labs for the API credits that paid for the runs. The project’s lead authors also wrote the book AI Snake Oil and the newsletter AI as Normal Technology, and one of them, Arvind Narayanan, directs the Princeton center that announced HAL’s Forum membership. This is not a psychometric profile. It is a funder list, an author list and a credits line. The finding is the shape: a university project whose reach is the reach of a reference, funded by the philanthropy behind the evaluator field, scoring models it is given access to by their makers. Not a conflict. A dependency, disclosed.

Institutional Archetype

THE UNIVERSITY’S AGENT RULER — HAL describes itself as “The standardized, cost-aware, and third-party leaderboard for evaluating agents,” made “By the SAgE team at Princeton University” (HAL). It has two parts: “HAL Leaderboards,” a “central platform for assessing agent capabilities” with “cost-controlled evaluations as default,” and “HAL Harness,” a “standalone evaluation harness for various benchmarks.” Its argument is that single-number leaderboards hide cost: “Agents can be 100x more expensive while only being 1% better.” The throughline is that agent evaluation was ad hoc and that a university could set the common harness. This profile scores the leaderboard, not its authors.

Mandate & Origin

• The paper. “Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation,” arXiv:2510.11977, submitted 13 October 2025, 31 authors, accepted to the Fourteenth International Conference on Learning Representations (ICLR 2026). The abstract: “We introduce the Holistic Agent Leaderboard (HAL) to address these challenges,” and “We validate the harness by conducting 21,730 agent rollouts across 9 models and 9 benchmarks in coding, web navigation, science, and customer service with a total cost of about $40,000” (arXiv; HAL paper PDF). Contact for the paper: Kapoor and Narayanan at Princeton.
• Who runs it, and in what role. Narayanan’s own bio: “a professor of computer science at Princeton University and the director of the Center for Information Technology Policy” (Narayanan, bio). Kapoor’s page: “In July 2026, I completed my Ph.D. in computer science from Princeton University,” and “I am an incoming assistant professor at the UC Berkeley School of Information, starting in July 2027” (Kapoor). HAL’s page lists no officers and no staff roles, only the authors with their affiliations: Princeton, Ohio State, Stanford, Illinois, Georgetown, KAIST, Stony Brook and others, with one author each at Microsoft Research, Amazon and xAI, and two independent researchers (HAL, Creators).

• The Forum seat. The AI Evaluator Forum’s members page lists “Princeton Holistic Agent Leaderboard” with HAL’s own tagline (AI Evaluator Forum, Members). Princeton’s Center for Information Technology Policy posted on 12 December 2025: “A group of leading AI researchers announced the creation of the AI Evaluator Forum (AEF), a network of independent organizations assessing AI capabilities and risks in the public interest,” and “The Forum includes Princeton’s Holistic Agent Leaderboard (HAL)” (Princeton CITP). Kapoor, Narayanan and Rishi Bommasani (Stanford, also a HAL author) are among the 26 listed authors of AEF-1; both Princeton authors also signed the Forum’s September 2026 letter, in their personal capacity (AEF-1 PDF; AI Evaluator Forum, letter). See the AI Evaluator Forum.
• The reliability turn. HAL’s home page: “We have paused updating HAL leaderboard with new models and are currently focusing on measuring reliability in AI agents.” Its dashboard promises “Accuracy isn’t enough — we evaluate agents on consistency, predictability, robustness, safety, and self-awareness.” The paper behind it, “Towards a Science of AI Agent Reliability” (arXiv:2602.16666, accepted at ICML 2026), by Rabanser, Kapoor, Kirgis, Liu, Utpala and Narayanan, finds “that recent capability gains have only yielded small improvements in reliability” (HAL; HAL, Reliability; arXiv).
• A benchmark declared solved. HAL’s page: “Running Opus 4.5 with an updated scaffold that uses Claude Code drastically outperforms the CORE-Agent scaffold we used, especially after fixing a few grading errors via manual scoring. We have now declared CORE-Bench solved.” The top CORE-Bench Hard entries are marked “Submitted by Nicholas Carlini,” who describes himself as “Research Scientist, Anthropic” (HAL; Carlini). Kapoor’s post, mirrored on HAL’s Insights page, says “Earlier this week, Nicholas Carlini reached out to share that an updated scaffold that uses…” (HAL, Insights).

Funding & Backers

• Named funders, from HAL. “HAL is funded by Coefficient Giving, Schmidt Sciences, the Princeton AI Lab, the Princeton Language and Intelligence Initiative, and the Princeton Catalysis Initiative. We are grateful to OpenAI and Google for providing API credits to evaluate their models” (HAL). The paper’s acknowledgments named “Open Philanthropy, Schmidt Sciences, the Princeton AI Lab, and the Princeton Language and Intelligence Initiative” and thanked only OpenAI for credits; the Reliability page adds “We thank OpenAI and Google for providing compute credits to support our experimentation” (HAL paper PDF; HAL, Reliability). No amount for HAL itself is published.
• Coefficient Giving’s grants to Princeton. The funder’s own listing: “Open Philanthropy recommended a grant of $863,143 to Princeton University to support a project led by Professor Arvind Narayanan to develop an AI R&D capabilities benchmark. This grant was funded via a request for proposals for projects benchmarking LLM agents on consequential real-world tasks” (7 November 2024). A second listing, to Assistant Professor Karthik Narasimhan, is $1,045,620 for a software-engineering agent benchmark (1 July 2024) (Coefficient Giving, grants search). Neither listing names HAL; HAL’s own page names the funder. See Open Philanthropy.
• Schmidt Sciences. On 11 February 2025 the funder announced its AI Safety Science program, “27 projects,” and named “‘AI Snake Oil’ author Arvind Narayanan at Princeton University” among the cohort, with “computational support from the Center for AI Safety and API access from OpenAI” (Schmidt Sciences). The announcement does not name HAL. See Schmidt Sciences and Eric Schmidt.
• The labs’ side of the ledger. OpenAI and Google provide credits; the models scored include OpenAI’s o3, o4-mini and GPT-5, Anthropic’s Claude Opus 4.5, 4.1 and Sonnet 4.5, and, on the Reliability Dashboard, Claude Opus 4.7, Gemini 3.1 Pro, Gemini 3.5 Flash and GPT-5.5 (HAL; HAL, Reliability). No credits from Anthropic are stated. HAL does not publish the credits’ value.
• The AEF-1 category. The Forum’s standard names, as a category to disclose, “a system provider, its employees, or its direct competitors” as sources of “a meaningful fraction of the evaluator’s funding” (AEF-1 PDF). HAL discloses the credits and does not give their size, so whether they are “a meaningful fraction” cannot be settled from the page.

Institutional Voice & Intent

The register is empirical and pedagogical, in the voice of researchers who wrote a book on what AI cannot do. HAL’s own account of the field: “Inconsistent harnesses used across evaluations make it hard to compare results, reproduce evaluations, and are prone to bugs.” Kapoor’s post on HAL’s Insights page: “We spent the last year evaluating agents for HAL. My biggest learning: We live in the Windows 95 era of agent evaluation” (HAL, Insights). The home page names four audiences: “Downstream Users & Procurers,” “Benchmark Developers,” “Agent Developers” and “Safety Researchers.”

Stated intent: Standardize agent evaluation “by providing a third-party platform for comparing agents” (HAL, About); “shift the focus from agents that ace benchmarks to agents that work reliably in the real world” (arXiv).

Observed intent: Three outputs in a year: the leaderboard and harness (26,597 rollouts across nine benchmarks), the published traces (“comprising 2.5B tokens of language model calls”), and a reliability dashboard that replaced new-model scoring. The labs’ models are the instrument’s inputs, and the project’s most-cited findings concern them: its press page lists the Financial Times (August 2025, “Is AI hitting a wall?”) and WIRED (August 2025, “Developers Say GPT-5 Is a Mixed Bag”) as covering HAL’s findings (HAL, Press).

Gap: HAL calls itself “third-party” and the credits line shows two of the developers it scores paying the compute bill for the runs. The record can settle who is named as funder and what the standard asks evaluators to disclose. It cannot settle whether a credit shaped a ranking: the page publishes full traces and an open harness precisely so a result can be re-run, and a headline result Kapoor posted in August 2025 was “While cost-effective, so far GPT-5 never tops agentic leaderboards” (HAL, Insights).

Interest: What each position would pay HAL’s authors and host, documented, with their own best case. The positions are HAL’s.

1. Position: agent evaluations should be standardized, cost-controlled and run by third parties on a common harness. HAL: “Cost-controlled evaluations as default” and “Agents can be 100x more expensive while only being 1% better” (HAL).
   • What adoption would pay HAL. Use of its harness and leaderboard by benchmark developers and agent developers, which the page names as audiences (“Gain improved visibility for your benchmark”); citations to a paper at ICLR 2026; and standing with the funders that run requests for proposals for benchmarking LLM agents, one of which paid Princeton $863,143 for a Narayanan-led benchmark.
   • HAL’s best case, steelmanned. “Inconsistent harnesses used across evaluations make it hard to compare results, reproduce evaluations, and are prone to bugs,” and “Downstream developers can’t tell the difference from 1D leaderboards.”
   • Who else gains from the same rule. Enterprise buyers, who get a price-performance frontier; labs whose cheaper models rank well on it (the first-listed agent on five of the nine benchmark summaries on HAL’s home page runs OpenAI’s o3 or GPT-5); and the other leaderboard operators, from Artificial Analysis to Stanford CRFM, whose HELM HAL names as its model.
2. Position: reliability, not accuracy, is the measure that matters for agents. HAL: “Despite 24 months of model development, overall reliability shows only small improvements over time while accuracy steadily climbs” (HAL, Reliability).
   • What adoption would pay HAL. A new metric family that HAL owns, a dashboard that continues the project after it stopped adding models, and the standing of the authors who wrote the framework.
   • HAL’s best case, steelmanned. “Agents that can solve a task often fail to do so consistently.”
   • Who else gains from the same rule. Funders that issue requests for proposals on agent benchmarking, as Coefficient Giving’s Princeton listing records; labs selling reliability as a feature; and evaluators such as METR that measure task success over time.
3. Position: independent evaluators should disclose funding and credits (AEF-1). HAL is a member and its authors co-wrote the standard.
   • What adoption would pay HAL. A standard that asks for disclosure describes HAL’s practice: its page names every funder and the credits.
   • HAL’s best case, steelmanned. “The operating conditions of third-party evaluations can be opaque, even though they can significantly impact whether an evaluation is trustworthy and impartial” (AEF-1 PDF).
   • Who else gains from the same rule. The Forum’s other seven members and regulators who want a reference text.

Position in the Apparatus

• The university scoreboard, beside the commercial ones. Artificial Analysis and LMArena run leaderboards on venture money and customers that include model makers; Stanford CRFM’s HELM, which HAL names as its inspiration (“HAL is inspired by these efforts”), runs on a university home with corporate affiliates. HAL runs on a university home with philanthropy and lab credits. Authors overlap with CRFM: Yifan Mai, Rishi Bommasani and Percy Liang of Stanford are HAL authors (HAL, Creators), and Narayanan sits on the advisory board of CRFM’s Foundation Model Transparency Index (CRFM, FMTI).
• Inside the evaluator field, by people. HAL’s acknowledgments thank individuals at Anthropic (Hailey Schoelkopf), Google DeepMind (Iason Gabriel, Laura Weidinger, William Isaac), Apollo Research (Marius Hobbhahn), Transluce (Kevin Meng) and, listed under “UK AISI,” JJ Allaire, Charles Teague, Jelena Luketina and Maximillian Kaufmann (HAL, Creators). AEF-1’s author list places Teague at Meridian Labs. Percy Liang is listed among Transluce’s advisors (Transluce, About); see Percy Liang.
• Shared funders with the other members. Coefficient Giving funds HAL and, per the Forum’s account, AVERI, SecureBio and RAND’s Canary; Schmidt Sciences funds HAL and METR. The Survival and Flourishing Fund and Halcyon Futures, funders of other members, are not named on HAL’s pages.
• A framework beside the instrument. Narayanan and Kapoor co-wrote the book “AI Snake Oil” and the essay and newsletter “AI as Normal Technology,” described on Narayanan’s page as “the leading intellectual alternative to the view of AI as impending superintelligence” and read by “over 75,000” (Narayanan, bio; Narayanan). The evaluation instrument and the framework share authors, a funder (Schmidt Sciences’ safety program lists Narayanan) and a house (CITP).
• Government. Narayanan’s bio lists advisory boards for an NSF council on big data, ethics and society and an NIH data-sharing project, and a “Presidential Early Career Award for Scientists and Engineers (PECASE).” Kapoor’s page lists policy input to the New Jersey Assembly on deepfakes and a congressional forum on predictive AI (Kapoor). HAL’s pages name no agency contract.

Actions & Leadership Choices

Read by deeds, HAL is a university group that built a harness, ran the labs’ models through it with the labs’ credits, published the traces and then changed the question from accuracy to reliability when the leaderboard saturated. The leadership choices are two academics who also run a newsletter and a book project, and a lead author who has moved on.

Actual founding purpose. To give agent evaluation the common ruler HELM gave language models. The home page lays out the problem in four bullets and the solution in two components; the paper followed in October 2025 (HAL; arXiv).

• The leaderboard. Nine benchmarks: AssistantBench, CORE-Bench Hard, GAIA, Online Mind2Web, Scicode, ScienceAgentBench, SWE-bench Verified Mini, TAU-bench Airline and USACO; “26 597 Rollouts” on the home page (HAL).
• The traces. “We make available the full traces of agent evaluations, including every single model call,” and “All agent traces are encrypted to prevent benchmark contamination through automated scraping” (HAL).
• The pause and the pivot. New-model scoring stopped; the Reliability Dashboard, whose current table runs from Claude Opus 4.5 to GPT-5.5 and Gemini 3.5 Flash, took its place (HAL, Reliability).
• The Forum. A member; Kapoor and Narayanan are AEF-1 authors; no HAL evaluation is listed among the three that carry the AEF-1 checklist (AI Evaluator Forum, AEF-1).

Leadership choices. Narayanan holds the directorship of the host center; Kapoor, first author of the HAL paper, finished his doctorate in July 2026 and starts at Berkeley in July 2027. The HAL pages name neither a successor nor a plan for the leaderboard after the pause.

CONDUCT: UNIVERSITY LEADERBOARD — A THIRD-PARTY RULER FOR THE LABS’ AGENTS, RUN ON PHILANTHROPY AND THE LABS’ CREDITS. HAL is not hostile to the labs and its findings are not written to please them: one of its early headline results was that GPT-5, “while cost-effective,” never topped the agentic leaderboards. What it has is a compute bill partly paid by two of the developers it scores, philanthropic funders that also fund peers on the Forum, and a seat at the table whose standard asks evaluators to disclose exactly that. The structure is the finding; the intent is not asserted.

Alliance Map

These run at the same time. One university project is paid by the philanthropies behind four other Forum members, has two of the developers it scores pay its compute bill, credits an Anthropic researcher with the scaffold it used to declare a benchmark solved, and sits at the table that writes the standard asking evaluators to disclose exactly that. Each tie is stated by HAL, a funder or an individual’s own page. The overlap is the finding.

Reach Assessment

Institutional: Moderate-low. Researchers and the press use it; no regulator, procurer or lab is documented as requiring it. The leaderboard has stopped adding models.

Memetic: Moderate. Its findings, that cost matters and that reliability lags accuracy, were picked up by the Financial Times and WIRED on HAL’s account, and the authors’ framework reaches a newsletter audience of 85,000 by Kapoor’s count.

Civilizational: Low to moderate, with low durability. The leaderboard builds no model and gates no release. It shapes what counts as a good agent, and a ranking that every lab can cite is a ranking every lab will optimize. The hazard is a reference funded by the field it scores and run by authors who are leaving it. The reach is the finding; the malice is not asserted.

Score move: none (new profile). Calibrated against the AI Evaluator Forum (5/6/4, hazard 50), AVERI (5/6/4, hazard 50), Stanford CRFM (7/6/6, hazard 63) and Apart Research (5/5/5, hazard 50), with Artificial Analysis (8/4/6, hazard 60) as the reach contrast.

Score basis. Atk 4 is reach: HAL’s home page shows 26,597 rollouts across nine benchmarks, its press page lists the Financial Times and WIRED, and the dashboard ranks current models through Claude Opus 4.7, GPT-5.5 and Gemini 3.5 Flash (HAL; HAL, Press; HAL, Reliability). It is one under the Forum and AVERI at 5, which reach a regulator by the Forum’s account or a statute, and three under Stanford CRFM at 7, whose Cybench the US and UK institutes run; it is four under Artificial Analysis at 8, whose index buyers use. It is not higher because nothing names HAL as a requirement and the home page says “We have paused updating HAL leaderboard with new models.” Def 6 is funding insulation: HAL “is funded by Coefficient Giving, Schmidt Sciences, the Princeton AI Lab, the Princeton Language and Intelligence Initiative, and the Princeton Catalysis Initiative,” and no lab pays it (HAL). It is not higher because “OpenAI and Google” provide the credits, Coefficient Giving also funds three other Forum members, and the amounts are not published; it matches the Forum, AVERI and Stanford CRFM at 6 and sits one above Apart Research at 5. Hp 4 is entrenchment: an ICLR 2026 paper, an open-source harness, published logs of 2.5 billion tokens and a university home (arXiv). It is not higher because no statute, contract or standard installs it, the leaderboard is paused and the lead author starts at Berkeley in July 2027 (Kapoor). It matches the Forum and AVERI at 4 and sits two under CRFM at 6, whose HELM and Foundation Model Transparency Index are program-of-record fixtures. The values hold at 4/6/4, hazard 47.

Sources: HAL; HAL — About; HAL — Creators; HAL — Press; HAL — Insights; HAL — Reliability; Holistic Agent Leaderboard (arXiv:2510.11977); Holistic Agent Leaderboard (PDF); Towards a Science of AI Agent Reliability (arXiv:2602.16666); Arvind Narayanan — bio; Arvind Narayanan; Sayash Kapoor; Princeton CITP — HAL is part of the Forum; Coefficient Giving — grants search, Princeton; Schmidt Sciences — AI Safety Science program; Nicholas Carlini; Transluce — About; Stanford CRFM — Foundation Model Transparency Index; AI Evaluator Forum — Members; AI Evaluator Forum — AEF-1; AEF-1 (PDF); AI Evaluator Forum — Public Letter.