ARTIFICIAL ANALYSIS

OLYMPUS opened an institutional file on Artificial Analysis because its Intelligence Index is the number a lab puts in the capability paragraph of a launch post, and because the company that produces it states, in its own founders’ words, how it is paid. This is not a psychometric profile, since an institution has no Dark Triad. It is a mandate, a methodology page, a version history, a contact form and a founders’ interview, and the finding is the shape of the arrangement: a ranking that began as a side project to choose between model hosts is now a company with a public index, a private benchmarking line sold to “companies throughout the AI stack,” a partner alliance for its newest index, and graders that are the models of the labs it ranks.

Institutional Archetype

THE RATINGS AGENCY WITH A CLIENT LIST — Artificial Analysis runs the same tests on every model it can reach, publishes the scores for free, and sells the surrounding services: standardized reports to enterprises and custom benchmarks to AI companies. Its own sentence for what it is: “Artificial Analysis is the independent benchmarking company for AI” (Artificial Analysis, About). The published ranking has no entry fee. The sold products are separate, and no page names a customer or a price. Unlike a contract evaluator on the evaluator map, it does not grade a lab’s model inside the lab’s system card. Its output is an index that the lab itself then quotes. OpenAI’s GPT-5.6 post of 9 July 2026 says “On the Artificial Analysis Intelligence Index, a broad measure of intelligence spanning agentic work, coding, scientific reasoning, and general capabilities, GPT-5.6 Sol with max reasoning comes within one point of Fable 5 while completing tasks in 61% less time at roughly half the estimated cost” (OpenAI, GPT-5.6). Where LMArena counts crowd votes, Artificial Analysis runs fixed task suites; where Epoch AI is a nonprofit funded mostly by philanthropy, this is a venture-backed company funded by individual investors and its own client revenue.

Mandate & Origin

Artificial Analysis’s account of its scope: “We measure the agents and models, the clouds that serve them, and the chips they run on, helping developers and companies optimize the entire AI stack” (Artificial Analysis, About). Its methodology page: “We benchmark both proprietary and open weights models” (Artificial Analysis, Methodology). It lists San Francisco as headquarters, with offices in Melbourne and remote staff (Artificial Analysis, About).

Founders. The About page names two: “Micah Hill-Smith is Co-Founder and CEO of Artificial Analysis, based in San Francisco,” and “George Cameron is based out of San Francisco, California and is the CPO and Co-Founder of Artificial Analysis” (Artificial Analysis, About). The same page gives Hill-Smith’s earlier job as consultant with McKinsey & Company and Cameron’s as “Strategy Consulting focusing on data centers and technology companies.”

Legal form. The website terms of use, last revised 15 September 2026, describe the site as “a copyrighted work belonging to Artificial Analysis, Inc.” (Artificial Analysis, Terms of Use). The terms do not state the state of incorporation.

Origin, in the founders’ words. In a Latent Space interview published 8 January 2026, Hill-Smith said: “Artificial analysis is going to be two years old in January 2026,” and that the site is free to use. He traced the project to 2023, when he was building a legal AI research assistant and found that “the more you go into building something using LLMs, the more each bit of what you’re doing ends up being a benchmarking problem” (Latent Space, interview). The company went through AI Grant’s fourth batch; Hill-Smith: “It was really great to get Nat and Daniel on board” (Latent Space, interview).

Funding & Backers

In Artificial Analysis’s words. The About page lists, under “Backed by,” eight individuals: Nat Friedman, Daniel Gross, Andrew Ng, Adam D’Angelo, Clem Delangue, Guillermo Rauch, Shawn Wang (swyx) and Charlie Songhurst, with no amount, round, date or valuation (Artificial Analysis, About). The careers page repeats seven of them under “Backed by industry leaders” (Artificial Analysis, Careers). The company is listed in AI Grant’s fourth batch as “Independent benchmarks and evaluations,” and AI Grant’s terms for that batch are “$250,000 on an uncapped SAFE for your AI-native product startup” (AI Grant). The page states the program’s terms, not what this company received, and no Artificial Analysis page states an amount.

• Who the page names, and who it does not. No model developer, venture firm or corporate investor is named on the pages read. The About page gives no role for any backer. AI Grant’s own page lists “Guillermo Rauch, CEO of Vercel” among its advisors (AI Grant); Vercel is a partner in the company’s Cyber Index Alliance, below.
• How it says it earns. Hill-Smith: “So no one pays to be on the website. We’ve been very clear about that from the very start because there’s no use doing what we do unless it’s independent AI benchmarking.” On the revenue side: “We have, along the way, built a business that is working out pretty sustainably. We’ve got just over 20 people now and two main customer groups” (Latent Space, interview). The first group is enterprises: Cameron, “we have a benchmarking and insight subscription, which looks like standardized reports that cover key topics or key challenges enterprises face when looking to understand AI and choose between all the technologies.” The second is AI companies: “we do private benchmarking for companies throughout the AI stack who build AI stuff” (same interview).
• The commercial menu. The contact page offers “Advisory services,” “Custom benchmarking,” “Data & API Access” with “Commercial licenses for higher limits and redistribution in products, reports, and data feeds,” and “Partnerships,” among them “Research collaborations and co-published analysis” and “Other partnership opportunities with labs, providers, and enterprises” (Artificial Analysis, Contact). A custom benchmark is offered to “organizations using AI and AI platform & technology providers” (same page). The careers page describes one job as owning “our client relationships with the companies building frontier AI end to end, backed by a technical ramp deep enough to hold your own in methodology conversations,” and another as work to “work with frontier labs on pre-release models” (Artificial Analysis, Careers).
• Size. About 20 people in January 2026 by the founders’ account, “A team of 40 with colleagues from Google, Microsoft, McKinsey & Company, and BCG” on the careers page in October 2026 (Latent Space, interview; Artificial Analysis, Careers).

Institutional Voice & Intent

On its own standard, the methodology page: “we are confident that it is a more useful synthesis comparison between language models than any other metric in existence today” (Artificial Analysis, Intelligence Index methodology). On why it runs its own tests, Hill-Smith: “we laser focus, like, on everything we do on having the best independent metrics and making sure that no one can manipulate them in any way.” On what a lab-supplied endpoint risks and how it answers: “if we’re working with a lab, if they’re giving us a private endpoint to evaluate a model, that it is totally possible. That what’s sitting behind that black box is not the same as they serve on a public endpoint. We’re very aware of that. We have what we call a mystery shopper policy,” under which, he says, the company registers accounts “not on our own domain” and runs the tests so the lab cannot identify it (Latent Space, interview). On the limit of any index: “once an eval becomes the thing that everyone’s looking at, scores can get better on it without there being a reflection of overall generalized intelligence of these models” (same interview).

Stated intent: Be “the independent benchmarking company for AI,” with a free public ranking that no one pays to appear on, and results that “no one can manipulate.”

Observed intent: Turn a free comparison site into the measuring layer the labs, chip vendors and enterprises all reference, and sell the work around it. The careers page: “We are the leading independent AI benchmarking and insights company, trusted by hundreds of thousands of users and referenced by OpenAI, Google, Meta, NVIDIA, Anthropic, the Wall Street Journal, The Economist, and Bloomberg” (Artificial Analysis, Careers). The About page carries the line “Publicly referenced by the leading AI labs, publications, and institutions” above lab logos and a wall of posts from lab, executive and its own accounts under the heading “The go-to reference” (Artificial Analysis, About).

Gap: The company states a clean separation between the free public ranking and the paid private work, publishes the weights and the version history, and describes a policy for catching a lab that serves a different model. What its pages do not settle is the size of the private line: no page gives the number of AI companies that buy private benchmarking, the share of revenue it earns, or whether any customer is also a ranked lab. The founders say the first group of customers is enterprises and the second is AI companies; the second group is the one the public ranking measures. Both statements are on the record and the figure that would let a reader weigh them is the one neither prints.

Interest: Artificial Analysis argues for independent third-party measurement, for an index weighted toward agentic and privately held tasks, and for industry partners helping design a cyber index. What each position would pay is set out from its own pages; no contract value is published.

• Independent measurement as the product. Position, Hill-Smith’s: “there’s no use doing what we do unless it’s independent AI benchmarking” (Latent Space, interview). What it pays: the independence claim is what the subscription and the private benchmarking are sold on, and a lab that quotes the Index in a launch post widens the audience that finds both. Best case, the founders’: a lab’s own table is “not the same” as a neutral run, and the company runs identical tests on every model (same interview). Who else gains: the labs that top the ranking, which OpenAI’s GPT-5.6 post and NVIDIA’s blog posts show using it as a headline; open-weights developers, because the company benchmarks “both proprietary and open weights models”; and enterprises that buy a vendor-neutral selection report.
• An index weighted toward agents and private test sets. Position, the v4.3 launch article: the changes “add more private test sets to prevent gaming, and reduce saturation,” and “Because we use a held-out test set for AutomationBench in collaboration with Zapier, the weight assigned to evaluations with private tasks or answers increases from 40% to 45%” (Artificial Analysis, Index v4.3). What it pays: each new version is a new article and a fresh set of scores for labs to quote, and the partner-held sets (Zapier’s AutomationBench; Harvey’s legal dataset of 120 private tasks) tie the company to the partners that hold the answers. Best case, its own: a public benchmark saturates and a private one resists training on the test. Who else gains: the partners whose benchmark becomes an Index component, and the labs that tune to the new weighting.
• Industry partners in the cyber index. Position, the Cyber Index page: “Partners contribute expert input on the design and implementation of the Index, and may contribute datasets and external research directly,” with “All evaluations are conducted independently by Artificial Analysis” (Artificial Analysis, Cyber Index). What it pays: the three component benchmarks come from “Collinear AI,” “Vercel” and “Berkeley RDI,” and the alliance members listed are Collinear AI, IBM, NVIDIA and Vercel, so each partner’s benchmark or interest sits inside a ranked index. Best case, the company’s: enterprise cyber defense needs practitioner input to be measured at all. Who else gains: NVIDIA, which is both an alliance member and, below, a subject of the company’s chip benchmarks; and OpenAI, whose restricted Daybreak model now leads the index.

Interest is a documented fact about each position, not a claim about motive, and not a verdict that the argument is wrong. Each best case is stated at the strength its author states it.

Position in the Apparatus

Artificial Analysis sits in the ranking layer, beside LMArena and Stanford CRFM’s HELM, not in the audit layer with METR, Irregular and Gray Swan, which test models for labs inside release decisions. Epoch AI and Scale AI build benchmarks; Artificial Analysis implements other people’s benchmarks and some of its own and publishes a composite. The AI evaluators map does not list it.

• The labs cite the Index in launch posts. OpenAI, 9 July 2026: “On the Artificial Analysis Intelligence Index, a broad measure of intelligence spanning agentic work, coding, scientific reasoning, and general capabilities, GPT-5.6 Sol with max reasoning comes within one point of Fable 5 while completing tasks in 61% less time at roughly half the estimated cost,” and “On the Artificial Analysis Coding Agent Index, GPT-5.6 Sol with max reasoning sets a new state of the art at 80” (OpenAI, GPT-5.6). The GPT-6 Astra page lists the “Artificial Analysis Intelligence Index v4.1.1” and “Artificial Analysis Coding Agent Index v1.4” in its benchmark tables (OpenAI, GPT-6 Astra). NVIDIA, 11 August 2026: “Nemotron 3.5 Lightning delivers leading accuracy at the highest output speed in its class, winning the accuracy-versus-speed Pareto frontier on the Artificial Analysis Intelligence Index” (NVIDIA, Nemotron 3.5 Lightning); 4 June 2026: “Nemotron 3 Ultra achieves 5x faster inference while delivering leading accuracy on the Artificial Analysis Intelligence Index leaderboard” (NVIDIA, Nemotron 3 Ultra); and “Nemotron 3.5 Super VL, on Artificial Analysis Intelligence Index leadboard, achieves up to 5x faster throughput compared to other models in its class” (NVIDIA, blog). Meta’s Hugging Face model card for Muse Spark Open, release date 15 October 2026, carries the Artificial Analysis Intelligence Index in its benchmark table against Mistral Large 4, Qwen3.8, GLM 5.3 and Kimi K3 (Meta, Hugging Face model card).
• The index is versioned, and the lab’s number belongs to a version. OpenAI’s July table is labeled “Artificial Analysis Intelligence Index v4.1” and the Astra page “v4.1.1” (OpenAI, GPT-5.6; OpenAI, GPT-6 Astra). The company’s own history: version 3.0 (2 September 2025) added Terminal-Bench Hard and τ²-Bench Telecom; version 4.0 (January 2026) added GDPval-AA, AA-Omniscience and CritPt and removed “MMLU-Pro, LiveCodeBench, AIME 2025”; version 4.3 (September 2026) replaced τ³-Banking with AutomationBench-AA and Terminal-Bench 2.1 with Terminal-Bench 4.0 (Artificial Analysis, Index methodology). The v4.3 launch article reports “Both Claude Fable 5.1 (max with fallback) and GPT-6 Astra (max) score 53 on Intelligence Index v4.3,” and calls the release “a continuation of our rollout of Intelligence Index v5” (Artificial Analysis, Index v4.3).
• The graders are the labs’ models. The Index “is calculated as a weighted average across four categories: Agents (30%), Coding (20%), Scientific Reasoning (20%) and General (30%)” and “incorporates 10 evaluations: AA-Briefcase v1.1, GDPval-AA v2.1, AutomationBench-AA, Terminal-Bench 4.0, SciCode, AA-LCR v1.1, AA-Omniscience, Humanity’s Last Exam, GDP.pdf, CritPt.” Version 4.3.1 “Refreshed the pairwise judge panels to current model versions: AA-Briefcase and GDPval-AA v2 pairwise comparisons now judge with Claude Opus 5, GPT-5.6 Sol and Gemini 3.8 Flash,” and version 4.1.1 made GPT-5.6 Luna the grader for HLE, AA-LCR and AA-Omniscience, “replacing GPT-4o, Qwen3 235B A22B 2507, and Gemini 3 Flash Preview respectively,” with the reason that the model was “selected for strong agreement with human judgment in our grader validation” (Artificial Analysis, Index methodology; Artificial Analysis, v4.1.1). GDPval-AA “is Artificial Analysis’ evaluation framework for OpenAI’s GDPval dataset.” The company reports that the grader change moved “most models” by less than a point, the largest rise being “Muse Spark 1.2 (xhigh, +2.7 points).”
• The hardware side. The company benchmarks the accelerators too. Of NVIDIA’s Vera Rubin NVL72 results on AA-AgentPerf it writes “NVIDIA shared how it built its Vera Rubin and GB300 configs,” and notes that NVIDIA’s DeepSeek V4.1 Flash run “capped replies at 2,048 output tokens, compared to 16,384 for the other DeepSeek V4.1 Flash results on our leaderboard,” which it says “modestly favors Vera Rubin’s throughput” (Artificial Analysis, Vera Rubin NVL72). The company states the difference in the article itself.
• Trusted-access models. On 8 October 2026 the Cyber Index added a model “only accessible through OpenAI’s Daybreak program”: “GPT-6 Sol (Daybreak Blue) now leads the Cyber Index” and “It hits no safety blocks across the entire Index” (Artificial Analysis, trusted-access models). The company’s ranking now includes a model that the public cannot use, supplied through a lab’s program.
• Output. The articles index shows 129 articles, the newest dated 16 October 2026 (Artificial Analysis, Articles).

Actions & Leadership Choices

CONDUCT: WATCH — AN INDEX THE LABS CITE, A PRIVATE-BENCHMARKING BUSINESS THE LABS CAN BUY, GRADED BY THE LABS’ OWN MODELS. No wrongdoing is alleged. The documented facts: a free ranking launched in January 2024, a company built around it, a subscription and a private benchmarking line sold to enterprises and AI companies, a mystery-shopper policy for lab-supplied endpoints, an index graded by the models of the labs it ranks, and an alliance for the newest index that includes a chip vendor and a startup whose CEO the accelerator it came through lists as an advisor. The structural point is the one every paid evaluator shares, sharpened by the ranking’s visibility: the labs it scores quote it, may give it private endpoints and may buy services from the company that runs it, and no page gives the weights of those flows.

• January 2024. Versions 1.0 to 1.3 of the Index run to 10 February 2025 (Artificial Analysis, Index methodology).
• 11 February 2025. Index version 2.0 (Artificial Analysis, Index methodology).
• 2 September 2025. Index version 3.0, adding Terminal-Bench Hard and τ²-Bench Telecom (Artificial Analysis, Index methodology).
• 8 January 2026. The founders publish the interview stating the revenue model and the mystery-shopper policy (Latent Space).
• January 2026. Index version 4.0 adds GDPval-AA, AA-Omniscience and CritPt and drops MMLU-Pro, LiveCodeBench and AIME 2025 (Artificial Analysis, Index methodology).
• 9 July 2026. OpenAI’s GPT-5.6 post cites the Index and the Coding Agent Index (OpenAI).
• 6 August 2026. Index v4.1.1 changes the graders to GPT-5.6 Luna (Artificial Analysis).
• 7 September 2026. Index v4.3 raises the weight of private-test evaluations from 40% to 45% (Artificial Analysis).
• 8 October 2026. The Cyber Index adds an OpenAI trusted-access model and takes the lead (Artificial Analysis).
• 15 October 2026. First AA-AgentPerf results for NVIDIA’s Vera Rubin NVL72 (Artificial Analysis).

Alliance Map

These run at the same time. One company is funded by individual investors, sells subscriptions and private benchmarking, is quoted by OpenAI and NVIDIA in launch material, takes private endpoints from labs under a mystery-shopper policy, grades with the labs’ own models, benchmarks the chips of a company that sits in its cyber alliance and quotes a bank of partner-held private test sets. Each tie is stated by Artificial Analysis, a lab or AI Grant. The overlap is the finding.

Reach Assessment

Institutional: Labs write the Index into launch posts, chip vendors cite its throughput results, and enterprises can buy reports and custom benchmarks from the company. No statute or procurement rule names it. Memetic: High: the careers page claims “hundreds of thousands of users,” and a score on the Index is a number a model announcement can lead with. Civilizational: If a composite score decides which model the world believes is smartest, the company that sets the weights sits between the labs and the buyers they sell to; if it also sells those labs private benchmarking, the same company stands on both sides of the number. The record shows published weights, a full version history, a stated policy against paid placement and a stated policy against lab-served variants. It does not show the size of the private line or whether any customer is a ranked lab. Those are the gap.

Score basis. Atk 8 is reach: OpenAI’s GPT-5.6 post uses the Index and the Coding Agent Index as headline measures, NVIDIA’s developer blog places three Nemotron releases on the Index leaderboard, Meta’s model card lists it, and the company’s careers page claims “hundreds of thousands of users” (OpenAI; NVIDIA; Meta, model card; Artificial Analysis, Careers). The points under 9 are that no statute, procurement rule or government pre-deployment test names it and that a lab can stop citing it. Peers on atk 8: LMArena, Gray Swan and Scale AI; Epoch AI holds 5, its output being widely read and commissioned but never required, and US CAISI and the UK AI Security Institute hold 9. Def 4 is funding insulation: Hill-Smith says “no one pays to be on the website,” but the company sells a subscription and “private benchmarking for companies throughout the AI stack who build AI stuff,” accepts private endpoints from labs, and counts NVIDIA among its Cyber Index Alliance members while benchmarking its chips; its backers are listed by name with no amount and no customer is named (Latent Space; Artificial Analysis, Cyber Index; Artificial Analysis, About). The points kept are the stated no-fee rule, the mystery-shopper policy, the published methodology and the absence of any lab or venture firm among the named backers. Peers on def 4: LMArena, which sells evaluation to model labs and names its investors but no customer; Meridian Labs and Andon Labs hold 5, Promptfoo holds 3 because a lab owns it, and Scale AI, Gray Swan and Hugging Face hold 6. Hp 6 is entrenchment: the Index is a versioned product of ten evaluations with partner-held private sets, a Cyber Index Alliance and chip and cloud benchmarks, and labs print its version numbers in their tables (Artificial Analysis, Index methodology; OpenAI, GPT-6 Astra); against that, no statute installs it, no multi-year contract is disclosed, the staff is about forty by its own count and every lab can swap to another leaderboard. Peers on hp 6: Andon Labs and Promptfoo; LMArena and Epoch AI hold 7, Hugging Face and Scale AI hold 8. Hazard 60 follows from 8/4/6 at the house formula and sits under LMArena (63), which has the same reach and insulation and one more point of entrenchment, and level with the neighborhood of Stanford CRFM (63) and Hugging Face (62).

Sources: Artificial Analysis — About; Artificial Analysis — Contact; Artificial Analysis — Careers; Artificial Analysis — Terms of Use; Artificial Analysis — Benchmarking Methodology; Artificial Analysis — Intelligence Index methodology; Artificial Analysis — Cyber Index methodology; Artificial Analysis — Cyber Index; Artificial Analysis — Articles; Launching v4.1.1 of the Intelligence Index — Artificial Analysis (6 Aug 2026); Announcing the Intelligence Index v4.3 — Artificial Analysis (7 Sep 2026); Introducing trusted-access models to the Cyber Index — Artificial Analysis (8 Oct 2026); First AA-AgentPerf results for NVIDIA Vera Rubin NVL72 — Artificial Analysis (15 Oct 2026); Artificial Analysis: Independent LLM Evals as a Service — Latent Space (8 Jan 2026); AI Grant; GPT-5.6 — OpenAI (9 Jul 2026); GPT-6 Astra — OpenAI; Nemotron 3.5 Lightning — NVIDIA Technical Blog (11 Aug 2026); Nemotron 3 Ultra — NVIDIA Technical Blog (4 Jun 2026); Nemotron 3.5 Super VL — NVIDIA; Muse Spark Open — Meta, Hugging Face.