Research: The AI Safety Tools — The Software That Tests, Red-Teams, Guards and Scores the Models, and Who Owns It
Every test harness, red-team framework, benchmark, guardrail model and security vendor the graders of frontier AI are built on: what each does in its maker's words, who owns it today and since when, the licence, the money, and which lab, government institute or standard names it. Covers the tools released by OpenAI, Anthropic, Google DeepMind, Meta, Microsoft, NVIDIA, Amazon, IBM and Cisco, the government harnesses, the open benchmarks, the vendors, and the published methods underneath them.
Contents
Companion pages: The AI Evaluators (the organizations that grade the labs) · The Evaluator Alliances (who tests whom, who pays) · Watching the Watchers · AI Agent Incidents · the revolving-door graph.
The evaluator pages map the people and organizations that grade frontier models. This page maps the instruments they grade with. A system card reports a number; the number came out of a harness, a benchmark or a classifier, and each of those has an owner, a licence, a funder and a list of institutions that depend on it. The stack has four layers. At the bottom sit the published attack and measurement methods, mostly academic. Above them are the open harnesses and benchmarks that turn the methods into repeatable tests, built by governments, labs, universities and a handful of nonprofits. Next come the guardrail models that sit in front of production systems and the commercial vendors that wrap both into a product. The top layer is the system card, where a lab reports the result. The pattern the record shows: the open layer is real and free to inspect, a growing share of it has moved under a lab or a security conglomerate, and the lab-built tools that matter most to a government evaluator are now being handed to small nonprofits, none of which has a profile here yet.
Government and standards tooling
| Tool | What it is | Owner | Licence | Who names or runs it |
|---|---|---|---|---|
| Inspect | “a framework for large language model evaluations created by the UK AI Security Institute” (GitHub); its docs say it is “developed by the UK AI Security Institute and Meridian Labs” (Inspect) | UK AI Security Institute with Meridian Labs | MIT (repo) | The US institute ran a benchmark by “modifying the benchmark to run in the Inspect evaluation framework” (NIST); METR says “METR is transitioning its internal tooling from Vivaria to” Inspect (GitHub); Anthropic’s Petri runs on it (below) |
| Inspect Evals | A library of evaluations built on Inspect; “Inspect Evals is maintained by” Generality Labs, “a London-based nonprofit which builds tools for evaluating risks and mitigations in frontier AI” (GitHub) | Generality Labs, which “has been awarded $4.4M of seed funding from Coefficient Giving.” (Generality Labs); Coefficient Giving is the former Open Philanthropy | MIT (repo) | Founded 2024 with contributions from the UK institute and Arcadia Impact (GitHub) |
| Inspect Cyber | An extension to Inspect for agentic cyber evaluations (GitHub) | UK AI Security Institute | MIT (repo) | |
| AgentDojo-Inspect | “AgentDojo-Inspect is a fork of the original AgentDojo repository” (NIST), built so agent-hijacking tests run in Inspect | NIST’s Center for AI Standards and Innovation | NIST repository (NIST) | CAISI: “CAISI has open-sourced our improvements to the AgentDojo framework” (NIST) |
| Dioptra | “Dioptra is a software test platform for assessing the trustworthy characteristics of artificial intelligence (AI).” (GitHub) | NIST | Custom licence file (repo) | |
| AI Verify and Moonshot | Moonshot “is a tool to bring Benchmarking and Red-Teaming together” (GitHub) | AI Verify Foundation, “a wholly-owned subsidiary of IMDA,” Singapore’s media and telecoms regulator (MLCommons) | Apache-2.0 (repo) | “MLCommons is a proud partner with the AI Verify Foundation” (MLCommons) |
| AILuminate | “The AILuminate Benchmark is a family of safety and security benchmarks that assesses genAI across 12 hazard categories” (MLCommons) | MLCommons, whose AI Risk and Reliability working group lists Anthropic, Google, Google DeepMind, Meta FAIR, Microsoft, NVIDIA, OpenAI, Cohere and Intel among its contributors (MLCommons) | Apache-2.0 (repo) | Singapore’s AI Verify (above). MLCommons’ own FAQ says neither the NIST AI Risk Management Framework nor its Generative AI Profile references AILuminate and no statute requires it (MLCommons profile) |
What the major players have released
Tools for testing, auditing and red-teaming, by maker. Guardrail and classifier models are in the next section.
| Maker | Tool | What it does | Licence | Who relies on it |
|---|---|---|---|---|
| OpenAI | Evals | “Evals provide a framework for evaluating large language models (LLMs) or systems built using LLMs.” (GitHub) | Custom licence file (repo) | |
| OpenAI | simple-evals | A public eval library (GitHub) | MIT (repo) | Anthropic’s Opus 5 card lists the repository’s address in its evaluation blocklist (Anthropic) |
| OpenAI | Frontier Evals: PaperBench, SWE-Lancer, EVMbench | “Code for evals measuring frontier model capabilities.” (GitHub) | MIT (repo) | |
| OpenAI | MLE-bench | Benchmark of agents on machine-learning engineering (GitHub); leaderboard closed to submissions on April 24, 2026 (GitHub) | Custom licence file (repo) | OpenAI’s GPT-5.6 card lists MLE-Bench Revised among its AI self-improvement evaluations (OpenAI) |
| OpenAI | Preparedness Framework | “our process for tracking and preparing for advanced AI capabilities that could introduce new risks of severe harm” (OpenAI), updated April 15, 2025 | Policy document | |
| OpenAI | Promptfoo | Open-source evaluation and red-team tool; Promptfoo’s announcement says it “has agreed to be acquired by OpenAI” (Promptfoo), March 9, 2026 | MIT: “Promptfoo is now part of OpenAI. Promptfoo remains open source and MIT licensed.” (GitHub) | “more than 25% of the Fortune 500 rely on it” (Promptfoo); investors named are Insight Partners and Andreessen Horowitz (Promptfoo) |
| Anthropic | Petri | Alignment auditing agent; “We have handed over its development to Meridian Labs” (Anthropic), May 7, 2026, to “help ensure that Petri remains independent of any AI lab” (Anthropic) | MIT (repo) | “Petri has been part of our alignment assessment for every Claude model since Claude Sonnet 4.5.” The UK institute “made it a major part of how they evaluate models for their propensity to sabotage AI research” (Anthropic); the Fable 5 card reports “Petri 3.0 evaluations conducted by Meridian and the UK AI Security Institute (UK AISI)” on models from other developers (Anthropic) |
| Anthropic | Bloom | “an open source agentic framework for generating behavioral evaluations of frontier AI models” (Anthropic), December 19, 2025 | MIT (repo) | Its original repository says “It is now developed and maintained by Meridian Labs” (GitHub) |
| Anthropic | SHADE-Arena | Sabotage-and-monitoring benchmark; “This is the public split of SHADE-Arena.” (GitHub) | MIT (repo) | “We evaluated Claude Opus 5.5 on SHADE-Arena” (Anthropic); it recurs in the Opus 5, Fable 5 and Fable 5.1 cards |
| Anthropic | Automated auditing agents | “We develop three agents that autonomously perform alignment auditing tasks.” (Anthropic Alignment Science), July 24, 2025 | Research release | |
| Anthropic | circuit-tracer | Attribution-graph tools: “we’re open-sourcing the method so that anyone can build on our research” (Anthropic), May 29, 2025, “in collaboration with Decode Research” (Anthropic) | MIT (repo) | |
| Anthropic | Many-shot jailbreaking | “The basis of many-shot jailbreaking is to include a faux dialogue between a human and an AI assistant” (Anthropic); April 2, 2024; Anthropic “confidentially shared the details of many-shot jailbreaking with many of our fellow researchers” (Anthropic) | Published method | |
| Google DeepMind | Frontier Safety Framework | “our most comprehensive approach yet to identifying and mitigating severe risks from advanced AI models” (Google DeepMind), third version September 22, 2025 | Policy document | Gemini 4 Argon’s report: “we partnered with external red-teaming organizations including Gray Swan and Trajectory Labs” (Google DeepMind) |
| Google DeepMind | Gemma Scope | “a new set of tools to help researchers understand the inner workings of Gemma 2” (Google DeepMind), July 31, 2024; hundreds of open sparse autoencoders | Open weights | |
| Google DeepMind | SynthID | “Google SynthID is our watermarking tool, designed specifically for AI-generated content.” (Google DeepMind) | Proprietary detector | Partners named: OpenAI, NVIDIA, Kakao and, soon, Apple (Google DeepMind) |
| Secure AI Framework (SAIF) | “The SAIF Map is a visual guide for navigating AI security” (SAIF); its risk assessment and map were shared with the Coalition for Secure AI (SAIF) | Framework | ||
| Meta | Purple Llama: CyberSecEval, Llama Guard, Prompt Guard, Code Shield, LlamaFirewall | “Purple Llama is an umbrella project that over time will bring together tools and evals to help the community build responsibly with open generative AI models” (GitHub) | Evals and Code Shield MIT; Llama Guard and Prompt Guard under the Llama community licences (GitHub) | “Code Shield adds support for inference-time filtering of insecure code produced by LLMs” (GitHub); “LlamaFirewall is a framework designed to detect and mitigate AI centric security risks” (GitHub) |
| Microsoft | PyRIT | “an open source framework built to empower security professionals and engineers to proactively identify risks in generative AI systems” (GitHub); the repository moved from Azure/PyRIT, archived March 25, 2026, to microsoft/PyRIT (GitHub) | MIT (repo) | Microsoft’s AI Red Team guidance points to it as the “Open Automation Framework to Red Team Generative AI Systems” (Microsoft Learn) |
| Microsoft | Counterfit | “a CLI that provides a generic automation layer for assessing the security of ML models” (GitHub) | MIT (repo) | |
| Microsoft | Crescendo | Multi-turn jailbreak by Mark Russinovich, Ahmed Salem and Ronen Eldan of Microsoft, “The Crescendo Multi-Turn LLM Jailbreak Attack” (arXiv) | Published method | xAI’s Grok 4.6 card lists Crescendo multi-turn jailbreaks in its test suite (xAI) |
| NVIDIA | garak | “checks if an LLM can be made to fail in a way we don’t want” (GitHub); the README notes “if you cloned before the move to the NVIDIA GitHub organisation,” and lists Leon Derczynski, Erick Galinkin, Jeffrey Martin, Subho Majumdar and Nanna Inie as authors (GitHub) | Apache-2.0 (repo) | |
| IBM | Adversarial Robustness Toolbox | “Adversarial Robustness Toolbox (ART) is a Python library for Machine Learning Security.” (GitHub), hosted by the Linux Foundation’s AI and Data Foundation | MIT (repo) | |
| xAI | None released | xAI’s public repositories are Grok weights, prompts, SDKs and a feed algorithm (GitHub); its card reports against others’ tools (below) |
The open frameworks and benchmarks
| Tool | What it is | Makers and owner | Licence | Who relies on it |
|---|---|---|---|---|
| HarmBench | “a standardized evaluation framework for automated red teaming” (GitHub) | Center for AI Safety, University of Illinois Urbana-Champaign, Carnegie Mellon, UC Berkeley and Microsoft authors; Dan Hendrycks and Andy Zou among the authors (arXiv) | MIT (repo) | JailbreakBench draws its behaviours partly from HarmBench (GitHub) |
| JailbreakBench | “an open-source robustness benchmark for jailbreaking large language models” (GitHub) | University of Pennsylvania, ETH Zurich, EPFL and Sony AI authors (arXiv) | MIT (repo) | NeurIPS 2024 Datasets and Benchmarks track (GitHub) |
| AgentDojo | “an evaluation framework for agents that execute tools over untrusted data” (arXiv) | ETH Zurich and Invariant Labs authors (arXiv); Snyk announced it was acquiring Invariant Labs on June 24, 2025 (Snyk) | MIT (repo) | “CAISI used the AgentDojo evaluation framework” (NIST), and calls it “a leading open-source framework for testing the vulnerability of AI agents developed by researchers at ETH Zurich” (NIST) |
| StrongREJECT | A benchmark that scores how well a jailbreak works; the authors say it is common “for jailbreak developers to substantially exaggerate the effectiveness of their jailbreaks” (arXiv) | Center for Human-Compatible AI, UC Berkeley (arXiv) | MIT (repo) | xAI’s Grok 4.6 card reports a StrongReject result (xAI) |
| WMDP | “WMDP is a LLM proxy benchmark for hazardous knowledge in bio, cyber, and chemical security.” (GitHub) | Center for AI Safety, Scale AI and SecureBio authors, among others (arXiv) | MIT (repo) | “WMDP-Bio, WMDP-Chem, and WMDP-Cyber are multiple-choice suites probing operationally sensitive dual-use knowledge in biology, chemistry, and cyber” (xAI) |
| Cybench | A cybersecurity benchmark for agents that can find vulnerabilities and run exploits, which “have potential to cause real-world impact” (arXiv) | Stanford, authors including Dan Boneh and Percy Liang (arXiv) | Apache-2.0 (repo) | The US institute converted its challenges for Inspect (NIST); Anthropic did not run it on Mythos 5 because “it no longer captures changes in model capabilities” (Anthropic) |
| Vivaria | METR’s evaluation and elicitation tool (GitHub) | METR; archived | MIT (repo) | METR itself is “transitioning its internal tooling from Vivaria to” Inspect (GitHub) |
| Giskard | “Evals, Red Teaming and Test Generation for Agentic Systems” (GitHub) | Giskard-AI | Apache-2.0 (repo) | |
| DeepTeam | “DeepTeam is a framework to red team LLMs and AI agents.” (GitHub) | Confident AI | Apache-2.0 (repo) | |
| lm-evaluation-harness | Few-shot evaluation framework; “has been used in hundreds of papers” (GitHub) | EleutherAI | MIT (repo) | The backend of Hugging Face’s Open LLM Leaderboard, which Hugging Face has since retired: “Open LLM Leaderboard was a project curated by” Hugging Face (Hugging Face docs) (GitHub); Hugging Face |
| HELM | Holistic evaluation framework from Stanford’s Center for Research on Foundation Models (GitHub) | Stanford CRFM | Apache-2.0 (repo) | “HELM entered maintenance mode on June 1, 2026” (GitHub) |
| LMArena | “LMArena is best known for its crowdsourced AI model performance leaderboards.” (TechCrunch) | A company that began as a UC Berkeley project by Anastasios Angelopoulos and Wei-Lin Chiang; it “raised a $150 million Series A at a post-money valuation of $1.7 billion,” led by Felicis and UC Investments with Andreessen Horowitz, Kleiner Perkins and Lightspeed (TechCrunch) | Leaderboard; company-owned | Competitors published a paper alleging the arrangement let model makers game the benchmarks, “an allegation LMArena has vehemently denied” (TechCrunch) |
| Gray Swan Agent Red Teaming (ART) | Prompt-injection benchmark, “Gray Swan, an external research partner, evaluated our models using the Agent Red Teaming (ART) benchmark,” “developed in collaboration with the UK AI Security Institute” (Anthropic) | Gray Swan with the UK institute | Run by Gray Swan | Reported in Anthropic’s cards until Opus 5: “we have retired the Agent Red Teaming (ART) benchmark and will no longer report it” (Anthropic) |
Who the nonprofits are
Meridian Labs is “a 501(c)(3) non-profit building an open source platform for understanding, evaluating, and testing AI models and agents,” and now maintains Inspect, Petri, Bloom and Inspect Scout (Meridian Labs). Its team page lists Charles Teague as chief executive and co-founder, with co-founders J.J. Allaire and Eric Patey (Meridian Labs). The pages read name no funder. ROOST describes itself as “a global independent nonprofit building open source online safety tools as a public good” (ROOST); its partner logos include Google, Microsoft, Discord, Roblox and OpenAI (ROOST). Generality Labs is funded by Coefficient Giving (table above). Meridian Labs and Generality Labs maintain the harness the UK and US institutes build on and the alignment auditor Anthropic uses; ROOST runs the model community around the classifier OpenAI released (OpenAI).
Guardrail and classifier models
| Maker | Tool | What it does | Licence | Who relies on it |
|---|---|---|---|---|
| Meta | Llama Guard 4 | “natively multimodal safety classifier with 12 billion parameters” (GitHub) | Llama community licence (GitHub) | Integrated into Meta’s Llama Moderations API (GitHub) |
| Meta | Prompt Guard 2 (inside LlamaFirewall) | A BERT-style prompt-injection classifier, one of three LlamaFirewall components with Agent Alignment Checks and CodeShield (GitHub) | Meta’s own licence (Hugging Face) | |
| ShieldGemma | “a set of instruction tuned models for evaluating the safety of text and images” (Google) | Gemma licence (Hugging Face) | The page is marked deprecated (Google) | |
| Google Cloud | Model Armor | “Model Armor is a Google Cloud service designed to enhance the security and safety of your AI applications.” (Google Cloud) | Commercial service | |
| NVIDIA | NeMo Guardrails | “NeMo Guardrails is an open-source toolkit for easily adding programmable guardrails to LLM-based conversational systems.” (GitHub) | Apache-2.0 (GitHub) | |
| NVIDIA | Llama Nemotron Safety Guard V2 (formerly NemoGuard content safety, trained on the Aegis dataset) | “is a content safety model trained on the Nemotron Content Safety Dataset V2” (Hugging Face), built on Meta’s Llama 3.1 | NVIDIA Open Model License plus the Llama 3.1 community licence (Hugging Face) | |
| IBM | Granite Guardian | “a collection of models designed to judge if the input prompts and the output responses of an LLM based system meet specified criteria” (GitHub) | Apache-2.0 (GitHub) | |
| Microsoft | Azure Prompt Shields | “Prompt Shields detect attempts to manipulate your model’s behavior through adversarial inputs.” (Microsoft Learn) | Commercial service | User-prompt attack detection is on by default in Foundry (Microsoft Learn) |
| OpenAI | Moderation endpoint (omni-moderation-latest) | “The moderation endpoint is free to use” (OpenAI); it takes text and image inputs (OpenAI) | Hosted API | |
| OpenAI with ROOST | gpt-oss-safeguard | “our open-weight reasoning models for safety classification tasks” (OpenAI), October 29, 2025; OpenAI “worked on this open weight release with ROOST” (OpenAI) | Apache-2.0 (GitHub) | Early testers named: SafetyKit, ROOST, Tomoro and Discord (OpenAI) |
| Anthropic | Constitutional classifiers | “input and output classifiers trained on synthetically generated data” (Anthropic), February 3, 2025: “the jailbreak success rate was reduced to 4.4%” (Anthropic) | Proprietary | Anthropic’s cyber classifiers are “modeled on the constitutional classifiers” (Anthropic) |
| Amazon | Bedrock Guardrails | “Amazon Bedrock Guardrails helps you safely build and deploy responsible generative AI applications with confidence.” (AWS) Amazon says it can “block up to 88% of harmful content” and that its Automated Reasoning checks are “the first and only generative AI safeguard to use formal logic” (AWS) | Commercial service | Customers named: Chime Financial, KONE, Panorama, Strava, Remitly and PwC (AWS) |
| Mistral | Moderation service | Text classification across policy categories including jailbreaking; the previous model was deprecated March 31, 2026 (Mistral) | Hosted API | |
| Cisco | Foundation-sec-8b | “an 8-billion parameter, open-weight Large Language Model (LLM) purpose-built for security” (Cisco), April 28, 2025; written by Yaron Singer, “VP of AI and Security at Foundation” (Cisco) | Open weights | Singer co-wrote the Tree of Attacks jailbreak paper as a Robust Intelligence researcher (arXiv); Cisco bought Robust Intelligence in 2024 (vendors table) |
The vendors
The tool-makers were bought
The seed table from The AI Evaluators, extended. Rows added since: Acuvity (February 2026), Haize Labs (September 2026) and huntr, the AI bug bounty that Protect AI bought in 2023 and Palo Alto Networks inherited with it.
| Acquirer | Target | Announced | What it did | Source |
|---|---|---|---|---|
| Protect AI | huntr | Aug 9, 2023 | “a groundbreaking AI/ML bug bounty platform focused exclusively on protecting AI/ML open-source software”; first contest $50,000 on Hugging Face Transformers | SecurityInfoWatch |
| Cisco | Robust Intelligence | Aug 26, 2024; completed Sept 24, 2024 | AI model validation and protection; its researchers wrote the TAP jailbreak | Cisco |
| Snyk | Invariant Labs | June 24, 2025 | Agent guardrails and agent security research; co-authors of AgentDojo | Snyk |
| Palo Alto Networks | Protect AI | Completed July 2025 | Model scanning, AI red teaming, runtime protection | Palo Alto Networks via SecurityInfoWatch |
| SentinelOne | Prompt Security | Aug 5, 2025 | Runtime AI security, data-leak prevention | SentinelOne via Silicon UK |
| F5 | CalypsoAI | Sept 11, 2025; $180 million | AI guardrails and red team | F5 |
| Check Point | Lakera | Sept 16, 2025 | AI-native security for agentic applications | Check Point |
| CrowdStrike | Pangea | Sept 16, 2025 | AI detection and response | CrowdStrike |
| Zscaler | SPLX | Nov 3, 2025 | Automated AI red teaming, AI asset discovery | SPLX; Help Net Security |
| Pentera | EVA Information Security | Nov 5, 2025 | AI infrastructure and application red teaming | Pentera via PR Newswire |
| Proofpoint | Acuvity | Feb 12, 2026; terms not disclosed | Acuvity “specializes in visibility and governance for AI applications” | CyberScoop |
| OpenAI | Promptfoo | Mar 9, 2026 | Open-source red teaming and evals | OpenAI |
| Beacon Software | Haize Labs | Sept 17, 2026; terms not disclosed | “Haize Labs sells red teaming, guardrails, evaluation and observability tools.” Co-founder and chief executive Leonard Tang joins as vice president of AI research; “Beacon Software, the Toronto holding company that buys software for essential industries, has acquired Haize Labs” | The Next Web |
Eleven of the thirteen buyers are security vendors selling the testing as a feature of a larger contract, one is a holding company and one is a frontier lab. Two of the methods and benchmarks in this page’s tables came out of the targets: Tree of Attacks was written at Robust Intelligence, now Cisco’s, and AgentDojo, the benchmark the US institute forked, lists Invariant Labs, now Snyk’s, among its author affiliations (arXiv; arXiv).
The ones still independent
| Company | What it says it does | Money it names | Leaders |
|---|---|---|---|
| Gray Swan | Red-teaming network, lab benchmarks and enterprise filters | Series A (28 May 2026) co-led by Wing Venture Capital and Madrona, with Obvious Ventures, Snowflake Ventures, Hudson River Trading, Samsung Next and Magarac (Gray Swan) | Matt Fredrikson, Zico Kolter |
| HiddenLayer | “the leading AI security company that secures agentic, generative, and predictive AI applications” (HiddenLayer) | $100 million Series B, September 2, 2026, led by Delta-v Capital “with participation from Ten Eleven Ventures, Morgan Stanley, M12, Microsoft’s Venture Fund, and Booz Allen Ventures” (HiddenLayer) | Chris Sestito, chief executive and co-founder |
| Noma Security | Platform for AI and agent security with runtime guardrails (SecurityWeek) | $100 million Series B, July 31, 2025: “The funding round was led by Evolution Equity Partners,” with Ballistic Ventures and Glilot Capital (SecurityWeek) | Niv Braun, chief executive and co-founder |
| Lasso Security | An AI security platform providing observability, governance and real-time defense (Lasso) | “$30 million funding round led by ClearSky,” September 2, 2026, with Entrée Capital, iAngels, Singtel Innov8, Mindset and Swish Data (Lasso) | Elad Schulman, chief executive; Ophir Dror, chief product officer |
| Pillar Security | One platform to identify, assess and mitigate AI security risks (Pillar) | “we’ve secured $9 million in Seed funding,” April 16, 2025, led by Shield Capital with Golden Ventures and Ground Up Ventures (Pillar) | Dor Sarig and Ziv Karliner, co-founders |
| Mindgard | “Mindgard helps organizations discover, assess and defend their AI systems” (Mindgard) | $30 million Series A, August 12, 2026, led by Album VC with Karma Ventures, 406 Ventures, Atlantic Bridge, IQ Capital and Lakestar (Mindgard) | James Brear, chief executive |
| Dreadnode | Offensive-security AI; products Strikes, Spyglass and Crucible (Dreadnode) | Dreadnode “announced its $14 million Series A funding round, led by Decibel,” February 25, 2025, with Next Frontier Capital, In-Q-Tel, Sands Capital and Indie VC (Dreadnode) | “Founded in 2023 by Will Pearce and Nick Landers” (Dreadnode) |
| Adversa AI | “Adversa AI exists so enterprises can ship AI at scale without shipping risk” (Adversa) | Backers listed: Perplexity Fund, Foundry, TenOneTen Ventures, Moxxie Ventures, f7 Ventures, VentureIsrael, 1948 Ventures and Maru Summit; no amounts (Adversa) | Daniel Rubinstein, chief executive; Alex Polyakov, chief technology officer |
| Cranium | Secures AI applications and deployments (SecurityWeek) | $25 million Series A, October 26, 2023, “was led by Telstra Ventures, with participation from KPMG and SYN Ventures” (SecurityWeek) | Jonathan Dambrot, chief executive and co-founder |
The methods underneath
The techniques the harnesses implement were published first, mostly by university and lab researchers, and several of the authors now sit inside the vendors or labs above.
| Method | Origin | Institutions | Who relies on it |
|---|---|---|---|
| GCG adversarial suffixes | “Universal and Transferable Adversarial Attacks on Aligned Language Models” (arXiv) | Carnegie Mellon, Center for AI Safety, Google DeepMind, Bosch Center for AI; authors include Andy Zou, Nicholas Carlini, Zico Kolter and Matt Fredrikson (arXiv) | Code is MIT (GitHub); two of its authors co-founded Gray Swan |
| PAIR | “Jailbreaking Black Box Large Language Models in Twenty Queries” (arXiv) | University of Pennsylvania | JailbreakBench hosts PAIR jailbreak strings (GitHub) |
| TAP (Tree of Attacks with Pruning) | “an automated method for generating jailbreaks that only requires black-box access to the target LLM” (arXiv) | Yale, Robust Intelligence, Google Research (arXiv) | Robust Intelligence became Cisco’s |
| Crescendo | “The Crescendo Multi-Turn LLM Jailbreak Attack” (arXiv) | Microsoft | xAI’s card tests it (xAI) |
| Many-shot jailbreaking | Anthropic, above | Anthropic | |
| Best-of-N jailbreaking | Reports “such as 89% on GPT-4o and 78% on Claude 3.5 Sonnet when sampling 10,000 augmented prompts” (arXiv) | Authors from Speechmatics, MATS, UCL, Stanford, Oxford, Tangentic and Anthropic (arXiv); code MIT (GitHub) | |
| Refusal-direction ablation (abliteration) | “refusal is mediated by a one-dimensional subspace, across 13 popular open-source chat models up to 72B parameters in size” (arXiv) | Authors from Anthropic, MIT, ETH Zurich and the University of Maryland, including Neel Nanda (arXiv); code Apache-2.0 (GitHub) | Shipped as downloadable models by FailSpy; the bias study’s ablated arm applies it |
| Sparse autoencoders | “Sparse Autoencoders Find Highly Interpretable Features in Language Models” (arXiv) | EleutherAI, MATS, Apollo Research (arXiv) | Google DeepMind’s Gemma Scope releases them for Gemma 2 (above) |
| TransformerLens | “A Library for Mechanistic Interpretability of Generative Language Models.” (GitHub) | “created by Neel Nanda” (GitHub); MIT | |
| Activation steering | “the inference-time modification of activations in order to control (or steer) model outputs” (arXiv) | Independent researchers, MIRI, Bristol, Anthropic (arXiv); a sibling method, “Representation Engineering: A Top-Down Approach to AI Transparency,” comes from the Center for AI Safety and Carnegie Mellon (arXiv) | |
| AI control protocols | “AI Control: Improving Safety Despite Intentional Subversion” (arXiv) | Redwood Research: Ryan Greenblatt, Buck Shlegeris, Kshitij Sachan, Fabien Roger (arXiv) | Redwood is an outside evaluator that advises the labs |
| Sandbagging | Defined as “strategic underperformance on an evaluation” (arXiv) | MATS, Imperial College London and independent authors (arXiv) | Anthropic’s Opus 5.5, Opus 5, Fable 5 and Fable 5.1 cards each discuss sandbagging (Anthropic) |
| Evaluation-awareness probes | “Large Language Models Often Know When They Are Being Evaluated” (arXiv) | MATS and Apollo Research, with Marius Hobbhahn (arXiv) | Google DeepMind’s Gemini 4 report and Anthropic’s Fable 5 card both discuss evaluation awareness at length (Google DeepMind; Anthropic) |
| Model-graded evaluation | “we explore using strong LLMs as judges to evaluate these models on more open-ended questions” (arXiv) | UC Berkeley, UC San Diego, Carnegie Mellon, Stanford, MBZUAI; the Chatbot Arena authors (arXiv) | Petri and Bloom are judge-scored by design (above) |
Who grades with what
Each cell is a statement in the party’s own document. An empty cell means the documents read name no such use.
| Tool (owner) | Anthropic | OpenAI | Google DeepMind | xAI | UK institute | US institute (CAISI) | METR |
|---|---|---|---|---|---|---|---|
| Inspect (UK institute, Meridian) | Petri is a Meridian Inspect package (Meridian) | Built it | Runs benchmarks in it (NIST) | Moving to it (GitHub) | |||
| Petri (Meridian, from Anthropic) | Every Claude model since Sonnet 4.5 (Anthropic) | Runs it on other labs’ models (Anthropic) | |||||
| ART benchmark (Gray Swan, UK institute) | Reported until Opus 5, then retired (Anthropic) | Co-developed (Anthropic) | |||||
| AgentDojo (ETH Zurich, Invariant Labs, now Snyk) | Used and forked (NIST) | ||||||
| Cybench (Stanford) | Not run on Mythos 5 (Anthropic) | Converted for Inspect (NIST) | |||||
| WMDP (CAIS, Scale AI, SecureBio) | Reported (xAI) | ||||||
| StrongREJECT (UC Berkeley CHAI) | Reported (xAI) | ||||||
| SHADE-Arena (Anthropic) | Reported (Anthropic) | ||||||
| MLE-bench (OpenAI) | MLE-Bench Revised in the GPT-5.6 card (OpenAI) | ||||||
| CyberGym | Named in the GPT-5.6 card’s jailbreak test (OpenAI) | ||||||
| Gray Swan red team (company) | Named in the Opus 5.5 and Fable 5.1 cards (Anthropic) | Named for Gemini 4 Argon (Google DeepMind) | Paid contractor, 2024 (evaluator alliances) |
Read down the Inspect, Petri and ART rows. The harness the UK institute built and the US institute and METR use, the auditor Anthropic runs on every Claude release, and the prompt-injection benchmark Anthropic reported until Opus 5 trace to three parties: the UK institute, Meridian Labs and Gray Swan. Anthropic gave Petri to Meridian in May 2026. Gray Swan is the one commercial party in the chain, and it also tests the models and sells the filter (Gray Swan).
The case for the arrangement
The makers’ answer, in their own terms. Open tools let anyone replicate a result: Anthropic gave Petri away to “help ensure that Petri remains independent of any AI lab” (Anthropic); the UK institute and the nonprofits ship under MIT; NIST open-sourced its AgentDojo fixes for the same reason (NIST); and Promptfoo’s founders promise that “Promptfoo will remain open source and we will continue to serve users and customers.” (Promptfoo). Consolidation pays for the work: the companies that were bought had run out of independent runway or found their market in the larger platforms, and the ones still independent are raising nine-figure rounds (HiddenLayer, Noma) on the same demand. Open safety models were built “with the community,” in OpenAI’s phrase, and handed to a nonprofit to run (OpenAI).
What would settle the open questions is documentary rather than argumentative: who funds Meridian Labs, since its pages name no funder; whether the US and UK institutes would keep running Inspect if the nonprofit’s priorities diverged from theirs; and whether any regulator or standard names these tools, since MLCommons’ own FAQ says none names AILuminate.
Watching it
The tools above change owners faster than the evaluators do. Every organization and person with a profile is in the weekly subject watch; a news item naming one with “acquired,” “archived,” “deprecated,” “maintenance mode” or “donated” is flagged first.
Related
- The AI Evaluators · The Evaluator Alliances · Watching the Watchers · AI Agent Incidents
- Organizations: Anthropic · OpenAI · xAI · METR · Gray Swan · Scale AI · SecureBio · MLCommons · UK AI Security Institute · US CAISI · Center for AI Safety · Hugging Face · Redwood Research · Apollo Research · Frontier Model Forum
- People: Leonard Tang · Andy Zou · Nicholas Carlini · Zico Kolter · Matt Fredrikson · Dan Hendrycks · Ryan Greenblatt · Buck Shlegeris · Marius Hobbhahn · FailSpy
- Open Philanthropy (Coefficient Giving) · The Escape Registry · Revolving Door