GENERALITY LABS

Institutional Archetype

THE LIBRARIAN WHO GRADES THE EXAMS — Generality Labs keeps the shelves and checks the books. It maintains Inspect Evals, the library of benchmark implementations that runs on the UK AI Security Institute’s Inspect framework, and since 2026 it publishes audits of other people’s benchmarks: BixBench, LABBench2, FORTRESS and an internal Epoch benchmark. Its own account of the rationale is a single sentence from its chief executive: “I started Generality Labs because, when the time comes, I don’t think we’re going to be able to accurately measure the safety of dangerously capable AI systems” (Generality Labs, An update on Generality Labs). The organization that decides which implementation of a benchmark is the correct one is the one the UK AI Security Institute and CAISI staff pull from, and it was founded three months ago.

Mandate & Origin

“We’re an R&D lab that solves practical challenges facing AI safety organisations, focusing on AI evaluation” (Generality Labs). Its chief executive’s post gives the history: “Generality Labs is an AI safety lab based in London,” “founded in July 2026, spinning off from Arcadia Impact with $4.4M in seed funding from Coefficient Giving,” and “a team of 2 cofounders (James and Justin), and 8+ contractors who have previously worked for places like METR, CERN and Anthropic” (Generality Labs, An update). The post is signed “Justin Olive, co-founder & CEO.” The Inspect Evals repository calls it “a London-based nonprofit which builds tools for evaluating risks and mitigations in frontier AI” and lists five maintainers: Alexander Putilin, James Mann, Justin Olive, Matt Fisher and Tania Sadhani (GitHub, Inspect Evals; GitHub, MAINTAINERS). Generality’s site carries no team or about page, no registered name and no charity or company number.

How it relates to the AI Safety Engineering Taskforce. The project it maintains began inside Arcadia Impact. Arcadia’s About page: “Through the AI Safety Engineering Taskforce, we have worked with the UK AI Security Institute to maintain the Inspect Evals framework” (Arcadia Impact, About). The 2025 funding proposal Justin Olive wrote on Manifund says “This project is run by ASET (Arcadia Impact),” “ASET was jointly launched by Justin Olive (Head of AI Safety) and Alexa Abbas (Technical Project Manager),” that Inspect Evals “started as a joint project between the UK AISI, Vector Institute, and ASET,” and that “ASET has been contracted with the UK AISI since January ‘25 to manage Inspect Evals” (Manifund, Inspect Evals). Meridian Labs’s team page says its Technical Program Manager, Alexandra Abbas, “co-founded the AI Safety Engineering Taskforce in 2024, where she led the development of Inspect Evals” (Meridian Labs, Team). So the Taskforce was an Arcadia Impact programme, Abbas co-launched it with Olive and has since moved to Meridian, and Generality is the entity Olive and a co-founder spun out of Arcadia in July 2026 to hold the project; Generality’s own pages do not name Abbas. The same names sit in both organizations’ orbit: the repository’s approved-contributor list includes @alexandraabbas, @jjallaire, @epatey and @ransomr, handles that match Meridian’s Abbas, Allaire, Patey and Richardson by name (GitHub, APPROVED_CONTRIBUTORS; Meridian Labs, Team).

Legal form. Generality’s pages do not state one. Its Head of Strategy & Operations advertisement lists the post’s first project as “establishing a new legal entity and spinning off from our current parent organisation” (Generality Labs, Careers). The parent is registered: “Arcadia Impact is registered as a Charitable Incorporated Organisation in England and Wales. We were founded in 2022 as a Community Interest Company and converted into a Charity in February 2025,” with charity number 1212323 (Arcadia Impact, About). Companies House lists Arcadia Impact as CE039259, registered externally as 1212323, and Arcadia Impact CIC 14362810 as converted or closed on 27 February 2025 (Companies House, search); the Charity Commission register lists charity 1212323, Arcadia Impact, registered, latest income £1,638,270 (Charity Commission, register).

Funding & Backers

One funder is named for the seed, and it is the same funder that supports the parent. “Generality Labs has been awarded $4.4M of seed funding from Coefficient Giving” (Generality Labs). One role advertisement words it “$4.4M in seed funding from a US donor” and another “$4.4M in seed funding” with no funder (Generality Labs, Senior Product Engineer; Generality Labs, Head of Strategy & Operations). Coefficient Giving is the former Open Philanthropy.

• Coefficient Giving’s own record. Its grant-search feed returns no grant to Generality Labs, and none to Inspect Evals. It returns five grants to Arcadia Impact: $445,000 (January 2023, to London Effective Altruism Hub, “now Arcadia Impact”), £318,410 (March 2024), £1,256,600 (January 2025), £401,027 (October 2025) and “Arcadia Impact Alignment Project” (18 June 2026, no amount in the feed) (Coefficient Giving, Arcadia Impact search). Arcadia says: “Our primary funder is Coefficient Giving, which has awarded us over £8 million since we were founded (this includes secured grants not yet received)” (Arcadia Impact, About). Coefficient’s team page for Erin Robertson, who joined in May 2026, says “Previously, she founded and co-directed Arcadia Impact, an AI safety field-building charity” (Coefficient Giving, Erin Robertson).
• The government contract before the spin-off. The 2025 proposal: “ASET has been contracted with the UK AISI since January ‘25 to manage Inspect Evals,” with “£53k in project funding via a contract with AISI” expected for September to November 2025, “via the team which has funded the project since its inception,” a requested increase from 1.5 to 3.0 full-time staff cut to 1.2, and a request for $50,000 in bridge funding (Manifund, Inspect Evals). The page shows $50,000 raised, with comments from regrantors including Marius Hobbhahn of Apollo Research: “I’m happy to throw in 10k of my regranting money as a sign of my support, but I think the value they provide is clearly big enough that a large funder should support them with hundreds of thousands or millions of dollars in support.”
• Contract revenue. “Prior to spinning off, we delivered >$1M in contracts with the UK AISI, Epoch AI and METR,” including “Designing and building 20+ evals across AI R&D and loss of control for the UK AISI internal suite” and “Eval quality control projects for METR and Epoch” (Generality Labs, An update). It also lists “Research services” and “Advising” as services, and says it “conducted an independent audit of Epoch’s internal benchmark for continual learning (EBR-Bench)” (Generality Labs, An update). Its advertised pay bands are £100,000 to £200,000 (Applied Research Scientist and Head of Strategy) and £110,000 to £220,000 (Senior Product Engineer) (Generality Labs, Careers). No budget, no revenue split and no conflicts policy is published. See The EA to AI-Safety Funding Pipeline.

Institutional Voice & Intent

Generality’s account: “Our mission is to ensure the field can accurately evaluate the safety of advanced AI,” pursued “by building tools and standards which address common problems, and then driving adoption of these solutions where they are most needed - governments, AI safety labs, evaluators, and frontier labs” (Generality Labs, An update). Its method, from the Applied Research Scientist advertisement: “Our mission is to define best practices and roll them out across the field. We do this with a carrot-stick method: adoption is streamlined with nice tools and forward deployed scientists (carrot), and incentivised with audits (stick)” (Generality Labs, Applied Research Scientist).

Stated intent: Make the field’s evaluations accurate: maintain the open library of benchmark implementations, audit benchmarks for flaws, and set an evaluation standard.

Observed intent: Hold the repository the institutes and evaluators run, turn the maintenance experience into an auditing business, and make the audit the incentive to adopt its tools. Its plan for 2027 is “Independent audits of >50% of evaluations used in risk assessments of frontier AI models,” “Inspect Auditor is adopted by all major evaluators and eval developers,” and “Develop an Inspect Evaluation Standard,” with a certification program for auditors under consideration (Generality Labs, An update).

Gap: The structure is disclosed and the gap is the one the apparatus keeps showing. The library is hosted in the UK government’s GitHub organization under an MIT licence, the funder of the steward is named as one philanthropy, the steward was paid by the institute whose staff use the library, and it audits evaluators it has been paid by. Generality says its audits are independent and publishes them; the record cannot settle whether the client relationships bear on what is audited. For a body whose product is the finding that a benchmark is broken, a published conflicts policy would settle it, and none appears on its site.

Interest: Generality argues that evaluations are the weak link in AI safety measurement, that third-party audits of them should be routine, and that a standard and a certification should follow. What adoption would pay it is set out from its own pages; no fee schedule is published.

• Routine third-party audits of evaluations. Position: “Independent audits of >50% of evaluations used in risk assessments of frontier AI models” by 2027, with “3rd party audits for evaluation developers who want to catch flaws in their evals” as a service (Generality Labs, An update). What it pays: audits are one of its two service lines, and its auditing tool, Inspect Auditor, is to be rolled out across Inspect Evals; the carrot-stick description puts the audit as the incentive that drives adoption of its tools (Generality Labs, Applied Research Scientist). Best case, its own: “Evaluations for tracking dual-use capabilities (e.g. cyber, bio) are mostly saturated or low quality.” Convergence: evaluators and labs that cite benchmark numbers get an outside quality mark, Epoch AI is both a client and the subject of an audit, and the UK institute is client, user and co-founder of the library.
• A common evaluation standard and certified auditors. Position: “Develop an Inspect Evaluation Standard, which eval developers can use to help them in building high quality evaluations,” and “A certification program for evaluation auditors, to help professionalise and scale up the area” (Generality Labs, An update). What it pays: the party that writes the standard and runs the certification sits upstream of every eval developer and auditor who wants to meet it, and it already maintains the reference implementations. Best case, its own: “we simply cannot afford to keep being surprised.” Convergence: Transluce argues for an access and conflicts standard for evaluators of models, Generality for a quality standard for the evaluations themselves; the two standards address different objects, the model’s evaluator and the evaluation itself.
• A gated central library. Position: the repository now “only accept[s] PRs from pre-approved contributors,” with problems raised as issues and .eval logs uploaded for review (GitHub, Inspect Evals). What it pays: control of what counts as a correct implementation stays with the maintainers, whose list of approved contributors names staff of the UK institute, Apollo Research, SecureBio, Palisade Research, Lakera and Anthropic by handle. Best case, the maintainers’ own, from the proposal: “Research code is notoriously low quality, which provides a suboptimal starting point for meeting user requirements” (Manifund, Inspect Evals). Convergence: the institutes and evaluators that run the suite want implementations somebody else has already checked.

Interest is a documented fact about each position, not a claim about motive, and not a verdict that the argument is wrong. Each best case is stated at the strength its author states it.

Position in the Apparatus

• The library (2024 to 2026). The UK institute announced Inspect Evals on 13 November 2024: “Over the past few months, we’ve worked closely with Arcadia Impact and the Vector Institute on this new project, which makes available dozens of high quality open-source evaluations for safety research” (UK AISI, Announcing Inspect Evals). The repository was created 2 October 2024, is MIT licensed, has 706 stars and 461 forks, and its top contributors are Matt Fisher (603), Scott Simmons (215), Jay Bailey (149) and Tania Sadhani (113); the package on PyPI lists “UK AI Security Institute” as author and reached version 0.25.0 on 17 October 2026 (GitHub, API; PyPI, inspect-evals). The docs list 184 evaluations, 129 in the repository and 55 external, and feature research from the UK institute, Goodfire and SaferAI (Inspect Evals docs).
• Who uses it, in Generality’s words and theirs. Generality says “Industry heavyweights like Weights & Biases, X.ai, Hugging Face and Alibaba have built directly on Inspect Evals to bootstrap their own evals frameworks,” that “Our evals are frequently used by government agencies conducting pre-deployment testing exercises,” and that it is used “by organisations such as NIST, Hugging Face, Epoch AI, and the EU AI Office” (Generality Labs, Senior Product Engineer; Generality Labs, Head of Strategy & Operations). Five UK institute staff are quoted in its testimonials; one of them: “Inspect evals is core to the work I do as a researcher.” The CAISI repository’s Cybench page says “This code is based off the public cybench fork maintained by the UK AI Security Institute,” linking the Inspect Evals directory (GitHub, usnistgov/caisi-cyber-evals). See Meridian Labs, which maintains the Inspect framework the library runs on, and whose docs name the institute and Meridian as its developers (Inspect).
• The audits (September to October 2026). The blog lists, among others, “Auditing BixBench: Broken Tasks and Faulty Judging Hide Saturation” (4 September), “Auditing FORTRESS: A Multi-Judge Panel Doesn’t Guarantee Robustness” (18 September), “Auditing LABBench2: SeqQA2 is Functionally Saturated” (29 September) and “LABBench2 Audit” (7 October), whose summary says “Most of LABBench2’s task sets now offer little signal, due to saturation, questionable construct validity or sensitivity to the judge model” (Generality Labs, Blog). The BixBench summary: “flawed tasks and judge grading errors explain roughly 70% of failed submissions, and correcting for them can raise measured agent performance by up to 42 percentage points.” A cyber post says GLM-5.3-Flash “matched Claude Mythos Preview’s performance at about 6% of the cost” on ExploitBench.
• Inspect Evals Lint and Inspect Auditor. It says it released Inspect Evals Lint, “a library of automatic quality checks that we’ve built throughout 2 years of maintaining Inspect Evals,” and built Inspect Auditor, “a tool which automates quality checks for common issues like reward hacking vulnerabilities and false ground-truth labels” (Generality Labs, An update). Its organization on GitHub holds 18 public repositories, including inspect_audit, scanner_evaluation and hawk (GitHub, Generality-Labs).

Actions & Leadership Choices

CONDUCT: WATCH — CUSTODIAN AND AUDITOR OF THE SAME LIBRARY, ONE NAMED FUNDER. No wrongdoing is alleged. The pages say who runs the library, who funds the steward and which institutes use it, and the audits are public. The structural point is the one Meridian shares from the other side of the stack: the people who decide which version of a benchmark is correct hold the repository, and here they have also made grading the benchmarks a product.

• 13 November 2024. The UK institute announces Inspect Evals with Arcadia Impact and the Vector Institute (UK AISI).
• January 2025. The Taskforce is contracted by the UK institute to manage Inspect Evals (Manifund).
• February 2025. Arcadia Impact converts from a CIC to a charity (Arcadia Impact).
• 29 September 2025. Olive opens the Manifund bridge-funding proposal (Manifund).
• May 2026. Erin Robertson, founder and co-director of Arcadia, joins Coefficient Giving (Coefficient Giving).
• 18 June 2026. Earliest post on the blog, “Why Are Evaluations Broken?” (Generality Labs, Blog).
• July 2026. Generality Labs is founded, spinning off from Arcadia Impact (Generality Labs).
• September 2026. Publishes the BixBench, FORTRESS, ExploitBench and LABBench2 audits and findings (Generality Labs, Blog).
• 14 October 2026. Olive’s update post sets the Q4 plan and the 2027 targets (Generality Labs).

Alliance Map

Where ties on different axes run at the same time, the overlap is the finding. The library the UK institute announced was kept running by a charity whose primary funder is Coefficient Giving, under a contract from the institute, and its maintainers have spun out with seed money from the same funder to hold it and to audit benchmarks for evaluators who were also their clients; the founder of the parent now works at the funder. The documents show disclosed arrangements, not coordination.

Reach Assessment

Institutional: The default library of benchmark implementations for the UK institute’s Inspect framework, used by its staff and copied by CAISI, and an auditor of benchmarks that evaluators and labs cite. Memetic: Low outside the field; the audits are blog posts, the BixBench headline of up to 42 percentage points is its strongest public claim, and the library name travels through repositories, not release documents. Civilizational: If evaluation is the layer that tells governments and labs what a model does, whoever decides which implementation of a benchmark counts as correct sits upstream of every score run in it, and a body that also audits the benchmarks and drafts the standard sits upstream of the standard. Generality’s pages say the right things about the problem: the field cannot currently measure what it needs to, and it names its funder, its past clients and its tools. The gap is the usual one: one named philanthropy, a government contract history with the library’s main user, a short list of approved contributors and no published conflicts policy for an auditor whose clients include the audited.

Score basis. Atk 6 is reach: the README says Generality maintains Inspect Evals; five UK AI Security Institute staff are quoted on its page, one calling it “core to the work I do as a researcher”; CAISI’s Cybench code is “based off the public cybench fork maintained by the UK AI Security Institute”; the docs list 184 evaluations; and Generality says Hugging Face, Weights & Biases, X.ai and Alibaba built on it (GitHub, Inspect Evals; Generality Labs, Senior Product Engineer; GitHub, usnistgov/caisi-cyber-evals; Inspect Evals docs). The points under a higher number are that no statute or procurement standard names it, no lab release document found cites it and its audits are blog posts. Peers on atk 6: Transluce, Andon Labs, Redwood Research and Alignment Research Center; Meridian Labs holds 8 for maintaining the framework itself with five institutes and Anthropic’s card, and FAR.AI 7 on a named EU tender. Def 5 is funding insulation: Generality says it launched on $4.4M from Coefficient Giving, the parent’s primary funder, and before the spin-off the project was paid by a UK institute contract and the team did paid work for Epoch AI and METR (Generality Labs, An update; Arcadia Impact, About; Manifund, Inspect Evals). The points above 5 are that it names its funder and its clients and publishes its audits; the points below 6 are that no cap, split or conflicts policy is published. Peers on def 5: Meridian Labs on an undisclosed funder and Andon Labs; FAR.AI and Palisade Research hold 6 on a published cap or split. Hp 6 is entrenchment: the library has run since October 2024 under the UK government’s GitHub organization, ships a release every one to two weeks and carries 184 evaluations, with a contribution gate maintained by Generality (GitHub, API; Inspect Evals docs). The points off are that no statute installs it, Generality was founded in July 2026 and had no legal entity of its own when its Head of Strategy post was advertised. Peers on hp 6: Andon Labs, FAR.AI, Palisade Research and Transluce; Meridian Labs holds 7 on the framework in a government repository with 240 contributors.

Sources: Generality Labs — home; An update on Generality Labs — Generality Labs; Blog — Generality Labs; Careers — Generality Labs; Senior Product Engineer (Inspect Evals) — Generality Labs; Applied Research Scientist — Generality Labs; Head of Strategy & Operations — Generality Labs; UKGovernmentBEIS/inspect_evals — GitHub; MAINTAINERS — Inspect Evals; APPROVED_CONTRIBUTORS — Inspect Evals; Generality-Labs — GitHub; Inspect Evals documentation; inspect-evals — PyPI; Inspect — UK AI Security Institute and Meridian Labs; Announcing Inspect Evals — UK AISI; caisi-cyber-evals — NIST CAISI, GitHub; Inspect Evals — Manifund; About — Arcadia Impact; Team — Meridian Labs; Erin Robertson — Coefficient Giving; Coefficient Giving — Arcadia Impact grants search; Companies House — Arcadia Impact search; Charity Commission — register entry 1212323.