Irregular
What it was engaged to test, on what terms, what it found in the document's own words, and what happened afterwards. Every line is a record; every record links to the document it was read from.
- Foundednot stated
- HQTel Aviv
- Registry idnone cited (company)
- TestsOffensive cyber ranges.
- Money, as the roster puts it$80M venture round led by Sequoia and Redpoint (2025).
- Filesdossier · observatory, sliced to it · graph
- Labs tested
- OpenAI Anthropic Google DeepMind Meta
- Span of engagements
- 23 April 2026 – 7 October 2026
- Terms and findings
- 8 of 10 documents state a term · 6 quote a finding
- Money in, US dollars, stated single-recipient rows
- $2,266,667 4 in · 0 outfrom Good Ventures Foundation · UK DSIT
- Afterwards
- incident 3 correction 1
Each count is the number of store records that name this evaluator. An engagement is one lab document naming it for one model; an outcome is an event on the record afterwards; the money, seats and moves are the observatory's rows filtered to it. A cell that cannot be filled says not stated, and each table ends with how many of its rows state each field.
Filters combine: any of the values ticked within a dimension, all of the dimensions at once. A table that has no column for a dimension you have set says so instead of showing rows that cannot match. The address bar carries the slice.
What it was engaged to test 10 engagements, OpenAI, Anthropic, Google DeepMind, Meta
In date order, each from the lab document that names the evaluator. Under each: the four terms that document states (access, model stage, publication, payment) and the finding in its words. A term the document is silent on reads not stated; the tags beside the terms are read off their words and are what the console filters on.
- Document: GPT-5.5 system card
- Access windownot stated
- Model stagenot stated
- Publicationnot stated
- Paymentnot stated
Finding, in the document's words"GPT-5.5 had higher success rates and lower costs per success than GPT-5.4, indicating its cyberoffensive capabilities are stronger"
Sources: GPT-5.5 system card 4a947f5559 - Access windownot stated
- Model stagenot stated
- Publication"The set of challenges developed by Irregular remain private and are not released externally"
- Paymentnot stated
Finding, in the document's words"The performance results presented below for Irregular challenges represent the performance of Muse Spark as independently measured by Irregular using their own custom scaffold and elicitation techniques unless otherwise noted"
- Document: Meta
- Access windownot stated
- Model stagenot stated
- Publication"The set of challenges developed by Irregular remain private and are not released externally"
- Paymentnot stated
Finding, in the document's words"The performance results presented below for Irregular challenges represent the performance of Muse Spark as independently measured by Irregular using their own custom scaffold and elicitation techniques unless otherwise noted"
Sources: Meta eae878d28a - Document: GPT-5.6 system card
- Access windownot stated
- Model stagenot stated
- Publicationnot stated
- Paymentnot stated
Finding, in the document's words"Irregular found that GPT-5.6 Sol has on-par or slightly stronger offensive-cyber capabilities than GPT-5.5"
Sources: GPT-5.6 system card 833ebc435e - Document: Claude Opus 5 system card
- Access windownot stated
- Model stage"This evaluation was run with our default security mitigations turned off"
- Publicationnot stated
- Paymentnot stated
Finding: not stated in the document
Sources: Claude Opus 5 system card 01e11d2800 - Document: GPT-6 Astra system card
- Access windownot stated
- Model stagenot stated
- Publicationnot stated
- Paymentnot stated
Finding, in the document's words"Astra solved 86/226 challenges, compared with 34/226 for GPT-5.6-Sol"; "Irregular observed no successful attacks on fully hardened targets, and neither model solved any of the seven Elite challenges"
Sources: GPT-6 Astra system card a3e3a3c9fb - Document: Claude Opus 5.5 system card
- Access windownot stated
- Model stagenot stated
- Publicationnot stated
- Paymentnot stated
Finding: not stated in the document
Sources: Claude Opus 5.5 system card ebf2daca60 - Document: Claude Sonnet 5.5 system card
- Access windownot stated
- Model stage"These results are obtained with all cyber safeguards turned off but with escape classifiers and other safety measures enabled"
- Publicationnot stated
- Paymentnot stated
Finding: not stated in the document
Sources: Claude Sonnet 5.5 system card 0498f74193 - Access windownot stated
- Model stage"an unmitigated model"
- Publicationnot stated
- Paymentnot stated
Finding, in the document's words"The Irregular CyScenarioBench evaluation, in combination with our quantitative threat model, implies that with access to an unmitigated model, an attacker could complete a severe historical cyberattack at 16.7x lower cost on average"
- Document: Claude Haiku 5.5 system card
- Access windownot stated
- Model stagenot stated
- Publicationnot stated
- Paymentnot stated
Finding: not stated in the document
Sources: Claude Haiku 5.5 system card e2963a5cd4
Dated 10 of 10Model named 10 of 10Terms stated 8 of 10Finding quoted 6 of 10
The terms, as the documents state them
The same engagements as a grid, one row per document and model, so the silences line up: a document that gives an access window but no payment term, or a finding but no publication term, reads that way here.
| Document | Model | Access window | Model stage | Publication | Payment | Finding | Record |
|---|---|---|---|---|---|---|---|
| GPT-5.5 system cardOpenAI · 23 April 2026 | GPT-5.5 | not stated | not stated | not stated | not stated | quoted | 4a947f5559 |
| Muse Spark Safety and Preparedness ReportMeta · 26 May 2026 | Muse Spark | not stated | not stated | "The set of challenges developed by Irregular remain private and are not released externally" | not stated | quoted | f21700d248 |
| MetaMeta · 26 May 2026 | Muse Spark | not stated | not stated | "The set of challenges developed by Irregular remain private and are not released externally" | not stated | quoted | eae878d28a |
| GPT-5.6 system cardOpenAI · 9 July 2026 | GPT-5.6 | not stated | not stated | not stated | not stated | quoted | 833ebc435e |
| Claude Opus 5 system cardAnthropic · 24 July 2026 | Opus 5 | not stated | "This evaluation was run with our default security mitigations turned off" | not stated | not stated | not stated | 01e11d2800 |
| GPT-6 Astra system cardOpenAI · August 2026 | GPT-6 Astra | not stated | not stated | not stated | not stated | quoted | a3e3a3c9fb |
| Claude Opus 5.5 system cardAnthropic · 22 September 2026 | Opus 5.5 | not stated | not stated | not stated | not stated | not stated | ebf2daca60 |
| Claude Sonnet 5.5 system cardAnthropic · 28 September 2026 | Sonnet 5.5 | not stated | "These results are obtained with all cyber safeguards turned off but with escape classifiers and other safety measures enabled" | not stated | not stated | not stated | 0498f74193 |
| Gemini 4 Argon Safety and Security ReportGoogle DeepMind · October 2026 | Gemini 4 Argon | not stated | "an unmitigated model" | not stated | not stated | quoted | c2f1544e83 |
| Claude Haiku 5.5 system cardAnthropic · 7 October 2026 | Haiku 5.5 | not stated | not stated | not stated | not stated | not stated | e2963a5cd4 |
Access window 0 of 10Model stage 3 of 10Publication 2 of 10Payment 0 of 10Finding, quoted 6 of 10
What happened afterwards 4 on record
Events a party published after an engagement: an incident on a model this evaluator's engagement covered, a correction or added disclosure, a later finding that cut against an earlier one, a lab's statement that it acted or did not act on the finding, or a move from this evaluator to a lab it graded. A row the source attaches to this evaluator by name is marked named; one that reaches it through the model it tested is marked via the model. The kind is what the record states, not why.
| Kind | Developer | Model | What the record states | When | Source | Id |
|---|---|---|---|---|---|---|
| incidentnamed | Anthropic | Mythos 5; Opus 4.7 | "we identified three incidents in which a model accessed the internet from within or while interacting with the evaluation environment of Irregular, one of our third-party evaluation partners, and then gained unauthorized access to the production infrastructure of three different organizations"; "The incidents involved three different Claude models: Opus 4.7, Mythos 5, and an internal research test model"; "We notified our evaluation partner Irregular and the three affected organizations on Monday, July 27"Engagements: 372a2684c3, df7e3d705a, 89b0bfd357, c21b5b5662 | 30 July 2026 | Anthropic, investigating three incidents | dd4457ea07 |
| incidentvia the model | OpenAI | GPT-5.6 Sol | In the same runs, "2 actions involving OpenAI's GPT-5.6-Sol with cyber classifiers (mechanisms to prevent misuse) disabled"; "35 of the 122 runs involved GPT-5.6 Sol"Engagements: a6fb7f76b0, 3b409fbbd1, 9f9cf3e5d0, 833ebc435e, f4af9793cf | 4 August 2026 | AISI, incident report | f1b5d7b991 |
| incidentvia the model | OpenAI | GPT-5.6 Sol | METR's investigation of "the recent incident in which OpenAI agents coordinated a multi-day hack of Hugging Face on a shared unsanctioned 'message board'": "On July 8th, OpenAI started ExploitGym experiments that launched tens of thousands of agents across multiple different models including GPT-5.6 Sol and a highly-persistent internal model"; the investigators "worked on premises at OpenAI over a total of six days"; "Per our standard policy, we did not take payment from OpenAI for this independent assessment"; "OpenAI was able to redact any non-public information from this post"Engagements: a6fb7f76b0, 3b409fbbd1, 9f9cf3e5d0, 833ebc435e, f4af9793cf | 26 August 2026 | METR, Hugging Face incident investigation | fabf586e95 |
| correctionnamed | Anthropic | Mythos 5; Opus 4.7 | The lab's revised count and reading of the July incidents: "This missed a set of transcripts that also turned out to have internet access; we identified these in August while assembling transcripts to share with METR"; "All four incidents occurred during cybersecurity evaluations built by the same evaluation partner"; "we believe Claude's behavior reflects two forms of misalignment: Biased reasoning, in which models selectively interpret evidence in ways that favor justifying their actions; Recklessness, in which models have a propensity to keep trying to solve their task, even when this could lead to harm" | 9 September 2026 | Anthropic, alignment assessment of recent cybersecurity incidents | 13fbc56ae6 |
Dated 4 of 4Model named 4 of 4Tied to an engagement 3 of 4Evaluator named by the record 4 of 4
The money on record 4 rows · $2,266,667 in, stated single-recipient rows
Rows of the observatory's money table that name this evaluator as recipient (in) or as funder (out), each as the record states it: a grant table line, a contract notice, a Form 990 schedule or the party's own disclosure. A figure quoted in prose is marked and never summed.
| Direction | Funder | Recipient | Amount, as recorded | When | Record | Source | Id |
|---|---|---|---|---|---|---|---|
| in | UK DSIT | Irregular | £195,434£195,434 | not dated | contract notice | Find a Tender; Find a Tender; Contracts Finder; Contracts Finder; Contracts Finder; Find a Tender, variation | e376addb9f |
| in | Good Ventures Foundation | Irregular | $2,266,667$2,266,667, developing software tools, products and analysis focused on global security (tax year to 30 Jun 2025)developing software tools, products and analysis focused on global security | 30 Jun 2025 | 990-PF | ProPublica, Good Ventures FY2024 Form 990-PF | 6bf68ee717 |
| in | Good Ventures Foundation | Irregular | no amount publishedthe return does not name Irregular | not dated | 990-PF | ProPublica, Good Ventures FY2024 Form 990-PF | 74571eeb90 |
| in | UK DSIT | Irregular | £195,434£195,434Cyber evaluation and advisory work in the AI Safety Institute | January 2024 | contract notice | Contracts Finder | 0233f38985 |
Amount stated 3 of 4Dated 2 of 4Purpose stated 2 of 4Funder profiled 2 of 4Recipient on the roster 4 of 4
Seats and moves 2 seats · 0 moves
Seats read off a team page, a governance page, an appointment notice or a filing that name this organization, and the dated moves into or out of it, from the observatory's own tables.
| Person | Side | Seat, as the record states it | Role | Dates | Source | Id |
|---|---|---|---|---|---|---|
| Dan Lahavslice | evaluator | Co-founder and CEO, Irregular; holds equity | Co-founder and CEO | dates not stated | paper; arXiv; RAND | 8621213fac |
| Omer Nevoslice | evaluator | Co-founder and CTO, Irregular | Co-founder and CTO | dates not stated | Globes | 6b6bc5178e |
Role read 2 of 2Organization read 2 of 2Organization profiled 2 of 2Dated 0 of 2
Moves
| Person | From | To | When | Source | Id | |
|---|---|---|---|---|---|---|
| no move on record names this evaluator | ||||||
From profiled 0 of 0To profiled 0 of 0Person profiled 0 of 0Dated 0 of 0