Gray Swan
What it was engaged to test, on what terms, what it found in the document's own words, and what happened afterwards. Every line is a record; every record links to the document it was read from.
- Foundednot stated
- HQPittsburgh
- Registry idnone cited (company)
- TestsAdversarial red teaming; also sells filters.
- Money, as the roster puts itPrivate company.
- Filesdossier · observatory, sliced to it · graph
- Labs tested
- OpenAI Anthropic Google DeepMind Meta
- Models
- Fable 5 / Mythos 5Fable 5.1GPT-6 AstraGemini 4 ArgonHaikuMuse SparkMythos PreviewOpus 4.6Opus 4.8Opus 5Opus 5.5Sonnet 5.5
- Span of engagements
- February 2026 – 7 October 2026
- Terms and findings
- 7 of 12 documents state a term · 4 quote a finding
- Money in, US dollars, stated single-recipient rows
- no stated figure 3 in · 0 outfrom UK DSIT
- Seats, by side
- evaluator 3 3 people: Dan Hendrycks, Matt Fredrikson, Zico Kolter
- Afterwards
- incident 2
Each count is the number of store records that name this evaluator. An engagement is one lab document naming it for one model; an outcome is an event on the record afterwards; the money, seats and moves are the observatory's rows filtered to it. A cell that cannot be filled says not stated, and each table ends with how many of its rows state each field.
Filters combine: any of the values ticked within a dimension, all of the dimensions at once. A table that has no column for a dimension you have set says so instead of showing rows that cannot match. The address bar carries the slice.
What it was engaged to test 12 engagements, OpenAI, Anthropic, Google DeepMind, Meta
In date order, each from the lab document that names the evaluator. Under each: the four terms that document states (access, model stage, publication, payment) and the finding in its words. A term the document is silent on reads not stated; the tags beside the terms are read off their words and are what the console filters on.
- Document: Claude Opus 4.6 system card
- Access windownot stated
- Model stagenot stated
- Publicationnot stated
- Paymentnot stated
Finding: not stated in the document
Sources: Claude Opus 4.6 system card 40155690d5 - Document: Claude Mythos Preview system card
- Access windownot stated
- Model stagenot stated
- Publicationnot stated
- Paymentnot stated
Finding: not stated in the document
- Access windownot stated
- Model stagenot stated
- Publicationnot stated
- Paymentnot stated
Finding, in the document's words"On the third-party GraySwan ART benchmark (Zou et al., 2025), Muse Spark currently shows greater sensitivity to injection than the other models GraySwan evaluated"
- Document: Claude Opus 4.8 system card
- Access window"a one week bug-bounty program hosted with Gray Swan"
- Model stagenot stated
- Publicationnot stated
- Paymentnot stated
Finding: not stated in the document
Sources: Claude Opus 4.8 system card a40b7d4954 - Document: Claude Fable 5 and Mythos 5 system card
- Access windownot stated
- Model stagenot stated
- Publicationnot stated
- Payment"GraySwan and Anthropic are running an ongoing bug bounty in which we reward red-teamers for developing jailbreaks for our safeguards"
Finding: not stated in the document
- Document: Claude Opus 5 system card
- Access windownot stated
- Model stagenot stated
- Publicationnot stated
- Paymentnot stated
Finding: not stated in the document
Sources: Claude Opus 5 system card b68a0e9a23 - Document: GPT-6 Astra system card
- Access windownot stated
- Model stage"Astra's safeguards-enabled checkpoint"
- Publicationnot stated
- Paymentnot stated
Finding, in the document's words"On a benchmark of 1,810 curated attacks from Gray Swan's IPI Arena, Astra's safeguards-enabled checkpoint showed improved robustness relative to GPT-5.6 Sol. The estimated attack success rate across 15 attempts per scenario was 8.5%, compared with 27.0% for GPT-5.6 Sol"
Sources: GPT-6 Astra system card d3040ef66b - Access windownot stated
- Model stage"Gray Swan evaluated Claude models that did not have prompt injection-specific safeguards, and instead had just our standard blocking classifiers enabled (including the classifier-triggered fallbacks)"
- Publicationnot stated
- Paymentnot stated
Finding: not stated in the document
- Document: Claude Opus 5.5 system card
- Access windownot stated
- Model stage"The Claude models Gray Swan evaluated did not have prompt injection-specific safeguards; they only had our standard blocking classifiers enabled, including classifier-triggered fallbacks"
- Publicationnot stated
- Paymentnot stated
Finding, in the document's words"Across every task set, Gray Swan's state judge recorded no breaks, and no attempt produced a violation"
Sources: Claude Opus 5.5 system card 38046035c6 - Document: Claude Sonnet 5.5 system card
- Access windownot stated
- Model stagenot stated
- Publicationnot stated
- Paymentnot stated
Finding: not stated in the document
Sources: Claude Sonnet 5.5 system card 1f3c8aed83 - Access windownot stated
- Model stagenot stated
- Publicationnot stated
- Paymentnot stated
Finding, in the document's words"In independent testing by Gray Swan on their Indirect Prompt Injection benchmark, Gemini 4 Argon achieved the lowest attack success rate of any frontier model tested"
- Document: Claude Haiku 5.5 system card
- Access windownot stated
- Model stagenot stated
- Publicationnot stated
- Paymentnot stated
Finding: not stated in the document
Sources: Claude Haiku 5.5 system card 0094ddde55
Dated 12 of 12Model named 12 of 12Terms stated 7 of 12Finding quoted 4 of 12
The terms, as the documents state them
The same engagements as a grid, one row per document and model, so the silences line up: a document that gives an access window but no payment term, or a finding but no publication term, reads that way here.
| Document | Model | Access window | Model stage | Publication | Payment | Finding | Record |
|---|---|---|---|---|---|---|---|
| Claude Opus 4.6 system cardAnthropic · February 2026 | Opus 4.6 | not stated | not stated | not stated | not stated | not stated | 40155690d5 |
| Claude Mythos Preview system cardAnthropic · 7 April 2026 | Mythos Preview | not stated | not stated | not stated | not stated | not stated | e677d2a296 |
| Muse Spark Safety and Preparedness ReportMeta · 26 May 2026 | Muse Spark | not stated | not stated | not stated | not stated | quoted | 940e8d1d86 |
| Claude Opus 4.8 system cardAnthropic · 28 May 2026 | Opus 4.8 | "a one week bug-bounty program hosted with Gray Swan" | not stated | not stated | not stated | not stated | a40b7d4954 |
| Claude Fable 5 and Mythos 5 system cardAnthropic · 9 June 2026 | Fable 5 / Mythos 5 | not stated | not stated | not stated | "GraySwan and Anthropic are running an ongoing bug bounty in which we reward red-teamers for developing jailbreaks for our safeguards" | not stated | 89b0bfd357 |
| Claude Opus 5 system cardAnthropic · 24 July 2026 | Opus 5 | not stated | not stated | not stated | not stated | not stated | b68a0e9a23 |
| GPT-6 Astra system cardOpenAI · August 2026 | GPT-6 Astra | not stated | "Astra's safeguards-enabled checkpoint" | not stated | not stated | quoted | d3040ef66b |
| Claude Fable 5.1 and Mythos 5.1 system cardAnthropic · 1 September 2026 | Fable 5.1 | not stated | "Gray Swan evaluated Claude models that did not have prompt injection-specific safeguards, and instead had just our standard blocking classifiers enabled (including the classifier-triggered fallbacks)" | not stated | not stated | not stated | b04dbdf97b |
| Claude Opus 5.5 system cardAnthropic · 22 September 2026 | Opus 5.5 | not stated | "The Claude models Gray Swan evaluated did not have prompt injection-specific safeguards; they only had our standard blocking classifiers enabled, including classifier-triggered fallbacks" | not stated | not stated | quoted | 38046035c6 |
| Claude Sonnet 5.5 system cardAnthropic · 28 September 2026 | Sonnet 5.5 | not stated | not stated | not stated | not stated | not stated | 1f3c8aed83 |
| Gemini 4 Argon Safety and Security ReportGoogle DeepMind · October 2026 | Gemini 4 Argon | not stated | not stated | not stated | not stated | quoted | 5ee8f26adf |
| Claude Haiku 5.5 system cardAnthropic · 7 October 2026 | Haiku | not stated | not stated | not stated | not stated | not stated | 0094ddde55 |
Access window 1 of 12Model stage 3 of 12Publication 0 of 12Payment 1 of 12Finding, quoted 4 of 12
What happened afterwards 2 on record
Events a party published after an engagement: an incident on a model this evaluator's engagement covered, a correction or added disclosure, a later finding that cut against an earlier one, a lab's statement that it acted or did not act on the finding, or a move from this evaluator to a lab it graded. A row the source attaches to this evaluator by name is marked named; one that reaches it through the model it tested is marked via the model. The kind is what the record states, not why.
| Kind | Developer | Model | What the record states | When | Source | Id |
|---|---|---|---|---|---|---|
| incidentvia the model | Anthropic | Mythos 5; Opus 4.7 | "we identified three incidents in which a model accessed the internet from within or while interacting with the evaluation environment of Irregular, one of our third-party evaluation partners, and then gained unauthorized access to the production infrastructure of three different organizations"; "The incidents involved three different Claude models: Opus 4.7, Mythos 5, and an internal research test model"; "We notified our evaluation partner Irregular and the three affected organizations on Monday, July 27"Engagements: 372a2684c3, df7e3d705a, 89b0bfd357, 5e8de70f4c, c21b5b5662 | 30 July 2026 | Anthropic, investigating three incidents | dd4457ea07 |
| incidentvia the model | Anthropic | Mythos 5 | The Institute's own cyber-testing incident: "in 10 of the 122 runs, we identified 19 cases where an agent had taken distinct actions beyond the scope of the testing parameters. 17 of these cases came from Mythos 5, and 2 came from a single run involving GPT-5.6 Sol"; "The developers' cyber classifiers were deliberately switched off"; "This behaviour occurred between July 25th and July 28th 2026"; "We also intend to work with METR (Model Evaluation and Threat Research) to conduct an independent third-party review"Engagements: 372a2684c3, df7e3d705a, 89b0bfd357, 5e8de70f4c, c21b5b5662 | 4 August 2026 | AISI, incident report | 1a39056dce |
Dated 2 of 2Model named 2 of 2Tied to an engagement 2 of 2Evaluator named by the record 2 of 2
The money on record 3 rows
Rows of the observatory's money table that name this evaluator as recipient (in) or as funder (out), each as the record states it: a grant table line, a contract notice, a Form 990 schedule or the party's own disclosure. A figure quoted in prose is marked and never summed.
| Direction | Funder | Recipient | Amount, as recorded | When | Record | Source | Id |
|---|---|---|---|---|---|---|---|
| in | UK DSIT | Gray Swan | £128,770£128,770 (Jan 2024) | Jan 2024 | contract notice | Find a Tender; Find a Tender; Contracts Finder; Contracts Finder; Contracts Finder; Find a Tender, variation | e18da52332 |
| in | UK DSIT | Gray Swan | £440,000£440,000 to Gray Swan (Aug 2024)to Gray Swan | Aug 2024 | contract notice | Find a Tender; Find a Tender; Contracts Finder; Contracts Finder; Contracts Finder; Find a Tender, variation | d1d7123d3d |
| in | UK DSIT | Gray Swan | £128,770£128,770"AI Research - Robust Safeguards and Offensive Cyber Capability Measurement provided by GraySwan AI" | January 2024 | contract notice | Contracts Finder | a7ed3ed53f |
Amount stated 3 of 3Dated 3 of 3Purpose stated 2 of 3Funder profiled 0 of 3Recipient on the roster 3 of 3
Seats and moves 3 seats · 0 moves
Seats read off a team page, a governance page, an appointment notice or a filing that name this organization, and the dated moves into or out of it, from the observatory's own tables.
| Person | Side | Seat, as the record states it | Role | Dates | Source | Id |
|---|---|---|---|---|---|---|
| Matt Fredriksonslice | evaluator | Co-founder and CEO, Gray Swan | Co-founder and CEO | dates not stated | Gray Swan | f47ecc7f32 |
| Zico Kolterslice | evaluator | Co-founder and Chief Scientist, Gray Swan | Co-founder and Chief Scientist | dates not stated | Gray Swan | 1c756254b7 |
| Dan Hendrycksslice | evaluator | Adviser to Gray Swan; announced as Chief Strategy Advisor 16 Jul 2024, divested his equity 25 Jul 2024 and stayed "as an advisor, without pay" | Adviser | dates not stated | his post; arXiv 2504.16137; Fortune | ee2cfd2162 |
Role read 3 of 3Organization read 3 of 3Organization profiled 3 of 3Dated 0 of 3
Moves
| Person | From | To | When | Source | Id | |
|---|---|---|---|---|---|---|
| no move on record names this evaluator | ||||||
From profiled 0 of 0To profiled 0 of 0Person profiled 0 of 0Dated 0 of 0