evilrobots.lol / tech / evaluator record

Scale AI

Scale AI (SEAL). What it was engaged to test, on what terms, what it found in the document's own words, and what happened afterwards. Every line is a record; every record links to the document it was read from.

▣ THE RECORD for-profit Commercial benchmark and certification firms STORE 2026-10-23
  • Founded2016
  • HQSan Francisco
  • Registry idnone cited (company)
  • TestsUS AISI's first authorized third-party evaluator (2025); Humanity's Last Exam.
  • Money, as the roster puts itMeta bought 49% (2025).
  • Filesdossier · observatory, sliced to it · graph
Labs tested
Anthropic Meta
Models
Opus 4.8Ran FORTRESS on Muse Spark
Span of engagements
26 May 2026 – 28 May 2026
Terms and findings
1 of 2 documents state a term · 1 quote a finding
Money in, US dollars, stated single-recipient rows
no stated figure 0 in · 0 out
Seats, by side
evaluator 1 between 1 2 people: Alexandr Wang, Dan Hendrycks
Afterwards
staff move 1

Each count is the number of store records that name this evaluator. An engagement is one lab document naming it for one model; an outcome is an event on the record afterwards; the money, seats and moves are the observatory's rows filtered to it. A cell that cannot be filled says not stated, and each table ends with how many of its rows state each field.

▣ SLICE 6 / 6 records
Lab
Model
Engagement
Terms
Outcome
Person
year, month or day

Filters combine: any of the values ticked within a dimension, all of the dimensions at once. A table that has no column for a dimension you have set says so instead of showing rows that cannot match. The address bar carries the slice.

What it was engaged to test 2 engagements, Anthropic, Meta

In date order, each from the lab document that names the evaluator. Under each: the four terms that document states (access, model stage, publication, payment) and the finding in its words. A term the document is silent on reads not stated; the tags beside the terms are read off their words and are what the console filters on.

  1. 26 May 2026 Meta · Ran FORTRESS on Muse Spark benchmark co-build
    Document: Meta
    • Access windownot stated
    • Model stagenot stated
    • Publicationnot stated
    • Paymentnot stated
    Finding, in the document's words"Scale AI ran this benchmark against Muse Spark, which achieved an overall propensity score (rate of harmful action) of 24.5%"
    Afterwards: staff move
    Sources: Meta 412d9c8959
  2. 28 May 2026 Anthropic · Opus 4.8 benchmark co-build
    Document: Claude Opus 4.8 system card
    • Access windownot stated
    • Model stagenot stated
    • Publicationnot stated
    • Paymentnot stated
    no term stated

    Finding: not stated in the document

    Sources: Claude Opus 4.8 system card ea48a753ff

Dated 2 of 2Model named 2 of 2Terms stated 1 of 2Finding quoted 1 of 2

The terms, as the documents state them

The same engagements as a grid, one row per document and model, so the silences line up: a document that gives an access window but no payment term, or a finding but no publication term, reads that way here.

DocumentModelAccess windowModel stagePublicationPaymentFindingRecord
MetaMeta · 26 May 2026Ran FORTRESS on Muse Sparknot statednot statednot statednot statedquoted412d9c8959
Claude Opus 4.8 system cardAnthropic · 28 May 2026Opus 4.8not statednot statednot statednot statednot statedea48a753ff

Access window 0 of 2Model stage 0 of 2Publication 0 of 2Payment 0 of 2Finding, quoted 1 of 2

What happened afterwards 1 on record

Events a party published after an engagement: an incident on a model this evaluator's engagement covered, a correction or added disclosure, a later finding that cut against an earlier one, a lab's statement that it acted or did not act on the finding, or a move from this evaluator to a lab it graded. A row the source attaches to this evaluator by name is marked named; one that reaches it through the model it tested is marked via the model. The kind is what the record states, not why.

KindDeveloperModelWhat the record statesWhenSourceId
staff movenamedMetano model namedAlexandr Wang: Scale AI (CEO; remains Chairman) -> Meta (Chief AI Officer)Engagement: 412d9c8959Jun 2025Meta; Scale AIb3823f9ffa

Dated 1 of 1Model named 0 of 1Tied to an engagement 1 of 1Evaluator named by the record 1 of 1

The money on record 0 rows

Rows of the observatory's money table that name this evaluator as recipient (in) or as funder (out), each as the record states it: a grant table line, a contract notice, a Form 990 schedule or the party's own disclosure. A figure quoted in prose is marked and never summed.

DirectionFunderRecipientAmount, as recordedWhenRecordSourceId
no money row names this evaluator

Amount stated 0 of 0Dated 0 of 0Purpose stated 0 of 0Funder profiled 0 of 0Recipient on the roster 0 of 0

Seats and moves 2 seats · 1 moves

Seats read off a team page, a governance page, an appointment notice or a filing that name this organization, and the dated moves into or out of it, from the observatory's own tables.

PersonSideSeat, as the record states itRoleDatesSourceId
Dan Hendrycksslice evaluatoradviser to Scale AI (reported)adviserdates not statedFortunef76c28706b
Alexandr Wangslice betweenFounder and Chairman, Scale AI, the US institute's first authorized outside evaluatorFounder and Chairmandates not statedScale AI; Metadf2665bcaf

Role read 2 of 2Organization read 2 of 2Organization profiled 2 of 2Dated 0 of 2

Moves

PersonFromToWhenSourceId
Alexandr WangsliceScale AI (CEO; remains Chairman)→Meta (Chief AI Officer)Jun 2025Meta; Scale AI629f007abb

From profiled 1 of 1To profiled 1 of 1Person profiled 1 of 1Dated 1 of 1