Research: MIRI and the Off-Switch Problem — Soares, Yampolskiy, Yudkowsky and the Case That AI Cannot Be Controlled
The Machine Intelligence Research Institute from its 2000 founding as the Singularity Institute to its 2024 turn from research to stopping AI development; Nate Soares and Eliezer Yudkowsky's bestseller and its reception; Roman Yampolskiy's argument that superintelligence cannot be controlled; and the corrigibility papers behind the off-switch problem. Every claim in the figures' own words.
Contents
Companion pages: The EA to AI-Safety Funding Pipeline (MIRI’s money) and the Eliezer Yudkowsky profile.
These are public intellectuals stating their own published positions. Each claim below is an institutional fact or the person’s own on-record words, attributed to the exact source.
1. MIRI: from accelerating AI to stopping it
MIRI’s history, in its own words: “2000 Eliezer Yudkowsky, along with Brian and Sabine Atkins, founded MIRI as the Singularity Institute for Artificial Intelligence,” initially with the goal of accelerating progress toward it; it ran the Singularity Summit; and in “2013 The organization changed its name from the Singularity Institute to the Machine Intelligence Research Institute” (MIRI, About).
Its mission now: “We do research and public outreach intended to help prevent human extinction from the development of artificial superintelligence (ASI).” And its conclusion: “Extinction from AI is a live possibility, and the only reasonable response is to stop AI development altogether, until such a time as the alignment problem has been solved” (MIRI, About).
The pivot. In its 2024 Mission and Strategy Update MIRI said it had become pessimistic that alignment would be solved in time and made broad public communication and policy work its top priority, aiming to persuade major governments to halt frontier AI development, while keeping a smaller technical-governance research team (MIRI, Jan 4 2024). The turn had been signposted in Yudkowsky’s April 2022 “Death With Dignity” post (LessWrong). Its December 2025 fundraiser, its first in six years, describes a communications team of “roughly seven full-time employees (if we include Nate and Eliezer)” (MIRI’s 2025 Fundraiser).
Leadership: Malo Bourgon, chief executive; Nate Soares, president; Eliezer Yudkowsky, co-founder (MIRI, Team). MIRI’s funding history, including the years Open Philanthropy supplied about 60 percent of its budget, is on the funding pipeline page.
2. Nate Soares and If Anyone Builds It, Everyone Dies
Before MIRI, Soares “worked as an engineer at Google and Microsoft, as a research associate at the National Institute of Standards and Technology, and as a contractor for the US Department of Defense.” He joined MIRI as a research fellow in 2014, ran it as executive director, and is now its president (MIRI, Nate Soares). He is lead author of the 2015 “Corrigibility” paper (section 4).
With Yudkowsky he wrote If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All, published by Little, Brown and Company on September 16, 2025, which the publisher bills as an “instant New York Times bestseller” (Hachette). Its thesis, in their words: “If any company or group, anywhere on the planet, builds an artificial superintelligence using anything remotely like current techniques, based on anything remotely like the present understanding of AI, then everyone, everywhere on Earth, will die” (as quoted by MIRI). They call for international coordination to halt large-scale general AI development.
The reception, both ways. Kirkus: “A timely and terrifying education on the galloping havoc AI could unleash—unless we grasp the reins and take control” (Kirkus). New Scientist’s Jacob Aron: the book is “blurbed by everyone from celebrities like Stephen Fry and Mark Ruffalo to policy heavyweights including Fiona Hill and Ben Bernanke. The problem is that, while compelling, the argument is fatally flawed” (New Scientist, Sep 8 2025). Bernanke now sits on Anthropic’s Long-Term Benefit Trust (Anthropic).
3. Roman Yampolskiy: control is not just unsolved, it is impossible
Roman V. Yampolskiy is a tenured associate professor of computer science and engineering at the University of Louisville’s Speed School and founding director of its Cyber Security Lab, often described as one of the founders of the AI-safety field (Newsweek). His 2024 book is AI: Unexplainable, Unpredictable, Uncontrollable (Chapman & Hall/CRC) (Routledge).
His argument: there is no evidence advanced AI can be safely controlled, and without proof it should not be built. In his words: “Less intelligent agents (people) can’t permanently control more intelligent agents (artificial superintelligences)” — “no such design is possible, it doesn’t exist” (Newsweek). And: “We are facing an almost guaranteed event with the potential to cause an existential catastrophe. This is not a low-risk, high-reward scenario, but a high-risk, negative-reward situation,” a line that originates in his 2022 working paper “Superintelligence Will Not Be Controlled Regardless of Its Level of Intelligence” (ResearchGate; SciTechDaily).
The number often attached to him comes from Lex Fridman’s introduction to their June 2024 conversation, which puts Yampolskiy’s estimate of AI killing all humans “at 99.99 and many more nines percent” (Lex Fridman Podcast #431, transcript). That is Fridman’s description of his view.
4. The off-switch problem: corrigibility
An AI is corrigible if it cooperates with what its makers regard as a corrective intervention, such as being shut down or having its goals changed, despite the default incentive of a goal-directed agent to resist.
- Utility indifference (2010): Stuart Armstrong’s proposal to make an agent indifferent between continuing and being shut down, credited in MIRI’s corrigibility work (MIRI, 2014).
- “Corrigibility” (2015): Soares, Fallenstein, Armstrong and Yudkowsky, at the AAAI-15 ethics and AI workshop. It analyzes shutdown-button utility functions that neither reward nor penalize the button press, and that carry the shutdown behavior through self-modification (MIRI).
- “Safely Interruptible Agents” (2016): Laurent Orseau of DeepMind and Armstrong, at UAI 2016.
The research program’s own verdict is the strongest evidence of how hard the problem is: the institute that named it concluded it would not be solved in time and turned to trying to stop the race instead.