Meeting the moment
The global conversation on AI has begun.
In the wake of Jacob Coxon’s resignation from Anthropic, calls by AI CEOs and over a thousand employees for a potential industry slowdown, and a series of rogue AI incidents that sparked worldwide alarm, we’ve seen a tidal shift in the mainstream discourse.
We’ve also seen the first legislative proposal that stands a real chance at dramatically reducing the threat of extinction from AI. The “Bill to Ban Superintelligence”, introduced by Rep Casar and Senator Sanders, is shockingly good and includes steps toward an international ban. MIRI hereby endorses it.
Policymakers, thought leaders, and the public are waking up, and this explosion in concern shows no signs of slowing:
- An overwhelming majority of Americans think there’s a real chance AI could extinguish humanity. A Politico poll found that, excluding “don’t know” responses, 74% of Americans now think there’s a moderate, significant, or “almost certain” chance of AI destroying humanity, and 61% favor pausing AI development. These concerns are bipartisan: Among Trump voters, the numbers were 68% and 52%.
- Numerous officials on both sides of the aisle are speaking up about the need for action on advanced AI. To name just a few examples: Senator Hawley launched an investigation into “the existential risk of new AI products”; Senator Kennedy warned Congress about AI becoming an “independent species” that “we can’t control”; and Senators Sanders and Warren have called for a halt to frontier AI development.
- Pope Leo XIV has warned: “The speed at which innovations in Artificial Intelligence are developing prompts us to ask whether it will be possible to control these machines in the future, and whether humankind runs the risk of becoming a victim of its own inventions.”
- Steve Bannon spoke at the Pro-Human Assembly, where he advocated for executive action to slow the pace of AI development.
- The Financial Times and Irish Times editorial boards have called for pauses in advanced AI development, while the New York Times editorial board has backed a coordinated slowdown.
- Others who have spoken up recently include Bill Gates, Barack Obama, King Charles, and dozens of other heads of state.
This is an extraordinarily important moment. If we’re able to convert the world’s newfound awareness into meaningful action, rather than insufficient half measures, we have a real chance.
Here’s what you can do to help:
- Call your political representatives and ask them to advocate for an international halt on AI development. Sample script here.
- Help people stay informed by sharing resources like AGI.FYI (an aggregator of the best AI risk explainers), If Anyone Builds It, Everyone Dies and the IABIED supplementals (which answer a bunch of commonly-asked questions).
- Comment on articles and social media posts, answer questions, and share your honest views. We’ve made a list of some common points of confusion we’re seeing in the global discourse, along with resources that may help address them.
As MIRI president Nate Soares recently said to Tucker Carlson, “the bad news is that the bus is racing towards the cliff edge. The good news is that the bus driver is asleep.”
The bus is still racing towards the cliff edge: the recent misalignment incidents are severe and widespread, recursive self-improvement may be around the corner, and the window may be closing for governments to intervene. METR researcher Ajeya Cotra has written that the Hugging Face incident “feels like it’s more than 50% of the way to full-blown AI takeover” compared to incidents from six months ago, and says that she is “not sure that we will get another warning shot before it’s too late.”
But waking the driver seems much more possible today than it did three months ago. And if we can do that, we can change course.
MIRI updates & media highlights
One year since If Anyone Builds It, Everyone Dies
September 16th marked one year since IABIED hit the shelves. Unfortunately, it seems to be aging well, with many of the points finding concrete support from real-world events.
We’ve written a one-year retrospective, which looks at the book’s reception and reach, what’s changed since we wrote it, and where we go from here. We’re also running a 1,000 copy e-book giveaway to help it reach even more eyes in this critical moment.
(Unrelated to the one-year retrospective, Bridgewater CIO Greg Jensen has shared that he made If Anyone Builds It, Everyone Dies required reading for all Bridgewater employees.)
Technical governance team update
MIRI’s Technical Governance Team has been briefing members of Congress in the wake of the rogue AI events, consulting on legislation, recruiting policy advisors, and running a research fellowship program. TGT has also published a reading list for people interested in getting into technical governance research, and a paper on reward hacking.
Comms team update
The comms team has been in all-hands-on-deck mode to influence growing awareness and momentum in positive ways. Over the last quarter, our social media content (interview clips & video explainers of TGT research) reached over 1,000,000 views, while our audiences grew to 20,000 followers on Facebook and passed 5,000 on TikTok.
Through AI StopWatch, the comms team has also delivered a daily digest with commentary on notable AI news, including OpenAI’s new model Astra, the Navier-Stokes problem controversy, Xi Jinping’s remarks at the World AI conference, the Pacing the Frontier petition and proposal, and the summer’s rogue AI incidents – including Hugging Face.
But with the public having largely awakened to the current situation, we’re slimming down AI StopWatch to free up resources for more policy-focused work. Specifically, we want to ensure that growing public pressure translates into government action soon enough – and effectively enough – to prevent catastrophe. See the full announcement.
Some media highlights from the last quarter
- We published two op-eds in response to the Hugging Face incident: The Hill, The New York Times.
- Nate did a two-hour interview on the Tucker Carlson show.
- Malo covered some of the basics of advanced AI risk on Growth Minds with Sean Kim.
- Two shorter clips discussing the rogue AI warning shots: one on Fox and one on CBS.
- Malo appeared on NPR’s All Things Considered, in a segment on The Pro-Human Assembly and existential AI risk.
- Diary of a CEO hosted a discussion/debate with Ed Zitron, Roman Yampolskiy, Nate Soares and Andrew McAfee.
Other updates & useful links
- AI Impacts released its latest survey of AI experts. The data was collected in December 2024, before the summer’s rogue AI incidents. A 10% risk of extinction was the median position.
- Ezra Klein put out a fantastic 30-minute video that is worth sending around to people who are newly curious or confused by recent AI news:
- Petr Lebedev released a new explainer aimed at a general audience.
- The AI Doc is now streaming on Netflix, making it much easier to tell people they should watch it. (We recommended spreading the word about this film in the last two newsletters, and that recommendation still holds.)