News and links

• Geoffrey Hinton and John Hopfield were awarded this year’s Nobel Prize in Physics for their foundational contributions to machine learning. In a press conference following the announcement, Hinton said that “if you look around, there are very few examples of more intelligent things being controlled by less intelligent things, which makes you wonder whether when AI gets smarter than us, it’s going to take over control.”

• In “A Narrow Path,” a group of AI policy researchers outlines a set of proposals for avoiding human extinction from AI.Their plan involves preventing smarter-than-human systems from being developed within the next twenty years. We are skeptical about instituting a moratorium for a fixed amount of time, rather than halting until humanity is truly ready to proceed, but we appreciate that they chose a duration measured in decades rather than months.

• In “Machines of Loving Grace,” Anthropic CEO Dario Amodei explores the potential benefits of smarter-than-human AI and also briefly argues that a coalition of democracies should race to quickly build it. Responding with his own essay, Max Tegmark argues that “from a game-theoretic point of view, this race is not an arms race but a suicide race[…] Because we are closer to building AGI than we are to figuring out how to align or control it.” We agree with Tegmark’s point here.

• California Governor Gavin Newsom vetoed SB 1047, the bill which would have mandated risk assessments for some AI developers. Unlike most of the bill’s critics, Newsom argued that the bill actually didn’t go far enough: “By focusing only on the most expensive and large-scale models, SB 1047 establishes a regulatory framework that could give the public a false sense of security about controlling this fast-moving technology.” In “SB 1047: Our Side Of The Story,” Scott Alexander strongly challenges this rationale and reflects on the bill’s rise and fall.

• OpenAI announced o1, a model trained with reinforcement learning to do more complex reasoning than previous systems. The model was shared with red teaming organizations, but only one week before it was released. One of the organizations, Apollo Research, found that o1 acts deceptively in some situations.

• There was recently a string of new research and discussion related to how the ability of language models to predict the future compares to that of humans:
  • Researchers at CAIS claimed that, given the right prompt, GPT-4o could forecast at a “superhuman level.”
  • Forecasting platform Metaculus ran its own study which found that “no AI has demonstrated superhuman forecasting skill yet.”
    
  • Another group of researchers introduced a forecasting benchmark and used it to find that AI systems can forecast as well as random survey-takers, but not as well as expert forecasters.
    

You can subscribe to the MIRI Newsletter here.

MIRI updates

• Aaron Scher and Joe Collman have joined the Technical Governance Team at MIRI as researchers. Aaron previously did independent research related to sycophancy in language models and mechanistic interpretability, while Joe previously did independent research related to AI safety via debate and contributed to field-building work at MATS and BlueDot Impact.
• In an interview with PBS News Hour’s Paul Solman, Eliezer Yudkowsky briefly explains why he expects smarter-than-human AI to cause human extinction.
• In an interview with The Atlantic’s Ross Andersen, Eliezer discusses the reckless behavior of the leading AI companies, and the urgent need to change course.

News and links

• Google DeepMind announced a hybrid AI system capable of solving International Mathematical Olympiad problems at the silver medalist level. In the wake of this development, a Manifold prediction market significantly increased its odds that AI will achieve gold level by 2025, a milestone that Paul Christiano gave less than 8% odds and Eliezer gave at least 16% odds to in 2021.
• The computer scientist Yoshua Bengio discusses and responds to some common arguments people have for not worrying about the AI alignment problem.
• SB 1047, a California bill establishing whistleblower protections and mandating risk assessments for some AI developers, has passed the State Assembly and moved on to the desk of Governor Gavin Newsom, to either be vetoed or passed into law. The bill has received opposition from several leading AI companies, but has also received support from a number of employees of those companies, as well as many academic researchers. At the time of this writing, prediction markets think it’s about 50% likely that the bill will become law.
• In a new report, researchers at Epoch AI estimate how big AI training runs could get by 2030, based on current trends and potential bottlenecks. They predict that by the end of the decade it will be feasible for AI companies to train a model with 2e29 FLOP, which is about 10,000 times the amount of compute used to train GPT-4.
• Abram Demski, who previously worked at MIRI as part of our recently discontinued Agent Foundations research program, shares an update about his independent research plans, some thoughts on public vs private research, and his current funding situation.

You can subscribe to the MIRI Newsletter here.

Update (7-15-2024): this newsletter originally included Eliezer’s interview with Bloomberg’s Nate Lanxon and Jackie Davalos, which I mistakenly thought was released recently. That interview was actually released last year.

MIRI updates

• Rob Bensinger suggests that AI risk discourse could be improved by adopting a new set of labels for different perspectives on existential risk from AI. One drawback of “AI doomer” (a label sometimes used in online discussions) is that it does not have a consistent meaning.
• AI researcher John Wentworth guesses that a central difference between his and Eliezer Yudkowsky’s views might be that Eliezer expects AI to not use abstractions which are similar to those used by humans. Eliezer clarifies that he expects similar abstractions for predictive parts of the natural world, but larger differences for utility-laden concepts like human values and reflectivity-laden concepts like corrigibility. Also, Eliezer says that he is still concerned about the difficulty of reliably steering advanced ML systems even if they do use similar abstractions internally.
• Eliezer joins Joscha Bach, Liv Boeree, and Scott Aaronson for a panel discussion (partly paywalled) on the nature and risks of advanced AI.

News and links

• In an excellent new video essay, Rob Miles discusses major developments in AI capabilities, policy and discourse over the last year, how those developments have shaped the strategic situation, and what individuals can do to help.
• A report from Convergence Analysis gives a high-level overview of the current state of AI regulation. The report summarizes legislative text and gives short analyses for a range of topics.
• A group of current and former employees of OpenAI and Google DeepMind released an open letter calling on frontier AI companies to adopt policies that allow whistleblowers to share risk-related information. This comes in the wake of news that former OpenAI employees were subject to an extreme non-disclosure policy that the company has since apologized for and seems to have removed.

You can subscribe to the MIRI Newsletter here.

MIRI updates

• MIRI Communications Manager Gretta Duleba explains MIRI’s current communications strategy. We hope to clearly communicate to policymakers and the general public why there’s an urgent need to shut down frontier AI development, and make the case for installing an “off-switch”. This will not be easy, and there is a lot of work to be done. Some projects we’re currently exploring include a new website, a book, and an online reference resource.

• Rob Bensinger argues, contra Leopold Aschenbrenner, that the US government should not race to develop artificial superintelligence. “If anyone builds it, everyone dies.” Instead, Rob outlines a proposal for the US to spearhead an international alliance to halt progress toward the technology.

• At the end of June, the Agent Foundations team, including Scott Garrabrant and others, will be parting ways with MIRI to continue their work as independent researchers. The team was originally set up and “sponsored” by Nate Soares and Eliezer Yudkowsky. However, as AI capabilities have progressed rapidly in recent years, Nate and Eliezer have become increasingly pessimistic about this type of work yielding significant results within the relevant timeframes. Consequently, they have shifted their focus to other priorities.
  
  Senior MIRI leadership explored various alternatives, including reorienting the Agent Foundations team’s focus and transitioning them to an independent group under MIRI fiscal sponsorship with restricted funding, similar to AI Impacts. Ultimately, however, we decided that parting ways made the most sense.
  
  The Agent Foundations team has produced some stellar work over the years, and made a true attempt to tackle one of the most crucial challenges humanity faces today. We are deeply grateful for their many years of service and collaboration at MIRI, and we wish them the very best in their future endeavors.

• The Technical Governance Team responded to NIST’s request for comments on draft documents related to the AI Risk Management Framework. The team also sent comments in response to the “Framework for MItigating AI Risks” put forward by U.S. Senators Mitt Romney (R-UT), Jack Reed (D-RI), Jerry Moran (R-KS), and Angus King (I-ME).

• Brittany Ferrero has joined MIRI’s operations team. Previously, she worked on projects such as the Embassy Network and Open Lunar Foundation. We’re excited to have her help to execute on our mission.

News and links

• AI alignment researcher Paul Christiano was appointed as head of AI safety at the US AI Safety Institute. Last fall, Christiano published some of his thoughts about AI regulation as well as responsible scaling policies.
• The Superalignment team at OpenAI has been disbanded following the departure of its co-leaders Ilya Sutskever and Jan Leike. The team was launched last year to try to solve the AI alignment problem in four years. However, Leike says that the team struggled to get the compute it needed and that “safety culture and processes have taken a backseat to shiny products” at OpenAI. This seems extremely concerning from the perspective of evaluating OpenAI’s seriousness when it comes to safety and robustness work, particularly given that a similar OpenAI exodus occurred in 2020 in the wake of concerns about OpenAI’s commitment to solving the alignment problem.
• Vox’s Kelsey Piper reports that employees who left OpenAI were subject to an extremely restrictive NDA indefinitely preventing them from criticizing the company (or admitting that they were under an NDA), under threat of losing their vested equity in the company. OpenAI executives have since contacted former employees to say that they will not enforce the NDAs. Rob Bensinger comments on these developments here, strongly criticizing OpenAI for this policy.
• Korea and the UK co-hosted the AI Seoul Summit, a virtual mini-summit following up on the first AI Safety Summit (which took place in the UK last November). At the Seoul summit, 16 AI companies committed to create and publish safety frameworks, including “thresholds at which severe risks posed by a model or system, unless adequately mitigated, would be deemed intolerable.”
• California State Senator Scott Wiener’s SB 1047 passed in the California State Senate and is now being considered in the California State Assembly. The bill requires pre-deployment testing and post-deployment monitoring for models trained with 10^26 FLOP and $100M.

You can subscribe to the MIRI Newsletter here.

Update (5-15-2024): I wrote that “it appears that not all of the leading AI labs are honoring the voluntary agreements they made at [AI Safety Summit],” citing a Politico article. However, after seeing more discussion about it (e.g. here), I am now highly uncertain about whether the labs made specific commitments, what those commitments were, and whether commitments were broken. These seem like important questions, so I hope that we can get more clarity.

MIRI updates:

• MIRI is shutting down the Visible Thoughts Project.
  • We originally announced the project in November of 2021. At the time we were hoping we could build a new type of data set for training models to exhibit more of their inner workings. MIRI leadership is pessimistic about humanity’s ability to solve the alignment problem in time, but this was an idea that seemed relatively promising to us, albeit still a longshot.
  • We also hoped that the $1+ million bounty on the project might attract someone who could build an organization to build the data set. Many of MIRI’s ambitions are bottlenecked on executive capacity, and we hoped that we might find individuals (and/or a process) that could help us spin up more projects without requiring a large amount of oversight from MIRI leadership.
  • Neither hope played out, and in the intervening time, the ML field has moved on. (ML is a fast-moving field, and alignment researchers are working on a deadline; a data set we’d find useful if we could start working with it in 2022 isn’t necessarily still useful if it would only become available 2+ years later.) We would like to thank the many writers and other support staff who contributed over the last two and a half years.
• Mitchell Howe and Joe Rogero joined the comms team as writers. Mitch is a longtime MIRI supporter with a background in education, and Joe is a former reliability engineer who has facilitated courses for BlueDot Impact. We’re excited to have their help in transmitting MIRI’s views to a broad audience.
• Additionally, Daniel Filan will soon begin working with MIRI’s new Technical Governance Team part-time as a technical writer. Daniel is the host of two podcasts: AXRP, and The Filan Cabinet. As a technical writer, Daniel will help to scale up our research output and make the Technical Governance Team’s research legible to key audiences.
• The Technical Governance Team submitted responses to the NTIA’s request for comment on open-weight AI models, the United Nations’ request for feedback on the Governing AI for Humanity interim report. and the Office of Management and Budget’s request for information on AI procurement in government.
• Eliezer Yudkowsky spoke with Semafor for a piece about the risks of expanding the definition of “AI safety”. “You want different names for the project of ‘having AIs not kill everyone’ and ‘have AIs used by banks make fair loans.”

A number of important developments in the larger world occurred during the MIRI Newsletter’s hiatus from July 2022 to April 2024. To recap just a few of these:

• In November of 2022, OpenAI released ChatGPT, a chatbot application that reportedly gained 100 million users within 2 months of its launch. As we mentioned in our 2024 strategy update, GPT-3.5 and GPT-4 were more impressive than some of the MIRI team expected, representing a pessimistic update for some of us “about how plausible it is that humanity could build world-destroying AGI with relatively few (or no) additional algorithmic advances”. ChatGPT’s success significantly increased public awareness of AI and sparked much of the post-2022 conversation about AI risk.
• In March of 2023, the Future of Life Institute released an open letter calling for a six-month moratorium on training runs for AI systems stronger than GPT-4. Following the letter’s release, Eliezer wrote in TIME that a six-month pause is not enough and that an indefinite worldwide moratorium is needed to avert catastrophe.
• In May of 2023, the Center for AI Safety released a one-sentence statement, “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.” We were especially pleased with this statement, because it focused attention on existential risk in particular, and did so in a way that would be maximally understandable to policymakers and the general public. The list of signatories included the three most cited researchers in AI (Bengio, Hinton, and Sutskever) and leadership at all three of the leading AI labs (Anthropic, DeepMind, and OpenAI).
• In October of 2023, President Biden signed an executive order on AI. The order’s provisions include reporting requirements for some large models, rules for federal procurement of AI products, and a directive for the NIST to develop safety standards for generative AI.
• In November of 2023, the UK’s AI Safety Summit brought experts and world leaders together to discuss risks from AI. The summit showed some promise, but its outcomes so far have seemed limited. Six months later, it appears that not all of the leading AI labs are honoring the voluntary agreements they made at the summit.
• In March of 2024, the European Union passed the AI Act, a broad regulatory framework for the use of all AI systems, organized into risk categories. The act includes evaluation and reporting requirements for “general-purpose AI” systems trained with more than 10^25 FLOP.
• Over the past year and a half, AI systems have exhibited many new capabilities, including generating high-quality images, expert-level Stratego, expert-level Diplomacy, writing code, generating music, generating video, acing AP exams, solving Olympiad-level geometry problems, and winning drone races against human world-champions.

You can subscribe to the MIRI Newsletter here.

The MIRI Newsletter is back in action after a hiatus since July 2022. To recap some of the biggest MIRI developments since then:

• MIRI released its 2024 Mission and Strategy Update, announcing a major shift in focus: While we’re continuing to support various technical research programs at MIRI, our new top priority is broad public communication and policy change.
  • In short, we’ve become increasingly pessimistic that humanity will be able to solve the alignment problem in time, while we’ve become more hopeful (relatively speaking) about the prospect of intergovernmental agreements to hit the brakes on frontier AI development for a very long time—long enough for the world to find some realistic path forward.

• Coinciding with this strategy change, Malo Bourgon transitioned from MIRI COO to CEO, and Nate Soares transitioned from CEO to President. We also made two new senior staff hires: Lisa Thiergart, who manages our research program; and Gretta Duleba, who manages our communications and media engagement.
• In keeping with our new strategy pivot, we’re growing our comms team: I (Harlan Stewart) recently joined the team, and will be spearheading the MIRI Newsletter and a number of other projects alongside Rob Bensinger. I’m a former math and programming instructor and a former researcher at AI Impacts, and I’m excited to contribute to MIRI’s new outreach efforts.
  • The comms team is at the tail end of another hiring round, and we expect to scale up significantly over the coming year. Our Careers page and the MIRI Newsletter will announce when our next comms hiring round begins.

• We are launching a new research team to work on technical AI governance, and we’re currently accepting applicants for roles as researchers and technical writers. The team currently consists of Lisa Thiergart and Peter Barnett, and we’re looking to scale to 5–8 people by the end of the year.
  • The team will focus on researching and designing technical aspects of regulation and policy which could lead to safe AI, with attention given to proposals that can continue to function as we move towards smarter-than-human AI. This work will include: investigating limitations in current proposals such as Responsible Scaling Policies; responding to requests for comments by policy bodies such as the NIST, EU, and UN; researching possible amendments to RSPs and alternative safety standards; and communicating with and consulting for policymakers.
    
• Now that the MIRI team is growing again, we also plan to do some fundraising this year, including potentially running an end-of-year fundraiser—our first fundraiser since 2019. We’ll have more updates about that later this year.

As part of our post-2022 strategy shift, we’ve been putting far more time into writing up our thoughts and making media appearances. In addition to announcing these in the MIRI Newsletter again going forward, we now have a Media page that will collect our latest writings and appearances in one place. Some highlights since our last newsletter in 2022:

• MIRI senior researcher Eliezer Yudkowsky kicked off our new wave of public outreach in early 2023 with a very candid TIME magazine op-ed and a follow-up TED Talk, both of which appear to have had a big impact. The TIME article was the most viewed page on the TIME website for a week, and prompted some concerned questioning at a White House press briefing.
• Eliezer and Nate have done a number of podcast appearances since then, attempting to share our concerns and policy recommendations with a variety of audiences. Of these, we think the best appearance on substance was Eliezer’s multi-hour conversation with Logan Bartlett.
• This December, Malo was one of sixteen attendees invited by Leader Schumer and Senators Young, Rounds, and Heinrich to participate in a bipartisan forum on “Risk, Alignment, and Guarding Against Doomsday Scenarios.” Malo’s written statement is the best current write-up of MIRI’s policy recommendations. At the event, Malo found it heartening to see how far the discourse has come in a very short time—Leader Schumer opened the event by asking attendees for their probability that AI could lead to a doomsday scenario, using the term “p(doom)”. 
• Nate has written several particularly important essays pertaining to AI risk:
  • AI as a Science, and Three Obstacles to Alignment Strategies.
  • Focus on the Places Where You Feel Shocked Everyone’s Dropping the Ball.
  • Ability to Solve Long-Horizon Tasks Correlates with Wanting Things in the Behaviorist Sense.
• In a new report, MIRI researchers Peter Barnett and Jeremy Gillen argue that without fundamental advances, misalignment and catastrophe are the default outcomes of training powerful AI.
• Other unusually-good podcast appearances and write-ups include Eliezer’s appearances on Bankless, Bloomberg, and the David Pakman Show, Nate’s comments on an OpenAI strategy document, and Rob Bensinger’s take on ten relatively basic reasons to expect AGI ruin. See the Media page for a fuller list.

In next month’s newsletter, we’ll discuss some of the biggest developments in the world at large since the MIRI Newsletter went on pause, as well as returning to form with a more detailed discussion of MIRI’s most recent activities and write-ups. You can subscribe to the MIRI Newsletter here.