MIRI updates

• MIRI’s Evan Hubinger uses a notion of optimization power to define whether AI systems are compatible with the strategy-stealing assumption.
• MIRI’s Abram Demski discusses debate approaches to AI safety that don’t rely on factored cognition.
• Evan argues that the first AGI systems are likely to be very similar to each other, and discusses implications for alignment.
• Jack Clark’s Import AI newsletter discusses the negative research results from our end-of-year update.
• Richard Ngo shares high-quality discussion of his great introductory sequence AGI Safety from First Principles, featuring Paul Christiano, Max Daniel, Ben Garfinkel, Adam Gleave, Matthew Graves, Daniel Kokotajlo, Will MacAskill, Rohin Shah, Jaan Tallinn, and MIRI’s Evan Hubinger and Buck Shlegeris.
• Tom Chivers discusses the rationality community and Rationality: From AI to Zombies. (Contrary to the headline, COVID-19 receives little discussion.)

News and links

• Alex Flint argues that growing the field of AI alignment researchers should be a side-effect of optimizing for “research depth”, rather than functioning as a target in its own right — much as software projects shouldn’t optimize for larger teams or larger codebases. Flint also comments on strategy/policy research.
• Daniel Kokotajlo of the Center on Long-Term Risk argues that AGI may enable decisive strategic advantage before GDP accelerates. Cf. a pithier comment from Eliezer Yudkowsky.
• TAI Safety Bibliographic Database: Jess Riedel and Angelica Deibel release a database of AI safety research, and analyze recent trends in the field.
• DeepMind’s AI safety team investigates optimality properties of meta-trained RNNs and the tampering problem: “How can we design agents that pursue a given objective when all feedback mechanisms for describing that objective are influenceable by the agent?”
• Facebook launches Forecast, a new community prediction platform akin to Metaculus.
• Effective altruists have released the microCOVID calculator, a very handy tool for assessing activities’ COVID-19 infection risk. Meanwhile, Zvi Mowshowitz’s weekly updates on LessWrong continue to be a good (US-centric) resource for staying up to date on COVID-19 developments such as the B117 variant.
• Rethink Priorities researcher Linchuan Zhang summarizes things he’s learned forecasting COVID-19 in 2020: forming good outside views is often hard; effective altruists tend to overrate superforecasters; etc.

MIRI COO Malo Bourgon reviews our past year and discusses our future plans in 2020 Updates and Strategy.

Our biggest update is that we've made less concrete progress than we expected on the new research we described in 2018 Update: Our New Research Directions. As a consequence, we're scaling back our work on these research directions, and looking for new angles of attack that have better odds of resulting in a solution to the alignment problem.

Other MIRI updates

• A new paper from MIRI researcher Evan Hubinger: "An Overview of 11 Proposals for Building Safe Advanced AI."
• A belated paper announcement from last year: Andrew Critch's "A Parametric, Resource-Bounded Generalization of Löb's Theorem, and a Robust Cooperation Criterion for Open-Source Game Theory", a result originally written up during his time at MIRI, has been published in the Journal of Symbolic Logic.
• MIRI's Abram Demski introduces Learning Normativity: A Research Agenda. See also Abram's new write-up, Normativity.
• Evan Hubinger clarifies inner alignment terminology.
• The Survival and Flourishing Fund (SFF) has awarded MIRI $563,000 in its latest round of grants! Our enormous gratitude to SFF's grant recommenders and funders.
• A Map That Reflects the Territory is a new print book set collecting the top LessWrong essays of 2018, including essays by MIRI researchers Eliezer Yudkowsky, Abram Demski, and Scott Garrabrant.
• DeepMind's Rohin Shah gives his overview of Scott Garrabrant's Cartesian Frames framework.

News and links

• Daniel Filan launches the AI X-Risk Research Podcast (AXRP) with episodes featuring Adam Gleave, Rohin Shah, and Andrew Critch.
• DeepMind's AlphaFold represents a very large advance in protein structure prediction.
• Metaculus launches Forecasting AI Progress, an open four-month tournament to predict advances in AI, with a $50,000 prize pool.
• Continuing the Takeoffs Debate: Richard Ngo responds to Paul Christiano's "changing selection pressures" argument against hard takeoff.
• OpenAI's Beth Barnes discusses the obfuscated arguments problem for AI safety via debate:
  
  

  Previously we hoped that debate/IDA could verify any knowledge for which such human-understandable arguments exist, even if these arguments are intractably large. We hoped the debaters could strategically traverse small parts of the implicit large argument tree and thereby show that the whole tree could be trusted.

  The obfuscated argument problem suggests that we may not be able to rely on debaters to find flaws in large arguments, so that we can only trust arguments when we could find flaws by recursing randomly—e.g. because the argument is small enough that we could find a single flaw if one existed, or because the argument is robust enough that it is correct unless it has many flaws.

   

• Some AI Research Areas and Their Relevance to Existential Safety: Andrew Critch compares out-of-distribution robustness, agent foundations, multi-agent RL, preference learning, and other research areas.
• Ben Hoskin releases his 2020 AI Alignment Literature Review and Charity Comparison.
• Open Philanthropy summarizes its AI governance grantmaking to date.

MIRI researcher Scott Garrabrant has completed his Cartesian Frames sequence. Scott also covers the first two posts' contents in video form.

Other MIRI updates

• Contrary to my previous announcement, MIRI won’t be running a formal fundraiser this year, though we’ll still be participating in Giving Tuesday and other matching opportunities. To donate and get information on tax-advantaged donations, employer matching, etc., see intelligence.org/donate. We’ll also be doing an end-of-the-year update and retrospective in the next few weeks.
• Facebook's Giving Tuesday matching event takes place this Tuesday (Dec. 1) at 5:00:00am PT.  Facebook will 100%-match the first $2M donated, something that will plausibly take only 2–3 seconds. To get 100%-matched, then, it's even more important than last year to start clicking at 4:59:59AM PT. Facebook will then 10%-match the next $50M of donations that are made. Details on optimizing your donation(s) to MIRI's Facebook Fundraiser can be found at EA Giving Tuesday, a Rethink Charity project.
• Video discussion: Stuart Armstrong, Scott Garrabrant, and the AI Safety Reading Group discuss Stuart's If I Were A Well-Intentioned AI….
• MIRI research associate Vanessa Kosoy raises questions about AI information hazards.
• Buck Shlegeris argues that we're likely at the “hinge of history” (assuming we aren't living in a simulation).
• To make it easier to find and cite old versions of MIRI papers (especially ones that aren't on arXiv), we've collected links to obsolete versions on intelligence.org/revisions.

News and links

• CFAR's Anna Salamon asks: Where do (did?) stable, cooperative institutions come from?
• The Center for Human-Compatible AI is accepting applications for research internships through Dec. 13.
• AI Safety (virtual) Camp is accepting applications through Dec. 15.
• The 4th edition of Artificial Intelligence: A Modern Approach is out, with expanded discussion of the alignment problem.
• DeepMind's Rohin Shah reviews Brian Christian's new book The Alignment Problem: Machine Learning and Human Values.
• Daniel Filan and Rohin Shah discuss security mindset and takeoff speeds.
• Fortune profiles existential risk philanthropist Jaan Tallinn.

Starting today, Scott Garrabrant has begun posting Cartesian Frames, a sequence introducing a new conceptual framework Scott has found valuable for thinking about agency.

In Scott's words: Cartesian Frames are “applying reasoning like Pearl's to objects like game theory's, with a motivation like Hutter's”.

Scott will be giving an online talk introducing Cartesian frames this Sunday at 12pm PT on Zoom (link). He'll also be hosting office hours on Gather.Town the next four Sundays; see here for details.

Other MIRI updates

• Abram Demski discusses the problem of comparing utilities, highlighting some non-obvious implications.
• In March 2020, the US Congress passed the CARES Act, which changes the tax advantages of donations to qualifying NPOs like MIRI in 2020. Changes include:
  • 1. A new “above the line” tax deduction: up to $300 per taxpayer ($600 for a married couple) in annual charitable contributions for people who take the standard deduction. Donations to donor-advised funds (DAFs) do not qualify for this new deduction.
  • 2. New charitable deduction limits: Taxpayers who itemize their deductions can deduct much greater amounts of their contributions. Individuals can elect to deduct donations up to 100% of their 2020 AGI — up from 60% previously. This higher limit also does not apply to donations to DAFs.

  As usual, consult with your tax advisor for more information.

• Our fundraiser this year will start on November 29 (two days before Giving Tuesday) and finish on January 2. We're hoping that having our fundraiser straddle 2020 and 2021 will give people more flexibility given the unusual tax law changes above.
• I'm happy to announce that MIRI has received a donation of $246,435 from an anonymous returning donor. Our thanks to the donor, and to Effective Giving UK for facilitating this donation!

News and links

• Richard Ngo tries to provide a relatively careful and thorough version of the standard argument for worrying about AGI risk: AGI Safety from First Principles. See also Rohin Shah's summary.
• The AI Alignment Podcast interviews Andrew Critch about his recent overview paper, “AI Research Considerations for Human Existential Safety.”

Abram Demski and Scott Garrabrant have made a major update to "Embedded Agency", with new discussions of ε-exploration, Newcomblike problems, reflective oracles, logical uncertainty, Goodhart's law, and predicting rare catastrophes, among other topics.

Abram has also written an overview of what good reasoning looks in the absence of Bayesian updating: Radical Probabilism. One recurring theme:

[I]n general (i.e., without any special prior which does guarantee convergence for restricted observation models), a Bayesian relies on a realizability (aka grain-of-truth) assumption for convergence, as it does for some other nice properties. Radical probabilism demands these properties without such an assumption.

[… C]onvergence points at a notion of "objectivity" for the radical probabilist. Although the individual updates a radical probabilist makes can go all over the place, the beliefs must eventually settle down to something. The goal of reasoning is to settle down to that answer as quickly as possible.

Meanwhile, Infra-Bayesianism is a new formal framework for thinking about optimal reasoning without requiring an reasoner's true environment to be in its hypothesis space. Abram comments: "Alex Appel and Vanessa Kosoy have been working hard at 'Infra-Bayesianism', a new approach to RL which aims to make it easier (ie, possible) to prove safety-relevant theorems (and, also, a new approach to Bayesianism more generally).

Other MIRI updates

• Abram Demski writes a parable on the differences between logical inductors and Bayesians: The Bayesian Tyrant.
• Building on the selection vs. control distinction, Abram contrasts "mesa-search" and "mesa-control".

News and links

• From OpenAI's Stiennon et al.: Learning to Summarize with Human Feedback. MIRI researcher Eliezer Yudkowsky comments:
  

  A very rare bit of research that is directly, straight-up relevant to real alignment problems! They trained a reward function on human preferences and then measured how hard you could optimize against the trained function before the results got actually worse.

  [… Y]ou can ask for results as good as the best 99th percentile of rated stuff in the training data (a la Jessica Taylor's quantilization idea).  Ask for things the trained reward function rates as "better" than that, and it starts to find "loopholes" as seen from outside the system; places where the trained reward function poorly matches your real preferences, instead of places where your real preferences would rate high reward.

• Chi Nguyen writes up an introduction to Paul Christiano's iterated amplification research agenda that seeks to be the first such resource that is "both easy to understand and [gives] a complete picture". The post includes inline comments by Christiano.
• Forecasters share visualizations of their AI timelines on LessWrong.

MIRI updates

• Three questions from MIRI's Abram Demski: What does it mean to apply decision theory?, How “honest” is GPT-3?, and How should AI debate be judged?
• A transcript from MIRI researcher Scott Garrabrant: What Would I Do? Self-Prediction in Simple Algorithms.
• MIRI researcher Buck Shlegeris reviews the debate on what the history of nuclear weapons implies about humanity's ability to coordinate.
• From MIRI's Evan Hubinger: Learning the Prior and Generalization and Alignment Proposals and Complexity Classes.
• Rafael Harth's Inner Alignment: Explain Like I'm 12 Edition summarizes the concepts and takeaways from “Risks from Learned Optimization”.
• Issa Rice reviews discussion to date on MIRI's research focus, “To what extent is it possible to have a precise theory of rationality?”, and the relationship between deconfusion research and safety outcomes. (Plus a short reply.)
• “Pitfalls of Learning a Reward Function Online” (IJCAI paper, LW summary): FHI researcher and MIRI research associate Stuart Armstrong, with DeepMind's Jan Leike, Laurent Orseau, and Shane Legg, explore ways to discourage agents from manipulating their reward signal to be easier to optimize.

News and links

• From Paul Christiano: Learning the Prior and Better Priors as a Safety Problem.
• From Victoria Krakovna: Tradeoff Between Desirable Properties for Baseline Choices in Impact Measures.
• Ben Pace summarizes Christiano's “What Failure Looks Like” post and the resultant discussion.
• Kaj Sotala collects recent examples of experiences from people working with GPT-3.

After completing a study fellowship at MIRI that he began in late 2019, Blake Jones is joining the MIRI research team full-time! Blake joins MIRI after a long career working on low-level software systems such as the Solaris operating system and the Oracle database.

Other MIRI updates

• MIRI researcher Evan Hubinger goes on the FLI podcast (transcript/discussion, audio) to discuss “inner alignment, outer alignment, and proposals for building safe advanced AI”.
• A revised version of Vanessa Kosoy’s “Optimal Polynomial-Time Estimators: A Bayesian Notion of Approximation Algorithm,” co-authored with Alex Appel, has been accepted to the Journal of Applied Logics.
• From MIRI researcher Abram Demski: Dutch-Booking CDT: Revised Argument argues that “causal” theories (ones using counterfactuals to evaluate expected value) must behave the same as theories using conditional probabilities. Relating HCH and Logical Induction discusses amplification in the context of reflective oracles. And Radical Probabilism reviews the surprising gap between Dutch-book arguments and Bayes' rule.

News and links

• Alex Flint summarizes the Center for Human-Compatible AI's assistance games research program.
• CHAI’s Andrew Critch and MILA’s David Krueger release “AI Research Considerations for Human Existential Safety (ARCHES)”, a review of 29 AI (existential) safety research directions, each with an illustrative analogy, examples of current work and potential synergies between research directions, and discussion of ways the research approach might lower (or raise) existential risk.
• OpenAI’s Danny Hernandez and Tom Brown present evidence that “for AI tasks with high levels of recent investment, algorithmic progress has yielded more gains than classical hardware efficiency”.
• DeepMind's Victoria Krakovna shares her takeaways from the COVID-19 pandemic for slow-takeoff scenarios.
• AI Impacts’ Daniel Kokotajlo discusses possible changes the world might undergo before reaching AGI.
• 80,000 Hours describes careers they view as promising but haven’t written up as priority career paths, including information security (previously discussed here).