MIRI updates

• Abram Demski distinguishes different versions of the problem of “pointing at” human values in AI alignment.
• Evan Hubinger discusses “Risks from Learned Optimization” on the AI X-Risk Research Podcast.
• Eliezer Yudkowsky comments on AI safety via debate and Goodhart’s law.
• MIRI supporters donated ~$135k on Giving Tuesday, of which ~26% was matched by Facebook and ~28% by employers for a total of $207,436! MIRI also received $6,624 from TisBest Philanthropy in late December, largely through Round Two of Ray Dalio’s #RedefineGifting initiative. Our thanks to all of you!
• Spencer Greenberg discusses society and education with Anna Salamon and Duncan Sabien on the Clearer Thinking podcast.
• We Want MoR: Eliezer participates in a (spoiler-laden) discussion of Harry Potter and the Methods of Rationality.

News and links

• Richard Ngo reflects on his time in effective altruism:
  

  […] Until recently, I was relatively passive in making big decisions. Often that meant just picking the most high-prestige default option, rather than making a specific long-term plan. This also involved me thinking about EA from a “consumer” mindset rather than a “producer” mindset. When it seemed like something was missing, I used to wonder why the people responsible hadn’t done it; now I also ask why I haven’t done it, and consider taking responsibility myself.

  Partly that’s just because I’ve now been involved in EA for longer. But I think I also used to overestimate how established and organised EA is. In fact, we’re an incredibly young movement, and we’re still making up a lot of things as we go along. That makes proactivity more important.

  Another reason to value proactivity highly is that taking the most standard route to success is often overrated. […] My inspiration in this regard is a friend of mine who has, three times in a row, reached out to an organisation she wanted to work for and convinced them to create a new position for her.

• Ngo distinguishes claims about goal specification, orthogonality, instrumental convergence, value fragility, and Goodhart’s law based on whether they refer to systems at training time versus deployment time.
• Connor Leahy, author of The Hacker Learns to Trust, argues (among other things) that “GPT-3 is our last warning shot” for coordinating to address AGI alignment. (Podcast version.) I include this talk because it’s a good talk and the topic warrants discussion, though MIRI staff don’t necessarily endorse this claim — and Eliezer would certainly object to any claim that something is a fire alarm for AGI.
• OpenAI safety researchers including Dario Amodei, Paul Christiano, and Chris Olah depart OpenAI.
• OpenAI’s DALL-E uses GPT-3 for image generation, while CLIP exhibits impressive zero-shot image classification capabilities. Gwern Branwen comments in his newsletter.

MIRI updates

• MIRI’s Evan Hubinger uses a notion of optimization power to define whether AI systems are compatible with the strategy-stealing assumption.
• MIRI’s Abram Demski discusses debate approaches to AI safety that don’t rely on factored cognition.
• Evan argues that the first AGI systems are likely to be very similar to each other, and discusses implications for alignment.
• Jack Clark’s Import AI newsletter discusses the negative research results from our end-of-year update.
• Richard Ngo shares high-quality discussion of his great introductory sequence AGI Safety from First Principles, featuring Paul Christiano, Max Daniel, Ben Garfinkel, Adam Gleave, Matthew Graves, Daniel Kokotajlo, Will MacAskill, Rohin Shah, Jaan Tallinn, and MIRI’s Evan Hubinger and Buck Shlegeris.
• Tom Chivers discusses the rationality community and Rationality: From AI to Zombies. (Contrary to the headline, COVID-19 receives little discussion.)

News and links

• Alex Flint argues that growing the field of AI alignment researchers should be a side-effect of optimizing for “research depth”, rather than functioning as a target in its own right — much as software projects shouldn’t optimize for larger teams or larger codebases. Flint also comments on strategy/policy research.
• Daniel Kokotajlo of the Center on Long-Term Risk argues that AGI may enable decisive strategic advantage before GDP accelerates. Cf. a pithier comment from Eliezer Yudkowsky.
• TAI Safety Bibliographic Database: Jess Riedel and Angelica Deibel release a database of AI safety research, and analyze recent trends in the field.
• DeepMind’s AI safety team investigates optimality properties of meta-trained RNNs and the tampering problem: “How can we design agents that pursue a given objective when all feedback mechanisms for describing that objective are influenceable by the agent?”
• Facebook launches Forecast, a new community prediction platform akin to Metaculus.
• Effective altruists have released the microCOVID calculator, a very handy tool for assessing activities’ COVID-19 infection risk. Meanwhile, Zvi Mowshowitz’s weekly updates on LessWrong continue to be a good (US-centric) resource for staying up to date on COVID-19 developments such as the B117 variant.
• Rethink Priorities researcher Linchuan Zhang summarizes things he’s learned forecasting COVID-19 in 2020: forming good outside views is often hard; effective altruists tend to overrate superforecasters; etc.

MIRI COO Malo Bourgon reviews our past year and discusses our future plans in 2020 Updates and Strategy.

Our biggest update is that we've made less concrete progress than we expected on the new research we described in 2018 Update: Our New Research Directions. As a consequence, we're scaling back our work on these research directions, and looking for new angles of attack that have better odds of resulting in a solution to the alignment problem.

Other MIRI updates

• A new paper from MIRI researcher Evan Hubinger: "An Overview of 11 Proposals for Building Safe Advanced AI."
• A belated paper announcement from last year: Andrew Critch's "A Parametric, Resource-Bounded Generalization of Löb's Theorem, and a Robust Cooperation Criterion for Open-Source Game Theory", a result originally written up during his time at MIRI, has been published in the Journal of Symbolic Logic.
• MIRI's Abram Demski introduces Learning Normativity: A Research Agenda. See also Abram's new write-up, Normativity.
• Evan Hubinger clarifies inner alignment terminology.
• The Survival and Flourishing Fund (SFF) has awarded MIRI $563,000 in its latest round of grants! Our enormous gratitude to SFF's grant recommenders and funders.
• A Map That Reflects the Territory is a new print book set collecting the top LessWrong essays of 2018, including essays by MIRI researchers Eliezer Yudkowsky, Abram Demski, and Scott Garrabrant.
• DeepMind's Rohin Shah gives his overview of Scott Garrabrant's Cartesian Frames framework.

News and links

• Daniel Filan launches the AI X-Risk Research Podcast (AXRP) with episodes featuring Adam Gleave, Rohin Shah, and Andrew Critch.
• DeepMind's AlphaFold represents a very large advance in protein structure prediction.
• Metaculus launches Forecasting AI Progress, an open four-month tournament to predict advances in AI, with a $50,000 prize pool.
• Continuing the Takeoffs Debate: Richard Ngo responds to Paul Christiano's "changing selection pressures" argument against hard takeoff.
• OpenAI's Beth Barnes discusses the obfuscated arguments problem for AI safety via debate:
  
  

  Previously we hoped that debate/IDA could verify any knowledge for which such human-understandable arguments exist, even if these arguments are intractably large. We hoped the debaters could strategically traverse small parts of the implicit large argument tree and thereby show that the whole tree could be trusted.

  The obfuscated argument problem suggests that we may not be able to rely on debaters to find flaws in large arguments, so that we can only trust arguments when we could find flaws by recursing randomly—e.g. because the argument is small enough that we could find a single flaw if one existed, or because the argument is robust enough that it is correct unless it has many flaws.

   

• Some AI Research Areas and Their Relevance to Existential Safety: Andrew Critch compares out-of-distribution robustness, agent foundations, multi-agent RL, preference learning, and other research areas.
• Ben Hoskin releases his 2020 AI Alignment Literature Review and Charity Comparison.
• Open Philanthropy summarizes its AI governance grantmaking to date.

MIRI researcher Scott Garrabrant has completed his Cartesian Frames sequence. Scott also covers the first two posts' contents in video form.

Other MIRI updates

• Contrary to my previous announcement, MIRI won’t be running a formal fundraiser this year, though we’ll still be participating in Giving Tuesday and other matching opportunities. To donate and get information on tax-advantaged donations, employer matching, etc., see intelligence.org/donate. We’ll also be doing an end-of-the-year update and retrospective in the next few weeks.
• Facebook's Giving Tuesday matching event takes place this Tuesday (Dec. 1) at 5:00:00am PT.  Facebook will 100%-match the first $2M donated, something that will plausibly take only 2–3 seconds. To get 100%-matched, then, it's even more important than last year to start clicking at 4:59:59AM PT. Facebook will then 10%-match the next $50M of donations that are made. Details on optimizing your donation(s) to MIRI's Facebook Fundraiser can be found at EA Giving Tuesday, a Rethink Charity project.
• Video discussion: Stuart Armstrong, Scott Garrabrant, and the AI Safety Reading Group discuss Stuart's If I Were A Well-Intentioned AI….
• MIRI research associate Vanessa Kosoy raises questions about AI information hazards.
• Buck Shlegeris argues that we're likely at the “hinge of history” (assuming we aren't living in a simulation).
• To make it easier to find and cite old versions of MIRI papers (especially ones that aren't on arXiv), we've collected links to obsolete versions on intelligence.org/revisions.

News and links

• CFAR's Anna Salamon asks: Where do (did?) stable, cooperative institutions come from?
• The Center for Human-Compatible AI is accepting applications for research internships through Dec. 13.
• AI Safety (virtual) Camp is accepting applications through Dec. 15.
• The 4th edition of Artificial Intelligence: A Modern Approach is out, with expanded discussion of the alignment problem.
• DeepMind's Rohin Shah reviews Brian Christian's new book The Alignment Problem: Machine Learning and Human Values.
• Daniel Filan and Rohin Shah discuss security mindset and takeoff speeds.
• Fortune profiles existential risk philanthropist Jaan Tallinn.

In Scott's words: Cartesian Frames are “applying reasoning like Pearl's to objects like game theory's, with a motivation like Hutter's”.

Scott will be giving an online talk introducing Cartesian frames this Sunday at 12pm PT on Zoom (link). He'll also be hosting office hours on Gather.Town the next four Sundays; see here for details.

Other MIRI updates

• Abram Demski discusses the problem of comparing utilities, highlighting some non-obvious implications.
• In March 2020, the US Congress passed the CARES Act, which changes the tax advantages of donations to qualifying NPOs like MIRI in 2020. Changes include:
  • 1. A new “above the line” tax deduction: up to $300 per taxpayer ($600 for a married couple) in annual charitable contributions for people who take the standard deduction. Donations to donor-advised funds (DAFs) do not qualify for this new deduction.
  • 2. New charitable deduction limits: Taxpayers who itemize their deductions can deduct much greater amounts of their contributions. Individuals can elect to deduct donations up to 100% of their 2020 AGI — up from 60% previously. This higher limit also does not apply to donations to DAFs.

  As usual, consult with your tax advisor for more information.

• Our fundraiser this year will start on November 29 (two days before Giving Tuesday) and finish on January 2. We're hoping that having our fundraiser straddle 2020 and 2021 will give people more flexibility given the unusual tax law changes above.
• I'm happy to announce that MIRI has received a donation of $246,435 from an anonymous returning donor. Our thanks to the donor, and to Effective Giving UK for facilitating this donation!

News and links

• Richard Ngo tries to provide a relatively careful and thorough version of the standard argument for worrying about AGI risk: AGI Safety from First Principles. See also Rohin Shah's summary.
• The AI Alignment Podcast interviews Andrew Critch about his recent overview paper, “AI Research Considerations for Human Existential Safety.”

Abram Demski and Scott Garrabrant have made a major update to "Embedded Agency", with new discussions of ε-exploration, Newcomblike problems, reflective oracles, logical uncertainty, Goodhart's law, and predicting rare catastrophes, among other topics.

Abram has also written an overview of what good reasoning looks in the absence of Bayesian updating: Radical Probabilism. One recurring theme:

[I]n general (i.e., without any special prior which does guarantee convergence for restricted observation models), a Bayesian relies on a realizability (aka grain-of-truth) assumption for convergence, as it does for some other nice properties. Radical probabilism demands these properties without such an assumption.

[… C]onvergence points at a notion of "objectivity" for the radical probabilist. Although the individual updates a radical probabilist makes can go all over the place, the beliefs must eventually settle down to something. The goal of reasoning is to settle down to that answer as quickly as possible.

Meanwhile, Infra-Bayesianism is a new formal framework for thinking about optimal reasoning without requiring an reasoner's true environment to be in its hypothesis space. Abram comments: "Alex Appel and Vanessa Kosoy have been working hard at 'Infra-Bayesianism', a new approach to RL which aims to make it easier (ie, possible) to prove safety-relevant theorems (and, also, a new approach to Bayesianism more generally).

Other MIRI updates

• Abram Demski writes a parable on the differences between logical inductors and Bayesians: The Bayesian Tyrant.
• Building on the selection vs. control distinction, Abram contrasts "mesa-search" and "mesa-control".

News and links

• From OpenAI's Stiennon et al.: Learning to Summarize with Human Feedback. MIRI researcher Eliezer Yudkowsky comments:
  

  A very rare bit of research that is directly, straight-up relevant to real alignment problems! They trained a reward function on human preferences and then measured how hard you could optimize against the trained function before the results got actually worse.

  [… Y]ou can ask for results as good as the best 99th percentile of rated stuff in the training data (a la Jessica Taylor's quantilization idea).  Ask for things the trained reward function rates as "better" than that, and it starts to find "loopholes" as seen from outside the system; places where the trained reward function poorly matches your real preferences, instead of places where your real preferences would rate high reward.

• Chi Nguyen writes up an introduction to Paul Christiano's iterated amplification research agenda that seeks to be the first such resource that is "both easy to understand and [gives] a complete picture". The post includes inline comments by Christiano.
• Forecasters share visualizations of their AI timelines on LessWrong.

MIRI updates

• Three questions from MIRI's Abram Demski: What does it mean to apply decision theory?, How “honest” is GPT-3?, and How should AI debate be judged?
• A transcript from MIRI researcher Scott Garrabrant: What Would I Do? Self-Prediction in Simple Algorithms.
• MIRI researcher Buck Shlegeris reviews the debate on what the history of nuclear weapons implies about humanity's ability to coordinate.
• From MIRI's Evan Hubinger: Learning the Prior and Generalization and Alignment Proposals and Complexity Classes.
• Rafael Harth's Inner Alignment: Explain Like I'm 12 Edition summarizes the concepts and takeaways from “Risks from Learned Optimization”.
• Issa Rice reviews discussion to date on MIRI's research focus, “To what extent is it possible to have a precise theory of rationality?”, and the relationship between deconfusion research and safety outcomes. (Plus a short reply.)
• “Pitfalls of Learning a Reward Function Online” (IJCAI paper, LW summary): FHI researcher and MIRI research associate Stuart Armstrong, with DeepMind's Jan Leike, Laurent Orseau, and Shane Legg, explore ways to discourage agents from manipulating their reward signal to be easier to optimize.

News and links

• From Paul Christiano: Learning the Prior and Better Priors as a Safety Problem.
• From Victoria Krakovna: Tradeoff Between Desirable Properties for Baseline Choices in Impact Measures.
• Ben Pace summarizes Christiano's “What Failure Looks Like” post and the resultant discussion.
• Kaj Sotala collects recent examples of experiences from people working with GPT-3.