Starting today, Scott Garrabrant has begun posting Cartesian Frames, a sequence introducing a new conceptual framework Scott has found valuable for thinking about agency.

In Scott's words: Cartesian Frames are “applying reasoning like Pearl's to objects like game theory's, with a motivation like Hutter's”.

Scott will be giving an online talk introducing Cartesian frames this Sunday at 12pm PT on Zoom (link). He'll also be hosting office hours on Gather.Town the next four Sundays; see here for details.

Other MIRI updates

• Abram Demski discusses the problem of comparing utilities, highlighting some non-obvious implications.
• In March 2020, the US Congress passed the CARES Act, which changes the tax advantages of donations to qualifying NPOs like MIRI in 2020. Changes include:
  • 1. A new “above the line” tax deduction: up to $300 per taxpayer ($600 for a married couple) in annual charitable contributions for people who take the standard deduction. Donations to donor-advised funds (DAFs) do not qualify for this new deduction.
  • 2. New charitable deduction limits: Taxpayers who itemize their deductions can deduct much greater amounts of their contributions. Individuals can elect to deduct donations up to 100% of their 2020 AGI — up from 60% previously. This higher limit also does not apply to donations to DAFs.

  As usual, consult with your tax advisor for more information.

• Our fundraiser this year will start on November 29 (two days before Giving Tuesday) and finish on January 2. We're hoping that having our fundraiser straddle 2020 and 2021 will give people more flexibility given the unusual tax law changes above.
• I'm happy to announce that MIRI has received a donation of $246,435 from an anonymous returning donor. Our thanks to the donor, and to Effective Giving UK for facilitating this donation!

News and links

• Richard Ngo tries to provide a relatively careful and thorough version of the standard argument for worrying about AGI risk: AGI Safety from First Principles. See also Rohin Shah's summary.
• The AI Alignment Podcast interviews Andrew Critch about his recent overview paper, “AI Research Considerations for Human Existential Safety.”

Abram Demski and Scott Garrabrant have made a major update to "Embedded Agency", with new discussions of ε-exploration, Newcomblike problems, reflective oracles, logical uncertainty, Goodhart's law, and predicting rare catastrophes, among other topics.

Abram has also written an overview of what good reasoning looks in the absence of Bayesian updating: Radical Probabilism. One recurring theme:

[I]n general (i.e., without any special prior which does guarantee convergence for restricted observation models), a Bayesian relies on a realizability (aka grain-of-truth) assumption for convergence, as it does for some other nice properties. Radical probabilism demands these properties without such an assumption.

[… C]onvergence points at a notion of "objectivity" for the radical probabilist. Although the individual updates a radical probabilist makes can go all over the place, the beliefs must eventually settle down to something. The goal of reasoning is to settle down to that answer as quickly as possible.

Meanwhile, Infra-Bayesianism is a new formal framework for thinking about optimal reasoning without requiring an reasoner's true environment to be in its hypothesis space. Abram comments: "Alex Appel and Vanessa Kosoy have been working hard at 'Infra-Bayesianism', a new approach to RL which aims to make it easier (ie, possible) to prove safety-relevant theorems (and, also, a new approach to Bayesianism more generally).

Other MIRI updates

• Abram Demski writes a parable on the differences between logical inductors and Bayesians: The Bayesian Tyrant.
• Building on the selection vs. control distinction, Abram contrasts "mesa-search" and "mesa-control".

News and links

• From OpenAI's Stiennon et al.: Learning to Summarize with Human Feedback. MIRI researcher Eliezer Yudkowsky comments:
  

  A very rare bit of research that is directly, straight-up relevant to real alignment problems! They trained a reward function on human preferences and then measured how hard you could optimize against the trained function before the results got actually worse.

  [… Y]ou can ask for results as good as the best 99th percentile of rated stuff in the training data (a la Jessica Taylor's quantilization idea).  Ask for things the trained reward function rates as "better" than that, and it starts to find "loopholes" as seen from outside the system; places where the trained reward function poorly matches your real preferences, instead of places where your real preferences would rate high reward.

• Chi Nguyen writes up an introduction to Paul Christiano's iterated amplification research agenda that seeks to be the first such resource that is "both easy to understand and [gives] a complete picture". The post includes inline comments by Christiano.
• Forecasters share visualizations of their AI timelines on LessWrong.

MIRI updates

• Three questions from MIRI's Abram Demski: What does it mean to apply decision theory?, How “honest” is GPT-3?, and How should AI debate be judged?
• A transcript from MIRI researcher Scott Garrabrant: What Would I Do? Self-Prediction in Simple Algorithms.
• MIRI researcher Buck Shlegeris reviews the debate on what the history of nuclear weapons implies about humanity's ability to coordinate.
• From MIRI's Evan Hubinger: Learning the Prior and Generalization and Alignment Proposals and Complexity Classes.
• Rafael Harth's Inner Alignment: Explain Like I'm 12 Edition summarizes the concepts and takeaways from “Risks from Learned Optimization”.
• Issa Rice reviews discussion to date on MIRI's research focus, “To what extent is it possible to have a precise theory of rationality?”, and the relationship between deconfusion research and safety outcomes. (Plus a short reply.)
• “Pitfalls of Learning a Reward Function Online” (IJCAI paper, LW summary): FHI researcher and MIRI research associate Stuart Armstrong, with DeepMind's Jan Leike, Laurent Orseau, and Shane Legg, explore ways to discourage agents from manipulating their reward signal to be easier to optimize.

News and links

• From Paul Christiano: Learning the Prior and Better Priors as a Safety Problem.
• From Victoria Krakovna: Tradeoff Between Desirable Properties for Baseline Choices in Impact Measures.
• Ben Pace summarizes Christiano's “What Failure Looks Like” post and the resultant discussion.
• Kaj Sotala collects recent examples of experiences from people working with GPT-3.

After completing a study fellowship at MIRI that he began in late 2019, Blake Jones is joining the MIRI research team full-time! Blake joins MIRI after a long career working on low-level software systems such as the Solaris operating system and the Oracle database.

Other MIRI updates

• MIRI researcher Evan Hubinger goes on the FLI podcast (transcript/discussion, audio) to discuss “inner alignment, outer alignment, and proposals for building safe advanced AI”.
• A revised version of Vanessa Kosoy’s “Optimal Polynomial-Time Estimators: A Bayesian Notion of Approximation Algorithm,” co-authored with Alex Appel, has been accepted to the Journal of Applied Logics.
• From MIRI researcher Abram Demski: Dutch-Booking CDT: Revised Argument argues that “causal” theories (ones using counterfactuals to evaluate expected value) must behave the same as theories using conditional probabilities. Relating HCH and Logical Induction discusses amplification in the context of reflective oracles. And Radical Probabilism reviews the surprising gap between Dutch-book arguments and Bayes' rule.

News and links

• Alex Flint summarizes the Center for Human-Compatible AI's assistance games research program.
• CHAI’s Andrew Critch and MILA’s David Krueger release “AI Research Considerations for Human Existential Safety (ARCHES)”, a review of 29 AI (existential) safety research directions, each with an illustrative analogy, examples of current work and potential synergies between research directions, and discussion of ways the research approach might lower (or raise) existential risk.
• OpenAI’s Danny Hernandez and Tom Brown present evidence that “for AI tasks with high levels of recent investment, algorithmic progress has yielded more gains than classical hardware efficiency”.
• DeepMind's Victoria Krakovna shares her takeaways from the COVID-19 pandemic for slow-takeoff scenarios.
• AI Impacts’ Daniel Kokotajlo discusses possible changes the world might undergo before reaching AGI.
• 80,000 Hours describes careers they view as promising but haven’t written up as priority career paths, including information security (previously discussed here).

MIRI researcher Evan Hubinger reviews “11 different proposals for building safe advanced AI under the current machine learning paradigm”, comparing them on outer alignment, inner alignment, training competitiveness, and performance competitiveness. 

Other updates

• We keep being amazed by new shows of support ⁠— following our last two announcements, MIRI has received a donation from another anonymous donor totaling ~$265,000 in euros, facilitated by Effective Giving UK and the Effective Altruism Foundation. Massive thanks to the donor for their generosity, and to both organizations for their stellar support for MIRI and other longtermist organizations!
• Hacker News discusses Eliezer Yudkowsky's There's No Fire Alarm for AGI.
• MIRI researcher Buck Shlegeris talks about deference and inside-view models on the EA Forum.
• OpenAI unveils GPT-3, a massive 175-billion parameter language model that can figure out how to solve a variety of problems without task-specific training or fine-tuning. Gwern Branwen's pithy summary:
  

  GPT-3 is terrifying because it's a tiny model compared to what's possible, trained in the dumbest way possible on a single impoverished modality on tiny data, yet the first version already manifests crazy runtime meta-learning—and the scaling curves still are not bending!

  Further discussion by Branwen and by Rohin Shah.

• Stuart Russell gives this year's Turing Lecture online, discussing “provably beneficial AI”.

MIRI has received an anonymous donation of ~$275,000 in euros, facilitated by Effective Giving UK. Additionally, the Survival and Flourishing Fund, working with funders Jaan Tallinn and Jed McCaleb, has announced $340,000 in grants to MIRI. SFF is a new fund that is taking over much of BERI's grantmaking work.

To everyone involved in both decisions to support our research, thank you!

Other updates

• An Orthodox Case Against Utility Functions: MIRI researcher Abram Demski makes the case against utility functions that rely on a microphysical “view from nowhere”.
• Stuart Armstrong’s “If I were a well-intentioned AI…” sequence looks at alignment problems from the perspective of a well-intentioned but ignorant agent.
• AI Impacts' Asya Bergal summarizes takeaways from “safety-by-default” researchers.
• Agarwal and Norouzi report improvements in offline RL.
• Daniel Kokotajlo’s Three Kinds of Competitiveness distinguishes performance-competitive, cost-competitive, and date-competitive AI systems.
• The Stanford Encyclopedia of Philosophy's new Ethics of AI and Robotics article includes a discussion of existential risk from AGI.

MIRI has been awarded its largest grant to date — $7,703,750 split over two years from Open Philanthropy, in partnership with Ben Delo, co-founder of the cryptocurrency trading platform BitMEX!

We have also been awarded generous grants by the Berkeley Existential Risk Initiative ($300,000) and the Long-Term Future Fund ($100,000). Our thanks to everybody involved!

Other updates

• Buck Shlegeris of MIRI and Rohin Shah of CHAI discuss Rohin's 2018–2019 overview of technical AI alignment research on the AI Alignment Podcast.
• From MIRI's Abram Demski: Thinking About Filtered Evidence Is (Very!) Hard and Bayesian Evolving-to-Extinction. And from Evan Hubinger: Synthesizing Amplification and Debate.
• From OpenAI's Beth Barnes, Paul Christiano, Long Ouyang, and Geoffrey Irving: Progress on AI Safety via Debate.
• Zoom In: An Introduction to Circuits: OpenAI's Olah, Cammarata, Schubert, Goh, Petrov, and Carter argue, “Features are the fundamental unit of neural networks. They correspond to directions [in the space of neuron activations]. […] Features are connected by weights, forming circuits. […] Analogous features and circuits form across models and tasks.”
• DeepMind's Agent57 appears to meet one of the AI benchmarks in AI Impacts' 2016 survey, “outperform professional game testers on all Atari games using no game specific knowledge”, earlier than NeurIPS/ICML authors predicted.
• From DeepMind Safety Research: Specification gaming: the flip side of AI ingenuity.