MIRI updates

• Three questions from MIRI's Abram Demski: What does it mean to apply decision theory?, How “honest” is GPT-3?, and How should AI debate be judged?
• A transcript from MIRI researcher Scott Garrabrant: What Would I Do? Self-Prediction in Simple Algorithms.
• MIRI researcher Buck Shlegeris reviews the debate on what the history of nuclear weapons implies about humanity's ability to coordinate.
• From MIRI's Evan Hubinger: Learning the Prior and Generalization and Alignment Proposals and Complexity Classes.
• Rafael Harth's Inner Alignment: Explain Like I'm 12 Edition summarizes the concepts and takeaways from “Risks from Learned Optimization”.
• Issa Rice reviews discussion to date on MIRI's research focus, “To what extent is it possible to have a precise theory of rationality?”, and the relationship between deconfusion research and safety outcomes. (Plus a short reply.)
• “Pitfalls of Learning a Reward Function Online” (IJCAI paper, LW summary): FHI researcher and MIRI research associate Stuart Armstrong, with DeepMind's Jan Leike, Laurent Orseau, and Shane Legg, explore ways to discourage agents from manipulating their reward signal to be easier to optimize.

News and links

• From Paul Christiano: Learning the Prior and Better Priors as a Safety Problem.
• From Victoria Krakovna: Tradeoff Between Desirable Properties for Baseline Choices in Impact Measures.
• Ben Pace summarizes Christiano's “What Failure Looks Like” post and the resultant discussion.
• Kaj Sotala collects recent examples of experiences from people working with GPT-3.

After completing a study fellowship at MIRI that he began in late 2019, Blake Jones is joining the MIRI research team full-time! Blake joins MIRI after a long career working on low-level software systems such as the Solaris operating system and the Oracle database.

Other MIRI updates

• MIRI researcher Evan Hubinger goes on the FLI podcast (transcript/discussion, audio) to discuss “inner alignment, outer alignment, and proposals for building safe advanced AI”.
• A revised version of Vanessa Kosoy’s “Optimal Polynomial-Time Estimators: A Bayesian Notion of Approximation Algorithm,” co-authored with Alex Appel, has been accepted to the Journal of Applied Logics.
• From MIRI researcher Abram Demski: Dutch-Booking CDT: Revised Argument argues that “causal” theories (ones using counterfactuals to evaluate expected value) must behave the same as theories using conditional probabilities. Relating HCH and Logical Induction discusses amplification in the context of reflective oracles. And Radical Probabilism reviews the surprising gap between Dutch-book arguments and Bayes' rule.

News and links

• Alex Flint summarizes the Center for Human-Compatible AI's assistance games research program.
• CHAI’s Andrew Critch and MILA’s David Krueger release “AI Research Considerations for Human Existential Safety (ARCHES)”, a review of 29 AI (existential) safety research directions, each with an illustrative analogy, examples of current work and potential synergies between research directions, and discussion of ways the research approach might lower (or raise) existential risk.
• OpenAI’s Danny Hernandez and Tom Brown present evidence that “for AI tasks with high levels of recent investment, algorithmic progress has yielded more gains than classical hardware efficiency”.
• DeepMind's Victoria Krakovna shares her takeaways from the COVID-19 pandemic for slow-takeoff scenarios.
• AI Impacts’ Daniel Kokotajlo discusses possible changes the world might undergo before reaching AGI.
• 80,000 Hours describes careers they view as promising but haven’t written up as priority career paths, including information security (previously discussed here).

MIRI researcher Evan Hubinger reviews “11 different proposals for building safe advanced AI under the current machine learning paradigm”, comparing them on outer alignment, inner alignment, training competitiveness, and performance competitiveness. 

Other updates

• We keep being amazed by new shows of support ⁠— following our last two announcements, MIRI has received a donation from another anonymous donor totaling ~$265,000 in euros, facilitated by Effective Giving UK and the Effective Altruism Foundation. Massive thanks to the donor for their generosity, and to both organizations for their stellar support for MIRI and other longtermist organizations!
• Hacker News discusses Eliezer Yudkowsky's There's No Fire Alarm for AGI.
• MIRI researcher Buck Shlegeris talks about deference and inside-view models on the EA Forum.
• OpenAI unveils GPT-3, a massive 175-billion parameter language model that can figure out how to solve a variety of problems without task-specific training or fine-tuning. Gwern Branwen's pithy summary:
  

  GPT-3 is terrifying because it's a tiny model compared to what's possible, trained in the dumbest way possible on a single impoverished modality on tiny data, yet the first version already manifests crazy runtime meta-learning—and the scaling curves still are not bending!

  Further discussion by Branwen and by Rohin Shah.

• Stuart Russell gives this year's Turing Lecture online, discussing “provably beneficial AI”.

MIRI has received an anonymous donation of ~$275,000 in euros, facilitated by Effective Giving UK. Additionally, the Survival and Flourishing Fund, working with funders Jaan Tallinn and Jed McCaleb, has announced $340,000 in grants to MIRI. SFF is a new fund that is taking over much of BERI's grantmaking work.

To everyone involved in both decisions to support our research, thank you!

Other updates

• An Orthodox Case Against Utility Functions: MIRI researcher Abram Demski makes the case against utility functions that rely on a microphysical “view from nowhere”.
• Stuart Armstrong’s “If I were a well-intentioned AI…” sequence looks at alignment problems from the perspective of a well-intentioned but ignorant agent.
• AI Impacts' Asya Bergal summarizes takeaways from “safety-by-default” researchers.
• Agarwal and Norouzi report improvements in offline RL.
• Daniel Kokotajlo’s Three Kinds of Competitiveness distinguishes performance-competitive, cost-competitive, and date-competitive AI systems.
• The Stanford Encyclopedia of Philosophy's new Ethics of AI and Robotics article includes a discussion of existential risk from AGI.

MIRI has been awarded its largest grant to date — $7,703,750 split over two years from Open Philanthropy, in partnership with Ben Delo, co-founder of the cryptocurrency trading platform BitMEX!

We have also been awarded generous grants by the Berkeley Existential Risk Initiative ($300,000) and the Long-Term Future Fund ($100,000). Our thanks to everybody involved!

Other updates

• Buck Shlegeris of MIRI and Rohin Shah of CHAI discuss Rohin's 2018–2019 overview of technical AI alignment research on the AI Alignment Podcast.
• From MIRI's Abram Demski: Thinking About Filtered Evidence Is (Very!) Hard and Bayesian Evolving-to-Extinction. And from Evan Hubinger: Synthesizing Amplification and Debate.
• From OpenAI's Beth Barnes, Paul Christiano, Long Ouyang, and Geoffrey Irving: Progress on AI Safety via Debate.
• Zoom In: An Introduction to Circuits: OpenAI's Olah, Cammarata, Schubert, Goh, Petrov, and Carter argue, “Features are the fundamental unit of neural networks. They correspond to directions [in the space of neuron activations]. […] Features are connected by weights, forming circuits. […] Analogous features and circuits form across models and tasks.”
• DeepMind's Agent57 appears to meet one of the AI benchmarks in AI Impacts' 2016 survey, “outperform professional game testers on all Atari games using no game specific knowledge”, earlier than NeurIPS/ICML authors predicted.
• From DeepMind Safety Research: Specification gaming: the flip side of AI ingenuity.

As the COVID-19 pandemic progresses, the LessWrong team has put together a database of resources for learning about the disease and staying updated, and 80,000 Hours has a new write-up on ways to help in the fight against COVID-19. In my non-MIRI time, I've been keeping my own quick and informal notes on various sources' COVID-19 recommendations in this Google Doc. Stay safe out there!

Updates

• My personal cruxes for working on AI safety: a talk transcript from MIRI researcher Buck Shlegeris.
• Daniel Kokotajlo of AI Impacts discusses Cortés, Pizarro, and Afonso as Precedents for Takeover.
• O'Keefe, Cihon, Garfinkel, Flynn, Leung, and Dafoe's “The Windfall Clause” proposes “an ex ante commitment by AI firms to donate a significant amount of any eventual extremely large profits” that result from “fundamental, economically transformative breakthroughs” like AGI.
• Microsoft announces the 17-billion-parameter language model Turing-NLG.
• Oren Etzioni thinks AGI is too far off to deserve much thought, and cites Andrew Ng's “overpopulation on Mars” metaphor approvingly — but he's also moving the debate in a very positive direction by listing specific observations that would make him change his mind.

Updates

• Colm reviews our 2019 fundraiser: taking into account matching, we received a total of $601,120 from 250+ donors. Our thanks again for all the support!
• Evan Hubinger's Exploring Safe Exploration clarifies points he raised in Safe Exploration and Corrigibility. The issues raised here are somewhat subtler than may be immediately apparent, since we tend to discuss things in ways that collapse the distinctions Evan is making.
• Logician Arthur Milchior reviews the AIRCS workshop and MIRI's application process based on his first-hand experience with both. See also follow-up discussion with MIRI and CFAR staff.
• Rohin Shah posts an in-depth 2018–19 review of the field of AI alignment.
• From Shevlane and Dafoe's new “The Offense-Defense Balance of Scientific Knowledge: Does Publishing AI Research Reduce Misuse?”:

   

  [T]he existing conversation around AI has heavily borrowed concepts and conclusions from one particular field: vulnerability disclosure in computer security. We caution against AI researchers treating these lessons as immediately applicable. There are important differences between vulnerabilities in software and the types of vulnerabilities exploited by AI. […]

  Patches to software are often easy to create, and can often be made in a matter of weeks. These patches fully resolve the vulnerability. The patch can be easily propagated: for downloaded software, the software is often automatically updated over the internet; for websites, the fix can take effect immediately.

  [… F]or certain technologies, there is no low-cost, straightforward, effective defence. [… C]onsider biological research that provides insight into the manufacture of pathogens, such as a novel virus. A subset of viruses are very difficult to vaccinate for (there is still no vaccination for HIV) or otherwise prepare against. This lowers the defensive benefit of publication, by blocking a main causal pathway by which publication leads to greater protection. This contrasts with the case where an effective treatment can be developed within a reasonable time period[.]

• Yann LeCun and Eliezer Yudkowsky discuss the concept “AGI”.
• CFAR's Anna Salamon contrasts “reality-revealing” and “reality-masking” puzzles.
• Scott Alexander reviews Stuart Russell's Human Compatible.

Links from the research team

MIRI researchers anonymously summarize and comment on recent posts and papers:

• Re ACDT: a hack-y acausal decision theory — “Stuart Armstrong calls this decision theory a hack. I think it might be more elegant than he's letting on (i.e., a different formulation could look less hack-y), and is getting at something.”
• Re Predictors exist: CDT going bonkers… forever — “I don't think Stuart Armstrong's example really adds much over some variants of Death in Damascus, but there's some good discussion of CDT vs. EDT stuff in the comments.”
• Re Is the term mesa optimizer too narrow? — “Matthew Barnett poses the important question, '[I]f even humans are not mesa optimizers, why should we expect mesa optimizers to be the primary real world examples of [malign generalization]?'”
• Re Malign generalization without internal search — “I think Matthew Barnett's question here is an important one. I lean toward the 'yes, this is a problem' camp—I don't think we can entirely eliminate malign generalization by eliminating internal search. But it is possible that this falls into other categories of misalignment (which we don't want to term 'inner alignment').”
• Re (A -> B) -> A in Causal DAGs and Formulating Reductive Agency in Causal Models — “I've wanted something like this for a while. Bayesian influence diagrams model agents non-reductively, by boldly asserting that some nodes are agentic. Can we make models which represent agents, without declaring a basic 'agent' type like that? John Wentworth offers an approach, representing agents via 'strange loops' across a use-mention boundary; and discusses how this might break down even further, with fully reductive agency. I'm not yet convinced that Wentworth has gotten it right, but it's exciting to see an attempt.”