One of our biggest updates, I'm happy to announce, is that we've hired five new research staff, with a sixth to join us in February. For details, see Workshops and Scaling Up in the fundraiser post.

Also: Facebook's Giving Tuesday matching opportunity is tomorrow at 5:00am PT! See Colm's post for details on how to get your donation matched.

Other updates

• Our most recent hire, “Risks from Learned Optimization” co-author Evan Hubinger, describes what he'll be doing at MIRI. See also Nate Soares' comment on how MIRI does nondisclosure-by-default.
• Buck Shlegeris discusses EA residencies as an outreach opportunity.
• OpenAI releases Safety Gym, a set of tools and environments for incorporating safety constraints into RL tasks.
• CHAI is seeking interns; application deadline Dec. 15.

Thoughts from the research team

This month, I'm trying something new: quoting MIRI researchers' summaries and thoughts on recent AI safety write-ups.

I've left out names so that these can be read as a snapshot of people's impressions, rather than a definitive “Ah, researcher X believes Y!” Just keep in mind that these will be a small slice of thoughts from staff I've recently spoken to, not anything remotely like a consensus take.

• Re Will transparency help catch deception? — “A good discussion of an important topic. Matthew Barnett suggests that any weaknesses in a transparency tool may turn it into a detrimental middle-man, and directly training supervisors to catch deception may be preferable.”
• Re Chris Olah’s views on AGI safety — “I very much agree with Evan Hubinger's idea that collecting different perspectives — different ‘hats’ — is a useful thing to do. Chris Olah's take on transparency is good to see. The concept of microscope AI seems like a useful one, and Olah's vision of how the ML field could be usefully shifted is quite interesting.”
• Re Defining AI Wireheading — “Stuart Armstrong takes a shot at making a principled distinction between wireheading and the rest of Goodhart.”
• Re How common is it to have a 3+ year lead? — “This seems like a pretty interesting question for AI progress models. The expected lead time and questions of expected takeoff speed greatly influence the extent to which winner-take-all dynamics are plausible.”
• Re Thoughts on Implementing Corrigible Robust Alignment — “Steve Byrnes provides a decent overview of some issues around getting ‘pointer’ type values.”

In other news, MIRI researcher Buck Shlegeris has written over 12,000 words on a variety of MIRI-relevant topics in an EA Forum AMA. (Example topics: advice for software engineers; what alignment plans tend to look like; and decision theory.)

Other updates

• Abram Demski's The Parable of Predict-O-Matic is a great read: the predictor/optimizer issues it covers are deep, but I expect a fairly wide range of readers to enjoy it and get something out of it.
• Evan Hubinger's Gradient Hacking describes an important failure mode that hadn't previously been articulated.
• Vanessa Kosoy's LessWrong shortform has recently discussed some especially interesting topics related to her learning-theoretic agenda.
• Stuart Armstrong's All I Know Is Goodhart constitutes nice conceptual progress on expected value maximizers that are aware of Goodhart's law and trying to avoid it.
• Reddy, Dragan, and Levine's paper on modeling human intent cites (of all things) Harry Potter and the Methods of Rationality as inspiration. 

News and links

• Artificial Intelligence Research Needs Responsible Publication Norms: Crootof provides a good review of the issue on Lawfare.
• Stuart Russell's new book is out: Human Compatible: Artificial Intelligence and the Problem of Control (excerpt). Rohin Shah's review does an excellent job of contextualizing Russell's views within the larger AI safety ecosystem, and Rohin highlights the quote:
  

  The task is, fortunately, not the following: given a machine that possesses a high degree of intelligence, work out how to control it. If that were the task, we would be toast. A machine viewed as a black box, a fait accompli, might as well have arrived from outer space. And our chances of controlling a superintelligent entity from outer space are roughly zero. Similar arguments apply to methods of creating AI systems that guarantee we won’t understand how they work; these methods include whole-brain emulation — creating souped-up electronic copies of human brains — as well as methods based on simulated evolution of programs. I won’t say more about these proposals because they are so obviously a bad idea.

• Jacob Steinhardt releases an AI Alignment Research Overview.
• Patrick LaVictoire's AlphaStar: Impressive for RL Progress, Not for AGI Progress raises some important questions about how capable today's state-of-the-art systems are.

Updates

• Ben Pace summarizes a second round of AI Alignment Writing Day posts.
• The Zettelkasten Method: MIRI researcher Abram Demski describes a note-taking system that's had a large positive effect on his research productivity.
• Will MacAskill writes a detailed critique of functional decision theory; Abram Demski (1, 2) and Matthew Graves respond in the comments.

News and links

• Recent AI alignment posts: Evan Hubinger asks “Are minimal circuits deceptive?”, Paul Christiano describes the strategy-stealing assumption, and Wei Dai lists his resolved confusions about Iterated Distillation and Amplification. See also Rohin Shah's comparison of recursive approaches to AI alignment.
• Also on LessWrong: A Debate on Instrumental Convergence Between LeCun, Russell, Bengio, Zador, and More.
• FHI's Ben Garfinkel and Allan Dafoe argue that conflicts between nations tend to exhibit “offensive-then-defensive scaling”.
• OpenAI releases a follow-up report on GPT-2, noting that several groups “have explicitly adopted similar staged release approaches” to OpenAI.
• NVIDIA Applied Deep Learning Research has trained a model that appears to essentially replicate GPT-2, with 5.6x as many parameters, slightly better WikiText perplexity, and slightly worse LAMBADA accuracy. The group has elected to share their training and evaluation code, but not the model weights.
• OpenAI fine-tunes GPT-2 for text continuation and summarization tasks that incorporate human feedback, noting, “Our motivation is to move safety techniques closer to the general task of ‘machines talking to humans,’ which we believe is key to extracting information about human values.”

Updates

• We ran a very successful MIRI Summer Fellows Program, which included a day where participants publicly wrote up their thoughts on various AI safety topics. See Ben Pace’s first post in a series of roundups.
• A few highlights from the writing day: Adele Lopez's Optimization Provenance; Daniel Kokotajlo's Soft Takeoff Can Still Lead to Decisive Strategic Advantage and The "Commitment Races" Problem; Evan Hubinger's Towards a Mechanistic Understanding of Corrigibility; and John Wentworth's Markets are Universal for Logical Induction and Embedded Agency via Abstraction.
• New posts from MIRI staff and interns: Abram Demski's Troll Bridge; Matthew Graves' View on Factored Cognition; Daniel Filan's Verification and Transparency; and Scott Garrabrant's Intentional Bucket Errors and Does Agent-like Behavior Imply Agent-like Architecture?
• See also a forum discussion on "proof-level guarantees" in AI safety.

News and links

• From Ben Cottier and Rohin Shah: Clarifying Some Key Hypotheses in AI Alignment
• Classifying Specification Problems as Variants of Goodhart's Law: Victoria Krakovna and Ramana Kumar relate DeepMind's SRA taxonomy to mesa-optimizers, selection and control, and Scott Garrabrant's Goodhart taxonomy. Also new from DeepMind: Ramana, Tom Everitt, and Marcus Hutter's Designing Agent Incentives to Avoid Reward Tampering.
• From OpenAI: Testing Robustness Against Unforeseen Adversaries. 80,000 Hours also recently interviewed OpenAI's Paul Christiano, with some additional material on decision theory.
• From AI Impacts: Evidence Against Current Methods Leading to Human-Level AI and Ernie Davis on the Landscape of AI Risks
• From Wei Dai: Problems in AI Alignment That Philosophers Could Potentially Contribute To
• Richard Möhn has put together a calendar of upcoming AI alignment events.
• The Berkeley Existential Risk Initiative is seeking an Operations Manager.

Updates

• MIRI research associate Stuart Armstrong is offering $1000 for good questions to ask an Oracle AI.
• Recent AI safety posts from Stuart: Indifference: Multiple Changes, Multiple Agents; Intertheoretic Utility Comparison: Examples; Normalising Utility as Willingness to Pay; and Partial Preferences Revisited.
• MIRI researcher Buck Shlegeris has put together a quick and informal AI safety reading list.
• There's No Fire Alarm for AGI reports on a researcher's January 2017 prediction that “in the next two years, we will not get 80, 90%” on Winograd schemas, an NLP test. Although this prediction was correct, researchers at Microsoft, Carnegie Mellon and Google Brain, and Facebook have now (2.5 years later) achieved Winograd scores of 89.0 and 90.4.
• Ortega et al.'s “Meta-Learning of Sequential Strategies” includes a discussion of mesa-optimization, independent of Hubinger et al.'s “Risks from Learned Optimization in Advanced Machine Learning Systems,” under the heading of “spontaneous meta-learning.”

News and links

• Wei Dai outlines forum participation as a research strategy.
• On a related note, the posts on the AI Alignment Forum this month were very good — I'll spotlight them all this time around. Dai wrote on the purposes of decision theory research; Shah on learning biases and rewards simultaneously; Kovarik on AI safety debate and its applications; Steiner on the Armstrong agenda and the intentional stance; Trazzi on manipulative AI; Cohen on IRL and imitation; and Manheim on optimizing and Goodhart effects (1, 2, 3).
• Jade Leung discusses AI governance on the AI Alignment Podcast.
• CMU and Facebook researchers' Pluribus program beats human poker professionals ⁠— using only $144 in compute. The developers also choose not to release the code: “Because poker is played commercially, the risk associated with releasing the code outweighs the benefits.”
• Microsoft invests $1 billion in OpenAI. From Microsoft's press release: “Through this partnership, the companies will accelerate breakthroughs in AI and power OpenAI’s efforts to create artificial general intelligence (AGI).” OpenAI has also released a paper on “The Role of Cooperation in Responsible AI Development.”
• Ought has a new preferred introduction to their work. See also Paul Christiano's Ought: Why it Matters and Ways to Help.
• FHI has 11 open research positions; applications are due by Aug. 16. You can also apply to CSER's AGI risk research associate position through Aug. 26.

In other news, we received an Ethereum donation worth $230,910 from Vitalik Buterin — the inventor and co-founder of Ethereum, and now our third-largest all-time supporter!

Also worth highlighting, from the Open Philanthropy Project's Claire Zabel and Luke Muehlhauser: there's a pressing need for security professionals in AI safety and biosecurity.

It’s more likely than not that within 10 years, there will be dozens of GCR-focused roles in information security, and some organizations are already looking for candidates that fit their needs (and would hire them now, if they found them).

It’s plausible that some people focused on high-impact careers (as many effective altruists are) would be well-suited to helping meet this need by gaining infosec expertise and experience and then moving into work at the relevant organizations.

Other updates

• Mesa Optimization: What It Is, And Why We Should Care — Rohin Shah's consistently excellent Alignment Newsletter discusses “Risks from Learned Optimization…” and other recent AI safety work.
• MIRI Research Associate Stuart Armstrong releases his Research Agenda v0.9: Synthesising a Human's Preferences into a Utility Function.
• OpenAI and MIRI staff help talk Munich student Connor Leahy out of releasing an attempted replication of OpenAI's GPT-2 model. (LessWrong discussion.) Although Leahy's replication attempt wasn't successful, write-ups like his suggest that OpenAI's careful discussion surrounding GPT-2 is continuing to prompt good reassessments of publishing norms within ML. Quoting from Leahy's postmortem:

   

  Sometime in the future we will have reached a point where the consequences of our research are beyond what we can discover in a one-week evaluation cycle. And given my recent experiences with GPT2, we might already be there. The more complex and powerful our technology becomes, the more time we should be willing to spend in evaluating its consequences. And if we have doubts about safety, we should default to caution.

  We tend to live in an ever accelerating world. Both the industrial and academic R&D cycles have grown only faster over the decades. Everyone wants “the next big thing” as fast as possible. And with the way our culture is now, it can be hard to resist the pressures to adapt to this accelerating pace. Your career can depend on being the first to publish a result, as can your market share.

  We as a community and society need to combat this trend, and create a healthy cultural environment that allows researchers to take their time. They shouldn’t have to fear repercussions or ridicule for delaying release. Postponing a release because of added evaluation should be the norm rather than the exception. We need to make it commonly accepted that we as a community respect others’ safety concerns and don’t penalize them for having such concerns, even if they ultimately turn out to be wrong. If we don’t do this, it will be a race to the bottom in terms of safety precautions.

   

• From Abram Demski: Selection vs. Control; Does Bayes Beat Goodhart?; and Conceptual Problems with Updateless Decision Theory and Policy Selection
• Vox's Future Perfect Podcast interviews Jaan Tallinn and discusses MIRI's role in originating and propagating AI safety memes.
• The AI Does Not Hate You, journalist Tom Chivers' well-researched book about the rationality community and AI risk, is out in the UK.

News and links

• Other recent AI safety write-ups: David Krueger's Let's Talk About “Convergent Rationality”; Paul Christiano's Aligning a Toy Model of Optimization; and Owain Evans, William Saunders, and Andreas Stuhlmüller's Machine Learning Projects on Iterated Distillation and Amplification
• From DeepMind: Vishal Maini puts together an AI reading list, Victoria Krakovna recaps the ICLR Safe ML workshop, and Pushmeet Kohli discusses AI safety on the 80,000 Hours Podcast.
• The EA Foundation is awarding grants for “efforts to reduce risks of astronomical suffering (s-risks) from advanced artificial intelligence”; apply by Aug. 11.
• Additionally, if you're a young AI safety researcher (with a PhD) based at a European university or nonprofit, you may want to apply for ~$60,000 in funding from the Bosch Center for AI.