MIRI’s September Newsletter
|
||||
|
|
||||
|
Laurent Orseau is an associate professor (maître de conférences) since 2007 at AgroParisTech, Paris, France. In 2003, he graduated from a professional master in computer science at the National Institute of Applied Sciences in Rennes and from a research master in artificial intelligence at University of Rennes 1. He obtained his PhD in 2007. His goal is to build a practical theory of artificial general intelligence. With his co-author Mark Ring, they have been awarded the Solomonoff AGI Theory Prize at AGI’2011 and the Kurzweil Award for Best Idea at AGI’2012.
Luke Muehlhauser: In the past few years you’ve written some interesting papers, often in collaboration with Mark Ring, that use AIXI-like models to analyze some interesting features of different kinds of advanced theoretical agents. For example in Ring & Orseau (2011), you showed that some kinds of advanced agents will maximize their rewards by taking direct control of their input stimuli — kind of like the rats who “wirehead” when scientists give them direct control of the input stimuli to their reward circuitry (Olds & Milner 1954). At the same time, you showed that at least one kind of agent, the “knowledge-based” agent, does not wirehead. Could you try to give us an intuitive sense of why some agents would wirehead, while the knowledge-based agent would not?
Laurent Orseau: You’re starting with a very interesting question!
A recent xkcd comic described the Saturn V rocket using only the 1000 most frequently used words (in English). The rocket was called “up-goer five,” and the liquid hydrogen feed line was the “thing that lets in cold wet air to burn.” This inspired a geneticist to make the Up-Goer Five Text Editor, which forces you to use only the 1000 most frequent words. Mental Floss recently collected 18 scientific ideas explained using this restriction.
What does this have to do with MIRI? Well, young philosopher Robby Bensinger has now re-written MIRI’s Five Theses using the Up-Goer Five text editor, with amusing results:
That is all. You’re welcome.
MIRI aims to do research now that increases humanity’s odds of successfully managing important AI-related events that are at least a few decades away. Thus, we’d like to know: To what degree can we take actions now that will predictably have positive effects on AI-related events decades from now? And, which factors predict success and failure in planning for decades-distant events that share important features with future AI events?
Or, more generally: How effectively can humans plan for future decades? Which factors predict success and failure in planning for future decades?
To investigate these questions, we asked Jonah Sinick to examine historical attempts to plan for future decades and summarize his findings. We pre-committed to publishing our entire email exchange on the topic (with minor editing), just as Jonah had done previously with GiveWell on the subject of insecticide-treated nets. The post below is a summary of findings from our full email exchange (.pdf) so far.
We decided to publish our initial findings after investigating only a few historical cases. This allows us to gain feedback on the value of the project, as well as suggestions for improvement, before continuing. It also means that we aren’t yet able to draw any confident conclusions about our core questions.
The most significant results from this project so far are:
The project has also produced a chapter-by-chapter list of some key lessons from Nate Silver’s The Signal and the Noise, available here.
Further details are given below. For sources and more, please see our full email exchange (.pdf).
In late 2008, economist Robin Hanson and AI theorist Eliezer Yudkowsky conducted an online debate about the future of artificial intelligence, and in particular about whether generally intelligent AIs will be able to improve their own capabilities very quickly (a.k.a. “foom”). James Miller and Carl Shulman also contributed guest posts to the debate.
The debate is now available as an eBook in various popular formats (PDF, EPUB, and MOBI). It includes:
Comments from the authors are included at the end of each chapter, along with a link to the original post.
Head over to intelligence.org/ai-foom-debate/ to download a free copy.
Stephen Hsu is Vice-President for Research and Graduate Studies and Professor of Theoretical Physics at Michigan State University. Educated at Caltech and Berkeley, he was a Harvard Junior Fellow and held faculty positions at Yale and the University of Oregon. He was also founder of SafeWeb, an information security startup acquired by Symantec. Hsu is a scientific advisor to BGI and a member of its Cognitive Genomics Lab.
Luke Muehlhauser: I’d like to start by familiarizing our readers with some of the basic facts relevant to the genetic architecture of cognitive ability, which I’ve drawn from the first half of a presentation you gave in February 2013:
From November 23-29, MIRI will host another Workshop on Logic, Probability, and Reflection, for the first time in Oxford, UK.
Participants will investigate problems related to reflective agents, probabilistic logic, and priors over logical statements / the logical omniscience problem.
Participants confirmed so far include:
If you have a strong mathematics background and might like to attend this workshop, it’s not too late to apply! And even if this workshop doesn’t fit your schedule, please do apply, so that we can notify you of other workshops (long before they are announced publicly).
In this post, I aim to summarize one common view on AI transparency and AI reliability. It’s difficult to identify the field’s “consensus” on AI transparency and reliability, so instead I will present a common view so that I can use it to introduce a number of complications and open questions that (I think) warrant further investigation.
Here’s a short version of the common view I summarize below:
Black box testing can provide some confidence that a system will behave as intended, but if a system is built such that it is transparent to human inspection, then additional methods of reliability verification are available. Unfortunately, many of AI’s most useful methods are among its least transparent. Logic-based systems are typically more transparent than statistical methods, but statistical methods are more widely used. There are exceptions to this general rule, and some people are working to make statistical methods more transparent.
Nusser (2009) writes:
…in the field of safety-related applications it is essential to provide transparent solutions that can be validated by domain experts. “Black box” approaches, like artificial neural networks, are regarded with suspicion – even if they show a very high accuracy on the available data – because it is not feasible to prove that they will show a good performance on all possible input combinations.
Unfortunately, there is often a tension between AI capability and AI transparency. Many of AI’s most powerful methods are also among its least transparent:
Methods that are known to achieve a high predictive performance — e.g. support vector machines (SVMs) or artificial neural networks (ANNs) — are usually hard to interpret. On the other hand, methods that are known to be well-interpretable — for example (fuzzy) rule systems, decision trees, or linear models — are usually limited with respect to their predictive performance.1
But for safety-critical systems — and especially for AGI — it is important to prioritize system reliability over capability. Again, here is Nusser (2009):
strict requirements [for system transparency] are necessary because a safety-related system is a system whose malfunction or failure can lead to serious consequences — for example environmental harm, loss or severe damage of equipment, harm or serious injury of people, or even death. Often, it is impossible to rectify a wrong decision within this domain.