New report: “UDT with known search order”

 |   |  Papers

UDT with known search orderToday we release a new technical report from MIRI research associate Tsvi Benson-Tilsen: “UDT with known search order.” Abstract:

We consider logical agents in a predictable universe running a variant of updateless decision theory. We give an algorithm to predict the behavior of such agents in the special case where the order in which they search for proofs is simple, and where they know this order. As a corollary, “playing chicken with the universe” by diagonalizing against potential spurious proofs is the only way to guarantee optimal behavior for this class of simple agents.

Singularity2014.com appears to be a fake

 |   |  News

Earlier today I was alerted to the existence of Singularity2014.com (archived screenshot). MIRI has nothing to do with that website and we believe it is a fake.

The website claims there is a “Singularity 2014″ conference “in the Bay Area” on “November 9, 2014.” We believe that there is no such event. No venue is listed, tickets are supposedly sold out already, and there are no links to further information. The three listed speakers are unknown to us, and their supposed photos are stock photos (1, 2, 3). The website prominently features an image of Ray Kurzweil, but Ray Kurzweil’s press staff confirms that he has nothing to do with this event. The website also features childish insults and a spelling error.

The website claims the event is “staged and produced by former organizers of the Singularity Summit from the Machine Intelligence Research Institute,” and that “All profits benefit the Machine Intelligence Research Institute,” but MIRI has nothing to do with this supposed event.

The Singularity2014.com domain name was registered via eNom reseller NameCheap.com on September 15th, 2014 by someone other than us, and is associated with a P.O. Box in Panama.

MIRI is collaborating with Singularity University to have the website taken down. If you have information about who is responsible for this, please contact luke@intelligence.org.

The next Singularity Summit will be organized primarily by Singularity University; for more information see here.

 

Update: The website has been taken down.

New paper: “Corrigibility”

 |   |  Papers

CorrigibilityToday we release a paper describing a new problem area in Friendly AI research we call corrigibility. The report (PDF) is co-authored by MIRI’s Friendly AI research team (Eliezer Yudkowsky, Benja Fallenstein, Nate Soares) and also Stuart Armstrong from the Future of Humanity Institute at Oxford University.

The abstract reads:

As artificially intelligent systems grow in intelligence and capability, some of their available options may allow them to resist intervention by their programmers. We call an AI system “corrigible” if it cooperates with what its creators regard as a corrective intervention, despite default incentives for rational agents to resist attempts to shut them down or modify their preferences. We introduce the notion of corrigibility and analyze utility functions that attempt to make an agent shut down safely if a shutdown button is pressed, while avoiding incentives to prevent the button from being pressed or cause the button to be pressed, and while ensuring propagation of the shutdown behavior as it creates new subsystems or self-modifies. While some proposals are interesting, none have yet been demonstrated to satisfy all of our intuitive desiderata, leaving this simple problem in corrigibility wide-open.

This paper was accepted to the AI & Ethics workshop at AAAI-2015.

Update: The slides for Nate Soares’ presentation at AAAI-15 are available here.

AGI outcomes and civilizational competence

 |   |  Analysis

David Victor

The [latest IPCC] report says, “If you put into place all these technologies and international agreements, we could still stop warming at [just] 2 degrees.” My own assessment is that the kinds of actions you’d need to do that are so heroic that we’re not going to see them on this planet.

—David Victor,1 professor of international relations at UCSD

 

A while back I attended a meeting of “movers and shakers” from science, technology, finance, and politics. We were discussing our favorite Big Ideas for improving the world. One person’s Big Idea was to copy best practices between nations. For example when it’s shown that nations can dramatically improve organ donation rates by using opt-out rather than opt-in programs, other countries should just copy that solution.

Everyone thought this was a boring suggestion, because it was obviously a good idea, and there was no debate to be had. Of course, they all agreed it was also impossible and could never be established as standard-practice. So we moved on to another Big Idea that was more tractable.

Later, at a meeting with a similar group of people, I told some economists that their recommendations on a certain issue were “straightforward econ 101,” and I didn’t have any objections to share. Instead, I asked, “But how can we get policy-makers to implement econ 101 solutions?” The economists laughed and said, “Well, yeah, we have no idea. We probably can’t.”

How do I put this? This is not a civilization that should be playing with self-improving AGIs.2
Read more »

  1. Quote taken from the Radiolab episode titled “In the Dust of This Planet.” ↩
  2. In Superintelligence, Bostrom made the point this way (p. 259):

    Before the prospect of an intelligence explosion, we humans are like small children playing with a bomb. Such is the mismatch between the power of our plaything and the immaturity of our conduct… For a child with an undetonated bomb in its hands, a sensible thing to do would be to put it down gently, quickly back out of the room, and contact the nearest adult. Yet what we have here is not one child but many, each with access to an independent trigger mechanism. The chances that we will all find the sense to put down the dangerous stuff seem almost negligible… Nor can we attain safety by running away, for the blast of an intelligence explosion would bring down the entire firmament. Nor is there a grown-up in sight.

     ↩

MIRI’s October Newsletter

 |   |  Newsletters

 

 

Machine Intelligence Research Institute

Research Updates

  • Our major project last month was our Friendly AI technical agenda overview and supporting papers, the former of which is now in late draft form but not yet ready for release.
  • 4 new expert interviews, including John Fox on AI safety.
  • MIRI research fellow Nate Soares has begun to explain some of the ideas motivating MIRI’s current research agenda at his blog. See especially Newcomblike problems are the norm.

News Updates

As always, please don’t hesitate to let us know if you have any questions or comments.

Best,
Luke Muehlhauser
Executive Director

 

 

Kristinn Thórisson on constructivist AI

 |   |  Conversations

krisDr. Kristinn R. Thórisson is an Icelandic Artificial Intelligence researcher, founder of the Icelandic Institute for Intelligent Machines (IIIM) and co-founder and former co-director of CADIA: Center for Analysis and Design of Intelligent Agents. Thórisson is one of the leading proponents of artificial intelligence systems integration. Other proponents of this approach are researchers such as Marvin Minsky, Aaron Sloman and Michael A. Arbib. Thórisson is a proponent of Artificial General Intelligence (AGI) (also referred to as Strong AI) and has proposed a new methodology for achieving artificial general intelligence. A demonstration of this constructivist AI methodology has been given in the FP-7 funded HUMANOBS project HUMANOBS project, where an artificial system autonomously learned how to do spoken multimodal interviews by observing humans participate in a TV-style interview. The system, called AERA, autonomously expands its capabilities through self-reconfiguration. Thórisson has also worked extensively on systems integration for artificial intelligence systems in the past, contributing architectural principles for infusing dialogue and human-interaction capabilities into the Honda ASIMO robot.

Kristinn R. Thórisson is currently managing director for the Icelandic Institute for Intelligent Machines and an associate professor at the School of Computer Science at Reykjavik University. He was co-founder of semantic web startup company Radar Networks, and served as its Chief Technology Officer 2002-2003.

 

Luke Muehlhauser: In some recent articles (1, 2, 3) you contrast “constructionist” and “constructivist” approaches in AI. Constructionist AI builds systems piece by piece, by hand, whereas constructivist AI builds and grows systems largely by automated methods.

Constructivist AI seems like a more general form of the earlier concept of “seed AI.” How do you see the relation between the two concepts?

Kristinn Thorisson: We sometimes use “seed AI”, or even “developmental AI”, when we describe what we are doing – it is often a difficult task to find a good term for an interdisciplinary research program, because each term will bring various things up in the mind of people depending on their background. There are subtle differences between both the meanings and histories of these terms that each bring along several pros and cons for each one.

I had been working on integrated constructionist systems for close to two decades, where the main focus was on how to integrate many things into a coherent system. When my collaborators and I started to seriously think about how to achieve artificial general intelligence we tired to explain, among other things, how transversal functions – functions of mind that seem to touch pretty much everything in a mind, such as attention, reasoning, and learning – could efficiently and sensibly be implemented in a single AI system. We also looked deeper into autonomy than I had done previously. This brought up all sorts of questions that were new to me, like: What is needed for implementing a system that can act relatively autonomously *after it leaves the lab*, without the constant intervention of its designers, and is capable of learning a pretty broad range of relatively unrelated things, on its own, and deal with new tasks, scenarios and environments – that were relatively unforeseen by the system’s designers? Read more »

Nate Soares speaking at Purdue University

 |   |  News

On Thursday, September 18th Purdue University is hosting the seminar Dawn or Doom: The New Technology Explosion. Speakers include James Barrat, author of Our Final Invention, and MIRI research fellow Nate Soares.

Nate’s talk title and abstract are:

Why ain’t you rich?: Why our current understanding of “rational choice” isn’t good enough for superintelligence.

The fate of humanity could one day depend upon the choices of a superintelligent AI. How will those choices be made? Philosophers have long attempted to define what it means to make rational decisions, but in the context of machine intelligence, these theories turn out to have undesirable consequences.

For example, there are many games where modern decision theories lose systematically. New decision procedures are necessary in order to fully capture an idealization of the way we make decisions.

Furthermore, existing decision theories are not stable under reflection: a self-improving machine intelligence using a modern decision theory would tend to modify itself to use a different decision theory instead. It is not yet clear what sort of decision process it would end up using, nor whether the end result would be desirable. This indicates that our understanding of decision theories is inadequate for the construction of a superintelligence.

Can we find a formal theory of “rationality” that we would want a superintelligence to use? This talk will introduce the concepts above in more detail, discuss some recent progress in the design of decision theories, and then give a brief overview of a few open problems.

For details on how to attend Nate’s talk and others, see here.