“Singularity Hypotheses” Published

 |   |  Papers

singularity hypothesesSingularity Hypotheses: A Scientific and Philosophical Assessment has now been published by Springer, in hardcover and ebook forms.

The book contains 20 chapters about the prospect of machine superintelligence, including 4 chapters by MIRI researchers and research associates.

“Intelligence Explosion: Evidence and Import” (pdf) by Luke Muehlhauser and (previous MIRI researcher) Anna Salamon reviews

the evidence for and against three claims: that (1) there is a substantial chance we will create human-level AI before 2100, that (2) if human-level AI is created, there is a good chance vastly superhuman AI will follow via an “intelligence explosion,” and that (3) an uncontrolled intelligence explosion could destroy everything we value, but a controlled intelligence explosion would benefit humanity enormously if we can achieve it. We conclude with recommendations for increasing the odds of a controlled intelligence explosion relative to an uncontrolled intelligence explosion.

“Intelligence Explosion and Machine Ethics” (pdf) by Luke Muehlhauser and Louie Helm discusses the challenges of formal value systems for use in AI:

Many researchers have argued that a self-improving artificial intelligence (AI) could become so vastly more powerful than humans that we would not be able to stop it from achieving its goals. If so, and if the AI’s goals differ from ours, then this could be disastrous for humans. One proposed solution is to program the AI’s goal system to want what we want before the AI self-improves beyond our capacity to control it. Unfortunately, it is difficult to specify what we want. After clarifying what we mean by “intelligence,” we offer a series of “intuition pumps” from the field of moral philosophy for our conclusion that human values are complex and difficult to specify. We then survey the evidence from the psychology of motivation, moral psychology, and neuroeconomics that supports our position. We conclude by recommending ideal preference theories of value as a promising approach for developing a machine ethics suitable for navigating an intelligence explosion or “technological singularity.”

“Friendly Artificial Intelligence” by Eliezer Yudkowsky is a shortened version of Yudkowsky (2008).

Finally, “Artificial General Intelligence and the Human Mental Model” (pdf) by Roman Yampolskiy and (MIRI research associate) Joshua Fox  reviews the dangers of anthropomorphizing machine intelligences:

When the first artificial general intelligences are built, they may improve themselves to far-above-human levels. Speculations about such future entities are already affected by anthropomorphic bias, which leads to erroneous analogies with human minds. In this chapter, we apply a goal-oriented understanding of intelligence to show that humanity occupies only a tiny portion of the design space of possible minds. This space is much larger than what we are familiar with from the human example; and the mental architectures and goals of future superintelligences need not have most of the properties of human minds. A new approach to cognitive science and philosophy of mind, one not centered on the human example, is needed to help us understand the challenges which we will face when a power greater than us emerges.

The book also includes brief, critical responses to most chapters, including responses written by Eliezer Yudkowsky and (previous MIRI staffer) Michael Anissimov.

Altair’s Timeless Decision Theory Paper Published

 |   |  Papers

Altair paper frontDuring his time as a research fellow for MIRI, Alex Altair wrote a paper on Timeless Decision Theory (TDT) that has now been published:  “A Comparison of Decision Algorithms on Newcomblike Problems.”

Altair’s paper is both more succinct and also more precise in its formulation of TDT than Yudkowsky’s earlier paper “Timeless Decision Theory.” Thus, Altair’s paper should serve as a handy introduction to TDT for philosophers, computer scientists, and mathematicians, while Yudkowsky’s paper remains required reading for anyone interested to develop TDT further, for it covers more ground than Altair’s paper.

Altair’s abstract reads:

When formulated using Bayesian networks, two standard decision algorithms (Evidential Decision Theory and Causal Decision Theory) can be shown to fail systematically when faced with aspects of the prisoner’s dilemma and so-called “Newcomblike” problems. We describe a new form of decision algorithm, called Timeless Decision Theory, which consistently wins on these problems.

We may submit to a journal later, but we’ve published the current version to our website so that readers won’t need to wait two years (from submission to acceptance to publication) to read it.

For a gentle introduction to the entire field of normative decision theory (including TDT), see Muehlhauser and Williamson’s Decision Theory FAQ.

MIRI’s April newsletter: Relaunch Celebration and a New Math Result

 |   |  Newsletters

Greetings from The Executive Director

Dear friends,

These are exciting times at MIRI.

After years of awareness-raising and capacity-building, we have finally transformed ourselves into a research institute focused on producing the mathematical research required to build trustworthy (or “human-friendly”) machine intelligence. As our most devoted supporters know, this has been our goal for roughly a decade, and it is a thrill to have made the transition.

It is also exciting to see how much more quickly one can get academic traction with mathematics research, as compared to philosophical research and technological forecasting research. Within hours of publishing a draft of our first math result, Field Medalist Timothy Gowers had seen the draft and commented on it (here), along with several other professional mathematicians.

We celebrated our “relaunch” at an April 11th party in San Francisco. It was a joy to see old friends and make some new ones. You can see photos and read some details below.

For more detail on our new strategic priorities, see our blog post: MIRI’s Strategy for 2013.


Luke Muehlhauser
Executive Director

Read more »

MIRI’s Strategy for 2013

 |   |  MIRI Strategy

This post is not a detailed strategic plan. For now, I just want to provide an update on what MIRI is doing in 2013 and why.

Our mission remains the same. The creation of smarter-than-human intelligence will likely be the most significant event in human history, and MIRI exists to help ensure that this event has a positive impact.

Still, much has changed in the past year:

  • The short-term goals in our August 2011 strategic plan were largely accomplished.
  • We changed our name from “The Singularity Institute” to “The Machine Intelligence Research Institute” (MIRI).
  • We were once doing three things — research, rationality training, and the Singularity Summit. Now we’re doing one thing: research. Rationality training was spun out to a separate organization, CFAR, and the Summit was acquired by Singularity University. We still co-produce the Singularity Summit with Singularity University, but this requires limited effort on our part.
  • After dozens of hours of strategic planning in January–March 2013, and with input from 20+ external advisors, we’ve decided to (1) put less effort into public outreach, and to (2) shift our research priorities to Friendly AI math research.

It’s this last pair of changes I’d like to explain in more detail below.

Read more »

Facing the Intelligence Explosion ebook

 |   |  News

Facing the Intelligence Explosion is now available as an ebook!

You can get it here. It is available as a “pay-what-you-want” package that includes the ebook in three formats: MOBI, EPUB, and PDF.

It is also available on Amazon Kindle (US, Canada, UK, and most others) and the Apple iBookstore (US, Canada, UK and most others).

All sources are DRM-free. Grab a copy, share it with your friends, and review it on Amazon or the iBookstore.

All proceeds go directly to funding the technical and strategic research of the Machine Intelligence Research Institute.

The Lean Nonprofit

 |   |  MIRI Strategy

Can Lean Startup methods work for nonprofits?

The Lean Startup‘s author Eric Ries seems to think so:

A startup is a human institution designed to create a new product or service under conditions of extreme uncertainty… Anyone who is creating a new product or business under conditions of extreme uncertainty is an entrepreneur whether he or she knows it or not, and whether working in a government agency, a venture-backed company, a nonprofit, or a decidedly for-profit company with financial investors.

In the past year, I helped launch one new nonprofit (Center for Applied Rationality), I massively overhauled one older nonprofit (MIRI), and I consulted with many nonprofit CEOs and directors. Now I’d like to share some initial thoughts on the idea of a “Lean Nonprofit.”

Read more »

Early draft of naturalistic reflection paper

 |   |  Papers

Update: See Reflection in Probabilistic Logic for more details on how this result relates to MIRI’s research mission.

In a recent blog post we described one of the results of our 1st MIRI Workshop on Logic, Probability, and Reflection:

The participants worked on the foundations of probabilistic reflective reasoning. In particular, they showed that a careful formalization of probabilistic logic can circumvent many classical paradoxes of self-reference. Applied to metamathematics, this framework provides (what seems to be) the first definition of truth which is expressive enough for use in reflective reasoning.

In short, the result described is a “loophole” in Tarski’s undefinability theorem (1936).

An early draft of the paper describing this result is now available: download it here. Its authors are Paul Christiano (UC Berkeley), Eliezer Yudkowsky (MIRI), Marcello Herreshoff (Google), and Mihály Bárász (Google). An excerpt from the paper is included below:

Unfortunately, it is impossible for any expressive language to contain its own truth predicate True

There are a few standard responses to this challenge.

The first and most popular is to work with meta-languages…

A second approach is to accept that some sentences, such as the liar sentence G, are neither true nor false…

Although this construction successfully dodges the “undefinability of truth” it is somewhat unsatisfying. There is no predicate in these languages to test if a sentence… is undefined, and there is no bound on the number of sentences which remain undefined. In fact, if we are specifically concerned with self-reference, then a great number of properties of interest (and not just pathological counterexamples) become undefined.

In this paper we show that it is possible to perform a similar construction over probabilistic logic. Though a language cannot contain its own truth predicate True, it can nevertheless contain its own “subjective probability” function P. The assigned probabilities can be reflectively consistent in the sense of an appropriate analog of the reflection property 1. In practice, most meaningful assertions must already be treated probabilistically, and very little is lost by allowing some sentences to have probabilities intermediate between 0 and 1.

Another paper showing an application of this result to set theory is forthcoming.

March Newsletter

 |   |  Newsletters



Greetings From The Executive Director


As previously announced on our blog, the Singularity Institute has been renamed as the Machine Intelligence Research Institute (MIRI). Naturally, both our staff and our supporters have positive associations with our original name, the “Singularity Institute.” As such, any new name will feel strange for a time. However, “MIRI” has sounded better and better to us over the past several weeks, and we think it will grow on you, too.

Some will worry, “But ‘MIRI’ doesn’t express what you do in any detail!” According to our market research, however, this is “a feature, not a bug.” Researchers, in particular, said they could feel awkward working for an organization with a name that sounded too narrow or “partisan.” They also warned us that the scope of an organization’s activities can change over time, so its name should be very general. University departments and independent research organizations learned these lessons long ago, and thus tend to have very general names (with the universities themselves usually named after their primary campus location).

“MIRI” has other nice properties, too. It’s easy to spell, it’s easy to pronounce, and it reflects our shifting priorities toward more technical research. Our mission, of course, remains the same: “to ensure that the creation of smarter-than-human intelligence benefits society.”

See our new website at Intelligence.org. The site guide here.

Our emails have changed, too. Be sure to update your email Contacts list with our new email addresses, e.g. luke@intelligence.org. Our previous email addresses at singinst.org and singularity.org no longer work. You can see all our new email addresses on the Team page.


Luke Muehlhauser

Executive Director

Read more »