March Newsletter

Greetings From The Executive Director

Friends,

As previously announced on our blog, the Singularity Institute has been renamed as the Machine Intelligence Research Institute (MIRI). Naturally, both our staff and our supporters have positive associations with our original name, the “Singularity Institute.” As such, any new name will feel strange for a time. However, “MIRI” has sounded better and better to us over the past several weeks, and we think it will grow on you, too.

Some will worry, “But ‘MIRI’ doesn’t express what you do in any detail!” According to our market research, however, this is “a feature, not a bug.” Researchers, in particular, said they could feel awkward working for an organization with a name that sounded too narrow or “partisan.” They also warned us that the scope of an organization’s activities can change over time, so its name should be very general. University departments and independent research organizations learned these lessons long ago, and thus tend to have very general names (with the universities themselves usually named after their primary campus location).

“MIRI” has other nice properties, too. It’s easy to spell, it’s easy to pronounce, and it reflects our shifting priorities toward more technical research. Our mission, of course, remains the same: “to ensure that the creation of smarter-than-human intelligence benefits society.”

See our new website at Intelligence.org. The site guide here.

Our emails have changed, too. Be sure to update your email Contacts list with our new email addresses, e.g. luke@intelligence.org. Our previous email addresses at singinst.org and singularity.org no longer work. You can see all our new email addresses on the Team page.

Cheers,

Luke Muehlhauser

Executive Director

Upcoming MIRI Research Workshops

From November 11-18, 2012, we held (what we now call) the 1st MIRI Workshop on Logic, Probability, and Reflection. The four workshop participants (Eliezer Yudkowsky, Paul Christiano, Marcello Herreschoff, and Mihály Bárász) worked on the foundations of probabilistic reflective reasoning. In particular, they showed that a careful formalization of probabilistic logic can circumvent many classical paradoxes of self-reference. Applied to metamathematics, this framework provides (what seems to be) the first definition of truth which is expressive enough for use in reflective reasoning. Applied to set theory, this framework provides an implementation of probabilistic set theory based on unrestricted comprehension which is nevertheless powerful enough to formalize ordinary mathematical reasoning (in contrast with similar fuzzy set theories, which were originally proposed for this purpose but later discovered to be incompatible with mathematical induction).

These results suggest a similar approach may be used to work around Löb’s theorem, but this has not yet been explored. This work will be written up over the coming months.

In the meantime, MIRI is preparing for the 2nd MIRI Workshop on Logic, Probability, and Reflection, to take place from April 3-24, 2013. For more details, see the relevant blog post.

Additional MIRI research workshops are also tentatively planned for the summer and fall of 2013.

Winter Fundraiser Success!

Thanks to our dedicated supporters, we met our goal for our 2012 Winter Fundraiser. Thank you!

The fundraiser ran for 45 days, from December 6, 2012 to January 20, 2013.

We met our $115,000 goal, raising a total of $230,000 for our operations in 2013.

Course Recommendations for MIRI Researchers

MIRI Deputy Director Louie Helm has prepared a list of Recommend Courses for MIRI Researchers, which answers the question “What should a researcher study if they want to equip themselves to tackle the technical problems on MIRI’s research agenda?” This new page provides a list of subjects to study, along with textbook recommendations, online course recommendations, and recommended courses at particular universities (UC Berkeley, Stanford, MIT, and CMU).

Decision Theory FAQ

If you want future AIs to cooperate in real-world prisoner’s dilemmas, you’d better hope they’re not using any of the standard decision algorithms discussed in philosophy and computer science journals. For this reason and others, decision theory represents a major focus of MIRI’s research agenda (for example see Yudkowsky 2010).

To help clarify some common confusions about decision theory and encourage more researchers to tackle these problems, MIRI Executive Director Luke Muehlhauser wrote a Decision Theory FAQ for the website Less Wrong. It is by far the most comprehensive decision theory FAQ on the internet, and section 11 is an especially handy summary of how different decision algorithms perform on a battery of standard problems from the literature (Newcomb’s Problem, Medical Newcomb’s Problem, Egan’s Psychopath Button, Parfit’s Hitchhiker, the Prisoner’s Dilemma, and more).

Brief History of Ethically Concerned Scientists

In 1956, Norbert Weiner wrote that “For the first time in history, it has become possible for a limited group of a few thousand people to threaten the absolute destruction of millions.” Today, the general attitude towards scientific discovery is that scientists are not themselves responsible for how their work is used. But this is not necessarily the attitude that we should encourage. As technology becomes more powerful, it also becomes more dangerous.

To celebrate the scientists who took seriously the potential social consequences of their work, and to make it easier for others to write about scientist’s social responsibility, MIRI researcher Kaj Sotala published A Brief History of Ethically Concerned Scientists. Click through to learn about:

• John Napier (1550-1617), who discovered a deadly new form of artillery, but kept its details a secret so that its destructive power could not be wielded.
• Lewis Fry Richardson (1881-1953), who turned down an invitation to optimize the spread of poison gas for the British military, destroyed his unpublished research, left meteorology, and began to study the causes of war instead, hoping to reduce armed conflict.
• Leó Szilárd (1898-1964), who discovered the nuclear chain reaction but arranged for his patent details to be kept secret so they could not be used by Germany to develop atomic bombs, and later campaigned against nuclear proliferation.
• Joseph Rotblat (1908-2005), who left the Manhattan Project over ethical concerns with the atomic bomb and campaigned against nuclear proliferation.

and many others.

Existential Risk Covered in Aeon Magazine

We don’t mention each new article about existential risk or AI risk, but this one by Ross Andersen in Aeon Magazine is particularly good. It’s based largely on the work of Nick Bostrom at Oxford University, a frequent collaborator with MIRI researchers (e.g. “The Ethics of Artificial Intelligence“). Bostrom is currently writing a scholarly monograph on machine superintelligence, and Andersen’s article properly highlights the centrality of AI risk. The piece also includes snippets of a conversation with MIRI research associate Daniel Dewey (author of “Learning What to Value“).

We also recommend Bostrom’s new article “Existential Risk Prevention as Global Priority,” forthcoming in Global Policy.

MetaMed Launches

Former MIRI President Michael Vassar’s new personalized medicine company has finally launched: behold MetaMed! MetaMed offers personalized medical research for patients who want to make sure they’re treatment is informed by the very latest medical breakthroughs. Eliezer Yudkowsky introduced the company thusly:

An Appreciation of Michael Anissimov

Due to Singularity University’s acquisition of the Singularity Summit and some major changes to MIRI’s public communications strategy, Michael Anissimov left MIRI in January 2013. Michael continues to support our mission and continues to volunteer for us.

It was a pleasure for me to work with Michael during our overlapping time at MIRI. Michael played a major role in “onboarding” me at MIRI and helping me to understand the history and culture of MIRI’s community, and he worked very hard on the Singularity Summit and on our 2012 efforts to transform MIRI into a more effective organization in general.

I owe Michael much gratitude for his many, many years of service to MIRI, and in particular for helping to build up the Singularity Summit to the point where it was acquired, and for applying himself (of his own accord) to the tasks that he saw needed to be done — for example in taking up MIRI’s public communications mantle when he saw that was a gap in our operations.

Michael: Thanks so much for your service to MIRI! I enjoyed working with you, and I wish you the best of luck on your future adventures.

Luke Muehlhauser