About MIRI

The Machine Intelligence Research Institute is a research nonprofit focused on the mathematical underpinnings of intelligent behavior. Our mission is to develop formal tools for the clean design and analysis of general-purpose artificial intelligence algorithms, with the intent of making such algorithms safe and reliable when they are developed.

The field of AI has a reputation for overselling its progress. In the “AI winters” of the late 1970s and 1980s, researchers’ failures to make good on ambitious promises led to a collapse of funding and interest in AI. Although the field is now undergoing a renaissance of sorts, discussion of the possibility of human-level machine intelligence continues to be restricted largely to the science fiction shelf, for fear of recapitulating our past mistakes.

Researchers nonetheless largely agree that AI is likely to begin outperforming humans on most cognitive tasks in this century. Given how disruptive such a development is likely to be, we believe that it is prudent to begin a conversation about domain-general AI now, and to investigate whether there are any limited areas in which we can predict the technology’s effects.

The most common position among MIRI’s researchers is that most of the strategic questions relevant to future advances in AI have yet to be adequately investigated. However, we broadly agree with the reasoning in these two books:

Smarter Than Us

A short, lively introduction to questions surrounding smarter-than-human artificial agents. It is humans’ intelligence, rather than our strength or speed, that has given us a dominant advantage over other species. AI’s largest risks (and largest benefits) stem from its potential to surpass us on that front.


An in-depth survey of possible scenarios in which AI algorithms surpass humans in cognitive capabilities. Bostrom argues that autonomous artificial agents, if programmed with imperfect goals, are likely to converge upon extremely dangerous instrumental strategies.

Stuart Russell, co-author of the leading textbook on artificial intelligence, argues in “The Long-Term Future of Artificial Intelligence” that we should integrate questions of robustness and safety into mainstream capabilities research:

Our goal as a field is to make better decision-making systems. And that is the problem. […If] you’re going to build a superintelligent machine, you have to give it something that you want it to do. The danger is that you give it something that isn’t actually what you really want — because you’re not very good at expressing what you really want, or even knowing what you really want — until it’s too late and you see that you don’t like it.

If you think about it just in terms of an optimization problem: The machine is solving an optimization problem for you, and you leave out some of the variables that you actually care about. Well, it’s in the nature of optimization problems that if the system gets to manipulate some variables that don’t form part of the objective function — so it’s free to play with those as much as it wants — often, in order to optimize the ones that it is supposed to optimize, it will set the other ones to extreme values.

My proposal is that we should stop doing AI in its simple definition of just improving the decision-making capabilities of systems. […] With civil engineering, we don’t call it “building bridges that don’t call down” — we just call it “building bridges.” Of course we don’t want them to fall down. And we should think the same way about AI: of course AI systems should be designed so that their actions are well-aligned with what human beings want. But it’s a difficult unsolved problem that hasn’t been part of the research agenda up to now.

We want to change the field so that it feels like civil engineering or like nuclear fusion. [… We] created a hydrogen bomb explosion — unlimited amounts of energy, more than we could possibly use. But it wasn’t in a socially beneficial form. And now it’s just what fusion researchers do — containment is what fusion research is. That’s the problem that they work on.

In line with Russell’s talk, MIRI’s work is aimed at helping jump-start a paradigm of AI research that is conscious of the field’s long-term societal impact. At present, our focus is on investigating theoretical prerequisites for modeling highly intelligent artificial agents and aligning their decision-making with human interests. Our work is intended to progress from mathematical theory to engineering applications as our understanding of the alignment problem matures.


Resources for Learning More