About MIRI

The Machine Intelligence Research Institute is a research nonprofit focused on the mathematical underpinnings of intelligent behavior. Our mission is to develop formal tools for the clean design and analysis of general-purpose artificially intelligent systems, with the intent of making such systems safe and reliable when they are developed.

The field of AI has a reputation for overselling its progress. In the “AI winters” of the late 1970s and 1980s, researchers’ failures to make good on ambitious promises led to a collapse of funding and interest in AI. Although the field is now undergoing a renaissance, discussion of the possibility of human-level machine intelligence continues to be restricted largely to the science fiction shelf, for fear of recapitulating our past overconfidence.

At the same time, researchers largely agree that AI is likely to begin outperforming humans on most cognitive tasks in this century. Given how disruptive domain-general AI technology would likely be, we believe that it is prudent to begin a conversation about this now, and to investigate whether there are any limited areas in which we can predict the technology’s effects.

The most common position among MIRI’s researchers is that the strategic questions relevant to future advances in AI have yet to be adequately investigated. However, we broadly agree with the reasoning in these two books:

Smarter Than Us

A short, lively introduction to questions surrounding smarter-than-human artificial agents. Humans’ intelligence (rather than, e.g., our strength or speed) is what gives us a dominant advantage over other species. AI’s largest risks (and largest benefits) stem from its potential to surpass us on that front.


A survey of possible scenarios in which AI algorithms surpass humans in cognitive capabilities. Bostrom argues that autonomous artificial agents, if programmed with imperfect goals, are likely to converge upon extremely dangerous instrumental strategies.

Stuart Russell, MIRI advisor and co-author of the leading textbook on artificial intelligence, argues in “The Long-Term Future of Artificial Intelligence” that we should integrate questions of robustness and safety into mainstream capabilities research:

Our goal as a field is to make better decision-making systems. And that is the problem. […If] you’re going to build a superintelligent machine, you have to give it something that you want it to do. The danger is that you give it something that isn’t actually what you really want — because you’re not very good at expressing what you really want, or even knowing what you really want — until it’s too late and you see that you don’t like it.

If you think about it just in terms of an optimization problem: The machine is solving an optimization problem for you, and you leave out some of the variables that you actually care about. Well, it’s in the nature of optimization problems that if the system gets to manipulate some variables that don’t form part of the objective function — so it’s free to play with those as much as it wants — often, in order to optimize the ones that it is supposed to optimize, it will set the other ones to extreme values.

My proposal is that we should stop doing AI in its simple definition of just improving the decision-making capabilities of systems. […] With civil engineering, we don’t call it “building bridges that don’t fall down” — we just call it “building bridges.” Of course we don’t want them to fall down. And we should think the same way about AI: of course AI systems should be designed so that their actions are well-aligned with what human beings want. But it’s a difficult unsolved problem that hasn’t been part of the research agenda up to now.

We want to change the field so that it feels like civil engineering or like nuclear fusion. [… We] created a hydrogen bomb explosion — unlimited amounts of energy, more than we could possibly use. But it wasn’t in a socially beneficial form. And now it’s just what fusion researchers do — containment is what fusion research is. That’s the problem that they work on.

In line with Russell’s talk, MIRI’s work is aimed at helping jump-start a paradigm of AI research that is conscious of the field’s long-term societal impact. At present, our focus is on investigating theoretical prerequisites for modeling highly intelligent artificial agents and aligning their decision-making with human interests. Our work is intended to progress from mathematical theory to engineering applications as our understanding of the alignment problem matures.


Resources for Learning More