What Sets MIRI Apart?

 |   |  Analysis

Last week, we received several questions from the effective altruist community in response to our fundraising post. Here’s Maxwell Fritz:

[…] My snap reaction to MIRI’s pitches has typically been, “yeah, AI is a real concern. But I have no idea whether MIRI are the right people to work on it, or if their approach to the problem is the right one” [… I]f you agree AI matters, why MIRI?

And here are two more questions in a similar vein, added by Tristan Tager:

[… W]hat can MIRI do? Why should I expect that the MIRI vision and the MIRI team are going to get things done? What exactly can I expect them to get done? […]

But the second and much bigger question is, what would MIRI do that Google wouldn’t? Google has a ton of money, a creative and visionary staff, the world’s best programmers, and a swath of successful products that incorporate some degree of AI — and moreover they recently acquired several AI businesses and formed an AI ethics board. It seems like they’re approaching the same big problem directly rather than theoretically, and have deep pockets, keen minds, and a wealth of hands-on experience.

These are great questions. My answer to “Why MIRI?”, in short, is that MIRI has a brilliant team of researchers focused on the fundamental theoretical research that almost nobody else is pursuing. We’re focused entirely on aligning smarter-than-human AI systems with humane values, for the long haul.

Most academics aren’t working on AI alignment problems yet, and none are doing it full-time. Most industry folks aren’t working on these problems yet, either. I know this because I’m in conversations with a number of them. (The field is large, but it isn’t that large.)

There are quite a few good reasons why academics and industry professionals aren’t working on these problems yet, and I’ll touch on a few of them in turn.


1. Most AI scientists focus predominantly on the short and medium term.

This makes sense: the field of AI has been burned more than once in the past by over-promising and under-delivering, and history shows that it’s often easier to do AI work when one keeps an eye on goals that are achievable within the next few years.

Many AI scientists have incentives to stick to practical work, look to the short term, and make only realistic promises. (This is evidenced in part by the field’s recent focus on “machine learning,” which has a narrower and more immediate focus than the “old-fashioned” field of AI.) This has paid off, and industry and academia both have made continual incremental breakthroughs that have led to some amazing new technologies. However, it has also made long-term considerations somewhat toxic, and “artificial general intelligence” has been a taboo topic of sorts in recent years. This sentiment is starting to shift, but most academics are still loath to work on foundational problems pertaining to artificial general intelligence, preferring research that improves the capabilities of practical systems today.


2. We’re solving a different sort of problem.

In MIRI’s Approach I spoke of two different classes of computer science problem. Class 1 problems involve figuring out how to do, in practice and with reasonable amounts of computing power, things which we know how to do in principle. Class 2 problems involve figuring out how to do in principle things that we can’t even do in principle yet.

Our current approach to alignment research is to try to move problems from Class 2 to Class 1. This kind of research has been pursued successfully in other areas in the past, and in the context of AI alignment I believe that it deserves significantly more attention than it is receiving.

Industry is traditionally best suited for the first problem class. Academia, too, also often focuses on the first class of problems instead of the second class — especially in the field of AI, for reasons related to point 1. It is common for academics to take some formalization of something like probability theory and then explore and extend the framework, figuring out where it applies and developing practical approximations of intractable algorithms and so on. It’s much rarer for academics to create theoretical foundations for problems that cannot yet be solved even in principle, and this tends to happen only when someone is searching for new theoretical foundations on purpose. For reasons discussed above, most academics aren’t attempting this sort of research yet when it comes to AI alignment.

This is what MIRI brings to the table: a laser focus on the relevant technical challenges.


3. We don’t have competing interests or priorities.

Academic research tends to follow the contours of tractability and curiosity, wherever they may lead. Industry research tends to follow short- and medium-term profits. It is not obvious that either of these will zero in on the AI alignment problems we raise in our technical agenda.1

By contrast, MIRI researchers focus full-time on the most important AI alignment problems they can identify, without taking breaks to teach classes, pursue other interesting theoretical questions, or tackle more immediately profitable or publishable topics. We aren’t going to switch to a different set of problems in order to win a grant; and when we win a three-year grant, we aren’t going to switch to a different set of problems as soon as the grant expires. We will simply continue working on the most important technical problems we can find.

Our mission is to solve the hard technical problems of AI alignment, and our current approach is to zero in on the open problems laid out in our research agenda, without distraction.


The takeaway is that at least in the near term, solutions to the specific problems we’re looking at — generated by the question “what would we still be unable to solve, even if the alignment problem were simpler?” — are most likely to come from MIRI and our collaborators.

Tristan asked: “What would MIRI do that Google wouldn’t?” The answer is: the fundamental theoretical research. I talk to industry folks fairly regularly about what they’re working on and about what we’re working on. Over and over, the reaction I get to our work is something along the lines of “Ah, yes, those are very important questions. We aren’t working on those, but it does seem like we’re missing some useful tools there. Let us know if you find some answers.”

Or, just as often, the response we get is some version of “Well, yes, that tool would be awesome, but getting it sounds impossible,” or “Wait, why do you think we actually need that tool?”2 Regardless, the conversations I’ve had tend to end the same way for all three groups: “That would be a useful tool if you can develop it; we aren’t working on that; let us know if you find some answers.”

Industry is interested in pushing modern practical systems towards increasingly general applications. They’re focused on problems such as hierarchical planning, continual learning, and transfer learning. When they work on safety research (and they often do!), they work on tools that improve the transparency of deep neural networks, and they work on improving the calibration and decreasing the error rates in their systems. Our relations with industry AI scientists are good, and we keep tabs on each other. But major industry groups simply aren’t competing with MIRI in AI alignment research; they aren’t doing our type of work at the moment.

It’s also worth keeping in mind that MIRI researchers tend to be agnostic about AI timelines. Last year, the question du jour was “why do you think your research can matter, when we’re so far off from smarter-than-human AI that it’s impossible to know what architecture will be used for machine intelligence?” This year, the question du jour is “why do you think your research can matter, when Google is obviously going to develop smarter-than-human AI using artificial neural networks?”

My answer has remained similar in both cases: we’re developing basic conceptual tools, akin to probability theory, that are likely to improve the transparency, verifiability, stability, and overall safety of AGI systems across a variety of different architectures. My timelines are about the same today as they were last year: I don’t expect to see smarter-than-human AI systems in the next decade, and I do expect to see them this century. It is not obvious to me when smarter-than-human AI systems will be developed, or how, or by whom.

A decade is a lot of time in industry, and a century is an eternity. The industry leaders today may look quite strong, and some groups may be developing a strong lead over others. However, I’m not nearly confident enough in any one group to put all our chips on their success.

MIRI is doing foundational research to develop tools that will help all industry groups design safer systems. Our goal is to make it easier to design robust and beneficial general-purpose reasoners from the ground up, and I think it’s valuable for those sorts of tools to be developed by an impartial third party that answers to the public rather than to its shareholders.

Superintelligence probably isn’t just around the corner. In the interim, we’re working on a different set of problems than existing groups in industry and academia; most teams aren’t focused on foundational theoretical research on long-term AI alignment problems.

MIRI is. The open problems we work on are underserved, and we’re well-positioned to make inroads on the most critical technical obstacles lying ahead. Our hope is that numerous brilliant and skilled organizations will take on these technical challenges as well, and that our work on these topics will help the research area grow more quickly. I would deeply appreciate more competition in this space. Until then, if you want there to be at least one group whose mission is tackling the “Class 2” technical research, MIRI is the team to fund.


  1. Indeed, the problems in the technical agenda were chosen in part because they seem important but don’t appear to be on the default path. 
  2. There isn’t much correlation between which of our open problems gets which of these three answers, across AI scientists. 
  • http://www.shealevy.com Shea Colton Levy

    I’ve been grappling with the “Assuming AI alignment is important, why MIRI specifically?” question myself lately, but in conjunction with the recent “Why Now Matters” post this is just leaving me more confused on that count. It seemed like part of the argument in that post was that fairly soon we should expect more players in the game, and thus we should support MIRI now so it would maintain influence, and so when I saw that this post was forthcoming I was expecting elaboration on why we should want MIRI to maintain influence. Now, though, it seems like the core of the argument is that MIRI is the only one even addressing this issue. Have I misunderstood something here?

    • http://www.nothingismere.com/ Rob Bensinger

      I’d say there are three parts to the argument. Starting with the first two:

      1. There’s currently a relatively small number of people working of smarter-than-human AI safety, but this number is growing quickly, and we expect it to grow a lot in the coming years.

      2. There are currently no large-scale, long-term technical research programs focused on smarter-than-human AI safety, other than MIRI.

      There are small-scale, long-term programs of that kind (e.g., Stuart Armstrong’s work at the Future of Humanity Institute), and we expect more programs like that to get started in the next couple of years. There are also short-term programs focused on risk from smarter-than-human AI, like Paul Christiano’s project grant on http://futureoflife.org/AI/2015awardees, and we’re confident that the number of projects like this will keep growing. And there are large-scale, long-term research programs that focus on nontechnical issues in this area, like the new Cambridge/Oxford Strategic Research Center for Artificial Intelligence and Stanford’s new AI100 study (https://ai100.stanford.edu/reflections-and-framing). But we don’t know of any rumblings that a large-scale, long-term technical research program specializing in smarter-than-human AI safety is likely to spring into existence in the next couple of years. It’s certainly a possibility — and a welcome one — but we aren’t relying on its happening, and we expect it to happen sooner if MIRI expands more quickly.

      We think individuals like Paul Christiano, Stuart Russell, and Stuart Armstrong should be funded to do independent alignment research. But we think it would be especially useful to have a larger group working on these problems in a sustained way.

      I’d also add a third piece:

      3. MIRI’s research methodology and technical agenda is unusually promising, and is more likely to be neglected by the larger AI field than many other technical approaches to this issue.

      So the point of MIRI getting a foot in the door isn’t purely to ensure that there are more individuals (and, in the longer run, more groups) doing technical work on smarter-than-human AI safety; it’s also to ensure that more of the people who do technical work on this problem focus on the most promising open problems, which we expect to either be problems related to MIRI’s technical agenda or problems that were selected by similar methods.

      In those terms, I’d say Nate’s post above is making two basic points. First: the field is growing, but academia and industry haven’t yet started a large-scale long-term program specializing in technical research in this area, and there are no clear signs that one will arise in the immediate future (especially if MIRI isn’t putting out eye-catching results on this topic). Second: to the extent academic and industry AI researchers increasingly direct efforts toward technical smarter-than-human AI safety research, competing priorities and distractions may cause them to focus on lower-priority technical problems (again, especially if MIRI isn’t putting out a lot of results that popularize its research agenda and methodology).

      We also expect groups in industry and academia to play a much more central role once it’s clearer what the basic formal tools we should be using in alignment work are — which is why MIRI is trying to identify and develop those tools.

      • http://www.shealevy.com Shea Colton Levy

        I see, thanks! This makes more sense to me now. Any pointers for verifying your point 3, particularly the first half?