Last week, we received several questions from the effective altruist community in response to our fundraising post. Here’s Maxwell Fritz:
[…] My snap reaction to MIRI’s pitches has typically been, “yeah, AI is a real concern. But I have no idea whether MIRI are the right people to work on it, or if their approach to the problem is the right one” [… I]f you agree AI matters, why MIRI?
And here are two more questions in a similar vein, added by Tristan Tager:
[… W]hat can MIRI do? Why should I expect that the MIRI vision and the MIRI team are going to get things done? What exactly can I expect them to get done? […]
But the second and much bigger question is, what would MIRI do that Google wouldn’t? Google has a ton of money, a creative and visionary staff, the world’s best programmers, and a swath of successful products that incorporate some degree of AI — and moreover they recently acquired several AI businesses and formed an AI ethics board. It seems like they’re approaching the same big problem directly rather than theoretically, and have deep pockets, keen minds, and a wealth of hands-on experience.
These are great questions. My answer to “Why MIRI?”, in short, is that MIRI has a brilliant team of researchers focused on the fundamental theoretical research that almost nobody else is pursuing. We’re focused entirely on aligning smarter-than-human AI systems with humane values, for the long haul.
Most academics aren’t working on AI alignment problems yet, and none are doing it full-time. Most industry folks aren’t working on these problems yet, either. I know this because I’m in conversations with a number of them. (The field is large, but it isn’t that large.)
There are quite a few good reasons why academics and industry professionals aren’t working on these problems yet, and I’ll touch on a few of them in turn.
1. Most AI scientists focus predominantly on the short and medium term.
This makes sense: the field of AI has been burned more than once in the past by over-promising and under-delivering, and history shows that it’s often easier to do AI work when one keeps an eye on goals that are achievable within the next few years.
Many AI scientists have incentives to stick to practical work, look to the short term, and make only realistic promises. (This is evidenced in part by the field’s recent focus on “machine learning,” which has a narrower and more immediate focus than the “old-fashioned” field of AI.) This has paid off, and industry and academia both have made continual incremental breakthroughs that have led to some amazing new technologies. However, it has also made long-term considerations somewhat toxic, and “artificial general intelligence” has been a taboo topic of sorts in recent years. This sentiment is starting to shift, but most academics are still loath to work on foundational problems pertaining to artificial general intelligence, preferring research that improves the capabilities of practical systems today.
2. We’re solving a different sort of problem.
In MIRI’s Approach I spoke of two different classes of computer science problem. Class 1 problems involve figuring out how to do, in practice and with reasonable amounts of computing power, things which we know how to do in principle. Class 2 problems involve figuring out how to do in principle things that we can’t even do in principle yet.
Our current approach to alignment research is to try to move problems from Class 2 to Class 1. This kind of research has been pursued successfully in other areas in the past, and in the context of AI alignment I believe that it deserves significantly more attention than it is receiving.
Industry is traditionally best suited for the first problem class. Academia, too, also often focuses on the first class of problems instead of the second class — especially in the field of AI, for reasons related to point 1. It is common for academics to take some formalization of something like probability theory and then explore and extend the framework, figuring out where it applies and developing practical approximations of intractable algorithms and so on. It’s much rarer for academics to create theoretical foundations for problems that cannot yet be solved even in principle, and this tends to happen only when someone is searching for new theoretical foundations on purpose. For reasons discussed above, most academics aren’t attempting this sort of research yet when it comes to AI alignment.
This is what MIRI brings to the table: a laser focus on the relevant technical challenges.
3. We don’t have competing interests or priorities.
Academic research tends to follow the contours of tractability and curiosity, wherever they may lead. Industry research tends to follow short- and medium-term profits. It is not obvious that either of these will zero in on the AI alignment problems we raise in our technical agenda.1
By contrast, MIRI researchers focus full-time on the most important AI alignment problems they can identify, without taking breaks to teach classes, pursue other interesting theoretical questions, or tackle more immediately profitable or publishable topics. We aren’t going to switch to a different set of problems in order to win a grant; and when we win a three-year grant, we aren’t going to switch to a different set of problems as soon as the grant expires. We will simply continue working on the most important technical problems we can find.
Our mission is to solve the hard technical problems of AI alignment, and our current approach is to zero in on the open problems laid out in our research agenda, without distraction.
The takeaway is that at least in the near term, solutions to the specific problems we’re looking at — generated by the question “what would we still be unable to solve, even if the alignment problem were simpler?” — are most likely to come from MIRI and our collaborators.
Tristan asked: “What would MIRI do that Google wouldn’t?” The answer is: the fundamental theoretical research. I talk to industry folks fairly regularly about what they’re working on and about what we’re working on. Over and over, the reaction I get to our work is something along the lines of “Ah, yes, those are very important questions. We aren’t working on those, but it does seem like we’re missing some useful tools there. Let us know if you find some answers.”
Or, just as often, the response we get is some version of “Well, yes, that tool would be awesome, but getting it sounds impossible,” or “Wait, why do you think we actually need that tool?”2 Regardless, the conversations I’ve had tend to end the same way for all three groups: “That would be a useful tool if you can develop it; we aren’t working on that; let us know if you find some answers.”
Industry is interested in pushing modern practical systems towards increasingly general applications. They’re focused on problems such as hierarchical planning, continual learning, and transfer learning. When they work on safety research (and they often do!), they work on tools that improve the transparency of deep neural networks, and they work on improving the calibration and decreasing the error rates in their systems. Our relations with industry AI scientists are good, and we keep tabs on each other. But major industry groups simply aren’t competing with MIRI in AI alignment research; they aren’t doing our type of work at the moment.
It’s also worth keeping in mind that MIRI researchers tend to be agnostic about AI timelines. Last year, the question du jour was “why do you think your research can matter, when we’re so far off from smarter-than-human AI that it’s impossible to know what architecture will be used for machine intelligence?” This year, the question du jour is “why do you think your research can matter, when Google is obviously going to develop smarter-than-human AI using artificial neural networks?”
My answer has remained similar in both cases: we’re developing basic conceptual tools, akin to probability theory, that are likely to improve the transparency, verifiability, stability, and overall safety of AGI systems across a variety of different architectures. My timelines are about the same today as they were last year: I don’t expect to see smarter-than-human AI systems in the next decade, and I do expect to see them this century. It is not obvious to me when smarter-than-human AI systems will be developed, or how, or by whom.
A decade is a lot of time in industry, and a century is an eternity. The industry leaders today may look quite strong, and some groups may be developing a strong lead over others. However, I’m not nearly confident enough in any one group to put all our chips on their success.
MIRI is doing foundational research to develop tools that will help all industry groups design safer systems. Our goal is to make it easier to design robust and beneficial general-purpose reasoners from the ground up, and I think it’s valuable for those sorts of tools to be developed by an impartial third party that answers to the public rather than to its shareholders.
Superintelligence probably isn’t just around the corner. In the interim, we’re working on a different set of problems than existing groups in industry and academia; most teams aren’t focused on foundational theoretical research on long-term AI alignment problems.
MIRI is. The open problems we work on are underserved, and we’re well-positioned to make inroads on the most critical technical obstacles lying ahead. Our hope is that numerous brilliant and skilled organizations will take on these technical challenges as well, and that our work on these topics will help the research area grow more quickly. I would deeply appreciate more competition in this space. Until then, if you want there to be at least one group whose mission is tackling the “Class 2” technical research, MIRI is the team to fund.
- Indeed, the problems in the technical agenda were chosen in part because they seem important but don’t appear to be on the default path. ↩
- There isn’t much correlation between which of our open problems gets which of these three answers, across AI scientists. ↩
Did you like this post? You may enjoy our other Analysis posts, including:
