James D. Miller is an associate professor of economics at Smith College. He is the author of Singularity Rising, Game Theory at Work, and a principles of microeconomics textbook along with several academic articles.
He has a PhD in economics from the University of Chicago and a J.D. from Stanford Law School where he was on Law Review. He is a member of cryonics provider Alcor and a research advisor to MIRI. He is currently co-writing a book on better decision making with the Center for Applied Rationality and will be probably be an editor on the next edition of the Singularity Hypotheses book. He is a committed bio-hacker currently practicing or consuming a paleo diet, neurofeedback, cold thermogenesis, intermittent fasting, brain fitness video games, smart drugs, bulletproof coffee, and rationality training.
Luke Muehlhauser: Your book chapter in Singularity Hypothesis describes some unusual economic incentives facing a future business that is working to create AGI. To explain your point, you make the simplifying assumption that “a firm’s attempt to build an AGI will result in one of three possible outcomes”:
- Unsuccessful: The firm fails to create AGI, losing value for its owners and investors.
- Riches: The firm creates AGI, bringing enormous wealth to its owners and investors.
- Foom: The firm creates AGI but this event quickly destroys the value of money, e.g. via an intelligence explosion that eliminates scarcity, or creates a weird world without money, or exterminates humanity.
How does this setup allow us to see the unusual incentives facing a future business that is working to create AGI?
James Miller: A huge asteroid might hit the earth, and if it does it will destroy mankind. You should be willing to bet everything you have that the asteroid will miss our planet because either you win your bet or Armageddon renders the wager irrelevant. Similarly, if I’m going to start a company that will either make investors extremely rich or create a Foom that destroys the value of money, you should be willing to invest a lot in my company’s success because either the investment will pay off, or you would have done no better making any other kind of investment.
Pretend I want to create a controllable AGI, and if successful I will earn great Riches for my investors. At first I intend to follow a research and development path in which if I fail to achieve Riches, my company will be Unsuccessful and have no significant impact on the world. Unfortunately, I can’t convince potential investors that the probability of my achieving Riches is high enough to make my company worth investing in. The investors assign too large a likelihood that other potential investments would outperform my firm’s stock. But then I develop an evil alternative research and development plan under which I have the exact same probability of achieving Riches as before but now if I fail to create a controllable AGI, an unfriendly Foom will destroy humanity. Now I can truthfully tell potential investors that it’s highly unlikely any other company’s stock will outperform mine.
Luke: In the case of the asteroid, the investor (me) has no ability to affect whether asteroid hits us or not. In contrast, assuming investors want to avoid risking the destruction of money, there is something they can do about it: they can simply not invest in the project to create controllable AGI, and they can encourage others not to invest in it either. If the dangerous AGI project fails to get investment, then those potential investors need not worry that the value of money — or indeed their very lives — will be destroyed by that particular AGI project.
Why would investors fund something that has a decent chance of destroying everything?
James: Consider a mine that probably contains the extremely valuable metal mithril. Alas, digging in the mine might awaken a Balrog who would then destroy our civilization. I come to you asking for money to fund my mining expedition. When you raise concerns about the Balrog, I explain that even if you don’t fund me I’m probably going to get the money elsewhere, perhaps from one of the many people who don’t believe in Balrogs. Plus, since you are just one of many individuals I’m hoping to get to back me, even if I never make up for losing your investment, I will almost certainly still get enough funds to dig for mithril. And even if you manage to stop my mining operation, someone else is going to eventually work the mine. At the very least, mithril’s military value will eventually cause some nations to seek it. Therefore, if the mine does contain an easily-awakened Balrog we’re all dead regardless of what you do. Consequently, since you have no choice but to bear the downside risk of the mine, I explain that you might as well invest some of your money with me so you can also share in the upside benefit.
If you’re a self-interested investor then when deciding on whether to invest in an AGI-seeking company you don’t look at (a) the probability that the company will destroy mankind, but rather consider (b) the probability of mankind being destroyed if you do invest in the company minus the probability if you don’t invest in it. For most investors (b) will be zero or at least really, really small because if you don’t invest in the company someone else probably will take your place, the company probably doesn’t need your money to become operational, and if this one company doesn’t pursue AGI another almost certainly will.
Luke: Thanks for that clarification. Now I’d like to give you an opportunity to reply to one of your critics. In response to your book chapter, Robin Hanson wrote: “[Miller’s analysis] is only as useful as the assumptions on which it is based. Miller’s chosen assumptions seem to me quite extreme, and quite unlikely.”
I asked Hanson if he could clarify (and be quoted), and he told me:
In [this] context the assumptions were: “a single public firm developing the entire system in one go. . . . it succeeds so quickly that there is no chance for others to react – the world is remade overnight.” I would instead assume a whole AI industry, where each firm only makes a small part of the total product and competes with several other firms to make that small part. Each new innovation by each firm only adds a small fraction to the capability of that small part.
Hanson also wrote some comments about your book Singularity Rising here.
How would you reply to Hanson, regarding the economic incentives facing a business working on AGI?
James: Hanson places a very low probability on the likelihood of an intelligence explosion. And my chapter’s analysis does become worthless if you sufficiently discount the possibility of such a Foom.
But in large part because of Eliezer Yudkowsky, I do think that there is a significant chance of their being an intelligence explosion in which an AGI within a matter of days goes from human-level intelligence to something orders of magnitude smarter. I described in Singularity Rising how this might happen:
Somehow, a human-level AI is created that wishes to improve its cognitive powers. The AI examines its own source code, looking for ways to augment itself. After finding a potential improvement, the original AI makes a copy of itself with the enhancement added. The original AI then tests the new AI to see if the new AI is smarter than the original one. If the new AI is indeed better, it replaces the current AI. This new AI then proceeds in turn to enhance its code. Since the new AI is smarter than the old one, it finds improvements that the old AI couldn’t. This process continues to repeat itself.
Intelligence is a reflective superpower, able to turn on itself to decipher its own workings. I think that there probably exists some intelligence threshold for an AGI, whose intelligence is based on easily alterable and testable computer code, at which it could Foom.
Hanson, however, is correct that an intelligence explosion would represent a sharp break with all past technological developments and so an outside view of technology supports his position that my chosen assumptions are “quite extreme, and quite unlikely.”
Luke: I think Yudkowsky would reply that there are many outside views, and some of them suggest a rapid intelligence explosion while others do not. For example in Intelligence Explosion Microeconomics he argues that an outside view with respect to brain size and hominid evolution suggests that an AGI would lead to intelligence explosion. My own preferred solution to “reference class tennis” is to take each outside view as a model for how the phenomena works, acknowledge our model uncertainty, and then use something like model combination to figure out what our posterior probability distribution should be.
Okay, last question. What are you working on now?
James: I’m co-authoring a book on better decision making with the Center for Applied Rationality. Although the book is still in the early stages, it might be based on short, original fictional stories. I’m also probably going to be an editor on the next edition of the Singularity Hypotheses book.
Luke: Thanks, James!
Did you like this post? You may enjoy our other Conversations posts, including:
