James Miller on Unusual Incentives Facing AGI Companies

 |   |  Conversations

rsz_11james-d-millerJames D. Miller is an associate professor of economics at Smith College. He is the author of Singularity Rising, Game Theory at Work, and a principles of microeconomics textbook along with several academic articles.

He has a PhD in economics from the University of Chicago and a J.D. from Stanford Law School where he was on Law Review. He is a member of cryonics provider Alcor and a research advisor to MIRI. He is currently co-writing a book on better decision making with the Center for Applied Rationality and will be probably be an editor on the next edition of the Singularity Hypotheses book. He is a committed bio-hacker currently practicing or consuming a paleo diet, neurofeedback, cold thermogenesis, intermittent fasting, brain fitness video games, smart drugs, bulletproof coffee, and rationality training.

 

Luke Muehlhauser: Your book chapter in Singularity Hypothesis describes some unusual economic incentives facing a future business that is working to create AGI. To explain your point, you make the simplifying assumption that “a firm’s attempt to build an AGI will result in one of three possible outcomes”:

  • Unsuccessful: The firm fails to create AGI, losing value for its owners and investors.
  • Riches: The firm creates AGI, bringing enormous wealth to its owners and investors.
  • Foom: The firm creates AGI but this event quickly destroys the value of money, e.g. via an intelligence explosion that eliminates scarcity, or creates a weird world without money, or exterminates humanity.

How does this setup allow us to see the unusual incentives facing a future business that is working to create AGI?

James Miller: A huge asteroid might hit the earth, and if it does it will destroy mankind. You should be willing to bet everything you have that the asteroid will miss our planet because either you win your bet or Armageddon renders the wager irrelevant. Similarly, if I’m going to start a company that will either make investors extremely rich or create a Foom that destroys the value of money, you should be willing to invest a lot in my company’s success because either the investment will pay off, or you would have done no better making any other kind of investment.

Pretend I want to create a controllable AGI, and if successful I will earn great Riches for my investors. At first I intend to follow a research and development path in which if I fail to achieve Riches, my company will be Unsuccessful and have no significant impact on the world. Unfortunately, I can’t convince potential investors that the probability of my achieving Riches is high enough to make my company worth investing in. The investors assign too large a likelihood that other potential investments would outperform my firm’s stock. But then I develop an evil alternative research and development plan under which I have the exact same probability of achieving Riches as before but now if I fail to create a controllable AGI, an unfriendly Foom will destroy humanity. Now I can truthfully tell potential investors that it’s highly unlikely any other company’s stock will outperform mine.

Read more »

MIRI’s July Newsletter: Fundraiser and New Papers

 |   |  Newsletters

Greetings from the Executive Director

Dear friends,

Another busy month! Since our last newsletter, we’ve published 3 new papers and 2 new “analysis” blog posts, we’ve significantly improved our website (especially the Research page), we’ve relocated to downtown Berkeley, and we’ve launched our summer 2013 matching fundraiser!

MIRI also recently presented at the Effective Altruism Summit, a gathering of 60+ effective altruists in Oakland, CA. As philosopher Peter Singer explained in his TED talk, effective altruism “combines both the heart and the head.” The heart motivates us to be empathic and altruistic toward others, while the head can “make sure that what [we] do is effective and well-directed,” so that altruists can do not just some good but as much good as possible.

As I explain in Friendly AI Research as Effective Altruism, MIRI was founded in 2000 on the premise that creating Friendly AI might be a particularly efficient way to do as much good as possible. Effective altruists focus on a variety of other causes, too, such as poverty reduction. As I say in Four Focus Areas of Effective Altruism, I think it’s important for effective altruists to cooperate and collaborate, despite their differences of opinion about which focus areas are optimal. The world needs more effective altruists, of all kinds.

MIRI engages in direct efforts — e.g. Friendly AI research — to improve the odds that machine superintelligence has a positive rather than a negative impact. But indirect efforts — such as spreading rationality and effective altruism — are also likely to play a role, for they will influence the context in which powerful AIs are built. That’s part of why we created CFAR.

If you think this work is important, I hope you’ll donate now to support our work. MIRI is entirely supported by private funders like you. And if you donate before August 15th, your contribution will be matched by one of the generous backers of our current fundraising drive.

Thank you,

Luke Muehlhauser

Executive Director

Read more »

2013 Summer Matching Challenge!

 |   |  News

Thanks to the generosity of several major donors,† every donation to the Machine Intelligence Research Institute made from now until August 15th, 2013 will be matched dollar-for-dollar, up to a total of $200,000!

 

Donate Now!

$0

$50,000

$100,000

$150,000

$200,000

We have reached our goal of $200,000!

Now is your chance to double your impact while helping us raise up to $400,000 (with matching) to fund our research program.

Early this year we made a transition from movement-building to research, and we’ve hit the ground running with six major new research papers, six new strategic analyses on our blog, and much more. Give now to support our ongoing work on the future’s most important problem.

 

Accomplishments in 2013 so far

Future Plans You Can Help Support

  • We will host many more research workshops, including one in September, and one in December (with John Baez attending, among others).
  • Eliezer will continue to publish about open problems in Friendly AI. (Here is #1 and #2.)
  • We will continue to publish strategic analyses, mostly via our blog.
  • We will publish nicely-edited ebooks (Kindle, iBooks, and PDF) for more of our materials, to make them more accessible: The Sequences, 2006-2009 and The Hanson-Yudkowsky AI Foom Debate.
  • We will continue to set up the infrastructure (e.g. new offices, researcher endowments) required to host a productive Friendly AI research team, and (over several years) recruit enough top-level math talent to launch it.

(Other projects are still being surveyed for likely cost and strategic impact.)

We appreciate your support for our high-impact work! Donate now, and seize a better than usual chance to move our work forward. If you have questions about donating, please contact Louie Helm at (510) 717-1477 or louie@intelligence.org.

† $200,000 of total matching funds has been provided by Jaan Tallinn, Loren Merritt, Rick Schwall, and Alexei Andreev.

MIRI Has Moved!

 |   |  News

For the past several months, MIRI and its child organization CFAR have been working from a much-too-small office on the outskirts of Berkeley. At the end of June, MIRI and CFAR took over the 3rd floor of 2030 Addison St. in downtown Berkeley, which has sufficient space for both organizations.

Our new office is 0.5 blocks from the Downtown Berkeley BART exit at Shattuck & Addison, and 2 blocks from the UC Berkeley campus. Here’s a photo of the campus from our roof:

view of campus from roof (500px)

The proximity to UC Berkeley will make it easier for MIRI to network with Berkeley’s professors and students. Conveniently, UC Berkeley is ranked 5th in the world in mathematics, and 1st in the world in mathematical logic.

Sharing an office with CFAR carries many benefits for both organizations:

  1. CFAR and MIRI can “flex” into each other’s space for short periods as needed, for example when MIRI is holding a week-long research workshop.
  2. We can share resources (printers, etc.).
  3. Both organizations can benefit from interaction between our two communities.

Getting the new office was a team effort, but the person most responsible for this success was MIRI Deputy Director Louie Helm.

Note that MIRI isn’t yet able to accommodate “drop in” visitors, as we keep irregular hours throughout the week. So if you’d like to visit, please contact us first.

We retain 2721 Shattuck Ave. #1023 as an alternate mailing address.

MIRI’s September 2013 Workshop

 |   |  News

Paul at April workshop

From September 7-13, MIRI will host its 4th Workshop on Logic, Probability, and Reflection. The focus of this workshop will be the foundations of decision theory.

Participants confirmed so far include:

If you have a strong mathematics background and might like to attend this workshop, it’s not too late to apply! And even if this workshop doesn’t fit your schedule, please do apply, so that we can notify you of other workshops (long before they are announced publicly).

Responses to Catastrophic AGI Risk: A Survey

 |   |  Papers

MIRI is self-publishing another technical report that was too lengthy (60 pages) for publication in a journal: Responses to Catastrophic AGI Risk: A Survey.

The report, co-authored by past MIRI researcher Kaj Sotala and University of Louisville’s Roman Yampolskiy, is a summary of the extant literature (250+ references) on AGI risk, and can serve either as a guide for researchers or as an introduction for the uninitiated.

Here is the abstract:

Many researchers have argued that humanity will create artificial general intelligence (AGI) within the next twenty to one hundred years. It has been suggested that AGI may pose a catastrophic risk to humanity. After summarizing the arguments for why AGI may pose such a risk, we survey the field’s proposed responses to AGI risk. We consider societal proposals, proposals for external constraints on AGI behaviors, and proposals for creating AGIs that are safe due to their internal design.

The preferred discussion page for the paper is here.

Update: This report has now been published in Physica Scripta, available here.

What is Intelligence?

 |   |  Analysis

When asked their opinions about “human-level artificial intelligence” — aka “artificial general intelligence” (AGI)1 — many experts understandably reply that these terms haven’t yet been precisely defined, and it’s hard to talk about something that hasn’t been defined.2 In this post, I want to briefly outline an imprecise but useful “working definition” for intelligence we tend to use at MIRI. In a future post I will write about some useful working definitions for artificial general intelligence.

 

Imprecise definitions can be useful

Precise definitions are important, but I concur with Bertrand Russell that

[You cannot] start with anything precise. You have to achieve such precision… as you go along.

Physicist Milan Ćirković agrees, and gives an example:

The formalization of knowledge — which includes giving precise definitions — usually comes at the end of the original research in a given field, not at the very beginning. A particularly illuminating example is the concept of number, which was properly defined in the modern sense only after the development of axiomatic set theory in the… twentieth century.3

For a more AI-relevant example, consider the concept of a “self-driving car,” which has been given a variety of vague definitions since the 1930s. Would a car guided by a buried cable qualify? What about a modified 1955 Studebaker that could use sound waves to detect obstacles and automatically engage the brakes if necessary, but could only steer “on its own” if each turn was preprogrammed? Does that count as a “self-driving car”?

What about the “VaMoRs” of the 1980s that could avoid obstacles and steer around turns using computer vision, but weren’t advanced enough to be ready for public roads? How about the 1995 Navlab car that drove across the USA and was fully autonomous for 98.2% of the trip, or the robotic cars which finished the 132-mile off-road course of the 2005 DARPA Grand Challenge, supplied only with the GPS coordinates of the route? What about the winning cars of the 2007 DARPA Grand Challenge, which finished an urban race while obeying all traffic laws and avoiding collisions with other cars? Does Google’s driverless car qualify, given that it has logged more than 500,000 autonomous miles without a single accident under computer control, but still struggles with difficult merges and snow-covered roads?4

Our lack of a precise definition for “self-driving car” doesn’t seem to have hindered progress on self-driving cars very much.5 And I’m glad we didn’t wait to seriously discuss self-driving cars until we had a precise definition for the term.

Similarly, I don’t think we should wait for a precise definition of AGI before discussing the topic seriously. On the other hand, the term is useless if it carries no information. So let’s work our way toward a stipulative, operational definition for AGI. We’ll start by developing an operational definition for intelligence.

Read more »

  1. I use the HLAI and AGI interchangeably, but lately I’ve been using AGI almost exclusively, because I’ve learned that many people in the AI community react negatively to any mention of “human-level” AI but have no objection to the concept of narrow vs. general intelligence. See also Ben Goertzel’s comments here. ↩
  2. Asked when he thought HLAI would be created, Pat Hayes (a past president of AAAI) replied: “I do not consider this question to be answerable, as I do not accept this (common) notion of ‘human-level intelligence’ as meaningful.” Asked the same question, AI scientist William Uther replied: “You ask a lot about ‘human level AGI’. I do not think this term is well defined,” while AI scientist Alan Bundy replied: “I don’t think the concept of ‘human-level machine intelligence’ is well formed.” ↩
  3. Sawyer (1943) gives another example: “Mathematicians first used the sign √-1, without in the least knowing what it could mean, because it shortened work and led to correct results. People naturally tried to find out why this happened and what √-1 really meant. After two hundreds years they succeeded.” Dennett (2013) makes a related comment: “Define your terms, sir! No, I won’t. That would be premature… My [approach] is an instance of nibbling on a tough problem instead of trying to eat (and digest) the whole thing from the outset… In Elbow Room, I compared my method to the sculptor’s method of roughing out the form in a block of marble, approaching the final surfaces cautiously, modestly, working by successive approximation.” ↩
  4. With self-driving cars, researchers did use many precise external performance measures (e.g. accident rates, speed, portion of the time they could run unassisted, frequency of getting stuck) to evaluate progress, as well as internal performance metrics (speed of search, bounded loss guarantees, etc.). Researchers could see that these bits of progress were in the right direction, even if their relative contribution long-term was unclear. And so it is with AI in general. AI researchers use many precise external and internal performance measures to evaluate progress, but it is difficult to know the relative contribution of these bits of progress toward the final goal of AGI. ↩
  5. Heck, we’ve had pornography for millennia and still haven’t been able to define it precisely. Encyclopedia entries for “pornography” often simply quote Justice Potter Stewart: “I shall not today attempt further to define the kinds of material I understand to be [pornography]… but I know it when I see it.” ↩

MIRI’s July 2013 Workshop

 |   |  News

Mihaly at April workshop

From July 8-14, MIRI will host its 3rd Workshop on Logic, Probability, and Reflection. The focus of this workshop will be the Löbian obstacle to self-modifying systems.

Participants confirmed so far include:

If you have a strong mathematics background and might like to attend this workshop, it’s not too late to apply! And even if this workshop doesn’t fit your schedule, please do apply, so that we can notify you of other workshops (long before they are announced publicly).

Information on past workshops: