Nick Beckstead on the Importance of the Far Future

 |   |  Conversations

Nick Beckstead recently finished a Ph.D in philosophy at Rutgers University, where he focused on practical and theoretical ethical issues involving future generations. He is particularly interested in the practical implications of taking full account of how actions taken today affect people who may live in the very distant future. His research focuses on how big picture questions in normative philosophy (especially population ethics and decision theory) and various big picture empirical questions (especially about existential risk, moral and economic progress, and the future of technology) feed into this issue.

Apart from his academic work, Nick has been closely involved with the effective altruism movement. He has been the director of research for Giving What We Can, he has worked as a summer research analyst at GiveWell, and he is currently on the board of trustees for the Centre for Effective Altruism, and he recently became a research fellow at the Future of Humanity Institute.

Read more »

Roman Yampolskiy on AI Safety Engineering

 |   |  Conversations

Roman V. Yampolskiy holds a PhD degree from the Department of Computer Science and Engineering at the University at Buffalo. There he was a recipient of a four year NSF IGERT fellowship. Before beginning his doctoral studies, Dr. Yampolskiy received a BS/MS (High Honors) combined degree in Computer Science from Rochester Institute of Technology, NY, USA.

After completing his PhD, Dr. Yampolskiy held a position of an Affiliate Academic at the Center for Advanced Spatial Analysis, University of London, College of London. In 2008 Dr. Yampolskiy accepted an assistant professor position at the Speed School of Engineering, University of Louisville, KY. He had previously conducted research at the Laboratory for Applied Computing (currently known as Center for Advancing the Study of Infrastructure) at the Rochester Institute of Technology and at the Center for Unified Biometrics and Sensors at the University at Buffalo. Dr. Yampolskiy is also an alumnus of Singularity University (GSP2012) and a past visiting fellow of MIRI.

Dr. Yampolskiy’s main areas of interest are behavioral biometrics, digital forensics, pattern recognition, genetic algorithms, neural networks, artificial intelligence and games. Dr. Yampolskiy is an author of over 100 publications including multiple journal articles and books. His research has been cited by numerous scientists and profiled in popular magazines both American and foreign (New Scientist, Poker Magazine, Science World Magazine), dozens of websites (BBC, MSNBC, Yahoo! News) and on radio (German National Radio, Alex Jones Show). Reports about his work have attracted international attention and have been translated into many languages including Czech, Danish, Dutch, French, German, Hungarian, Italian, Polish, Romanian, and Spanish

Read more »

James Miller on Unusual Incentives Facing AGI Companies

 |   |  Conversations

rsz_11james-d-millerJames D. Miller is an associate professor of economics at Smith College. He is the author of Singularity Rising, Game Theory at Work, and a principles of microeconomics textbook along with several academic articles.

He has a PhD in economics from the University of Chicago and a J.D. from Stanford Law School where he was on Law Review. He is a member of cryonics provider Alcor and a research advisor to MIRI. He is currently co-writing a book on better decision making with the Center for Applied Rationality and will be probably be an editor on the next edition of the Singularity Hypotheses book. He is a committed bio-hacker currently practicing or consuming a paleo diet, neurofeedback, cold thermogenesis, intermittent fasting, brain fitness video games, smart drugs, bulletproof coffee, and rationality training.

 

Luke Muehlhauser: Your book chapter in Singularity Hypothesis describes some unusual economic incentives facing a future business that is working to create AGI. To explain your point, you make the simplifying assumption that “a firm’s attempt to build an AGI will result in one of three possible outcomes”:

  • Unsuccessful: The firm fails to create AGI, losing value for its owners and investors.
  • Riches: The firm creates AGI, bringing enormous wealth to its owners and investors.
  • Foom: The firm creates AGI but this event quickly destroys the value of money, e.g. via an intelligence explosion that eliminates scarcity, or creates a weird world without money, or exterminates humanity.

How does this setup allow us to see the unusual incentives facing a future business that is working to create AGI?

James Miller: A huge asteroid might hit the earth, and if it does it will destroy mankind. You should be willing to bet everything you have that the asteroid will miss our planet because either you win your bet or Armageddon renders the wager irrelevant. Similarly, if I’m going to start a company that will either make investors extremely rich or create a Foom that destroys the value of money, you should be willing to invest a lot in my company’s success because either the investment will pay off, or you would have done no better making any other kind of investment.

Pretend I want to create a controllable AGI, and if successful I will earn great Riches for my investors. At first I intend to follow a research and development path in which if I fail to achieve Riches, my company will be Unsuccessful and have no significant impact on the world. Unfortunately, I can’t convince potential investors that the probability of my achieving Riches is high enough to make my company worth investing in. The investors assign too large a likelihood that other potential investments would outperform my firm’s stock. But then I develop an evil alternative research and development plan under which I have the exact same probability of achieving Riches as before but now if I fail to create a controllable AGI, an unfriendly Foom will destroy humanity. Now I can truthfully tell potential investors that it’s highly unlikely any other company’s stock will outperform mine.

Read more »

MIRI’s July Newsletter: Fundraiser and New Papers

 |   |  Newsletters

Greetings from the Executive Director

Dear friends,

Another busy month! Since our last newsletter, we’ve published 3 new papers and 2 new “analysis” blog posts, we’ve significantly improved our website (especially the Research page), we’ve relocated to downtown Berkeley, and we’ve launched our summer 2013 matching fundraiser!

MIRI also recently presented at the Effective Altruism Summit, a gathering of 60+ effective altruists in Oakland, CA. As philosopher Peter Singer explained in his TED talk, effective altruism “combines both the heart and the head.” The heart motivates us to be empathic and altruistic toward others, while the head can “make sure that what [we] do is effective and well-directed,” so that altruists can do not just some good but as much good as possible.

As I explain in Friendly AI Research as Effective Altruism, MIRI was founded in 2000 on the premise that creating Friendly AI might be a particularly efficient way to do as much good as possible. Effective altruists focus on a variety of other causes, too, such as poverty reduction. As I say in Four Focus Areas of Effective Altruism, I think it’s important for effective altruists to cooperate and collaborate, despite their differences of opinion about which focus areas are optimal. The world needs more effective altruists, of all kinds.

MIRI engages in direct efforts — e.g. Friendly AI research — to improve the odds that machine superintelligence has a positive rather than a negative impact. But indirect efforts — such as spreading rationality and effective altruism — are also likely to play a role, for they will influence the context in which powerful AIs are built. That’s part of why we created CFAR.

If you think this work is important, I hope you’ll donate now to support our work. MIRI is entirely supported by private funders like you. And if you donate before August 15th, your contribution will be matched by one of the generous backers of our current fundraising drive.

Thank you,

Luke Muehlhauser

Executive Director

Read more »

2013 Summer Matching Challenge!

 |   |  News

Thanks to the generosity of several major donors,† every donation to the Machine Intelligence Research Institute made from now until August 15th, 2013 will be matched dollar-for-dollar, up to a total of $200,000!

 

Donate Now!

$0

$50,000

$100,000

$150,000

$200,000

We have reached our goal of $200,000!

Now is your chance to double your impact while helping us raise up to $400,000 (with matching) to fund our research program.

Early this year we made a transition from movement-building to research, and we’ve hit the ground running with six major new research papers, six new strategic analyses on our blog, and much more. Give now to support our ongoing work on the future’s most important problem.

 

Accomplishments in 2013 so far

Future Plans You Can Help Support

  • We will host many more research workshops, including one in September, and one in December (with John Baez attending, among others).
  • Eliezer will continue to publish about open problems in Friendly AI. (Here is #1 and #2.)
  • We will continue to publish strategic analyses, mostly via our blog.
  • We will publish nicely-edited ebooks (Kindle, iBooks, and PDF) for more of our materials, to make them more accessible: The Sequences, 2006-2009 and The Hanson-Yudkowsky AI Foom Debate.
  • We will continue to set up the infrastructure (e.g. new offices, researcher endowments) required to host a productive Friendly AI research team, and (over several years) recruit enough top-level math talent to launch it.

(Other projects are still being surveyed for likely cost and strategic impact.)

We appreciate your support for our high-impact work! Donate now, and seize a better than usual chance to move our work forward. If you have questions about donating, please contact Louie Helm at (510) 717-1477 or louie@intelligence.org.

† $200,000 of total matching funds has been provided by Jaan Tallinn, Loren Merritt, Rick Schwall, and Alexei Andreev.

MIRI Has Moved!

 |   |  News

For the past several months, MIRI and its child organization CFAR have been working from a much-too-small office on the outskirts of Berkeley. At the end of June, MIRI and CFAR took over the 3rd floor of 2030 Addison St. in downtown Berkeley, which has sufficient space for both organizations.

Our new office is 0.5 blocks from the Downtown Berkeley BART exit at Shattuck & Addison, and 2 blocks from the UC Berkeley campus. Here’s a photo of the campus from our roof:

view of campus from roof (500px)

The proximity to UC Berkeley will make it easier for MIRI to network with Berkeley’s professors and students. Conveniently, UC Berkeley is ranked 5th in the world in mathematics, and 1st in the world in mathematical logic.

Sharing an office with CFAR carries many benefits for both organizations:

  1. CFAR and MIRI can “flex” into each other’s space for short periods as needed, for example when MIRI is holding a week-long research workshop.
  2. We can share resources (printers, etc.).
  3. Both organizations can benefit from interaction between our two communities.

Getting the new office was a team effort, but the person most responsible for this success was MIRI Deputy Director Louie Helm.

Note that MIRI isn’t yet able to accommodate “drop in” visitors, as we keep irregular hours throughout the week. So if you’d like to visit, please contact us first.

We retain 2721 Shattuck Ave. #1023 as an alternate mailing address.

MIRI’s September 2013 Workshop

 |   |  News

Paul at April workshop

From September 7-13, MIRI will host its 4th Workshop on Logic, Probability, and Reflection. The focus of this workshop will be the foundations of decision theory.

Participants confirmed so far include:

If you have a strong mathematics background and might like to attend this workshop, it’s not too late to apply! And even if this workshop doesn’t fit your schedule, please do apply, so that we can notify you of other workshops (long before they are announced publicly).

Responses to Catastrophic AGI Risk: A Survey

 |   |  Papers

MIRI is self-publishing another technical report that was too lengthy (60 pages) for publication in a journal: Responses to Catastrophic AGI Risk: A Survey.

The report, co-authored by past MIRI researcher Kaj Sotala and University of Louisville’s Roman Yampolskiy, is a summary of the extant literature (250+ references) on AGI risk, and can serve either as a guide for researchers or as an introduction for the uninitiated.

Here is the abstract:

Many researchers have argued that humanity will create artificial general intelligence (AGI) within the next twenty to one hundred years. It has been suggested that AGI may pose a catastrophic risk to humanity. After summarizing the arguments for why AGI may pose such a risk, we survey the field’s proposed responses to AGI risk. We consider societal proposals, proposals for external constraints on AGI behaviors, and proposals for creating AGIs that are safe due to their internal design.

The preferred discussion page for the paper is here.

Update: This report has now been published in Physica Scripta, available here.