2014 in review

 |   |  MIRI Strategy

It’s time for my review of MIRI in 2014.1 A post about our next strategic plan will follow in the next couple months, and I’ve included some details about ongoing projects at the end of this review.

 

2014 Summary

Since early 2013, MIRI’s core goal has been to help create a new field of research devoted to the technical challenges of getting good outcomes from future AI agents with highly general capabilities, including the capability to recursively self-improve.2

Launching a new field has been a team effort. In 2013, MIRI decided to focus on its comparative advantage in defining open problems and making technical progress on them. We’ve been fortunate to coordinate with other actors in this space — FHI, CSER, FLI, and others — who have leveraged their comparative advantages in conducting public outreach, building coalitions, pitching the field to grantmakers, interfacing with policymakers, and more.3

MIRI began 2014 with several open problems identified, and with some progress made toward solving them, but with very few people available to do the work. Hence, most of our research program effort in 2014 was aimed at attracting new researchers to the field and making it easier for them to learn the material and contribute. This was the primary motivation for our new technical agenda overview, the MIRIx program, our new research guide, and more (see below). Nick Bostrom’s Superintelligence was also quite helpful for explaining why this field of research should exist in the first place.

Today the field is much larger and healthier than it was at the beginning of 2014. MIRI now has four full-time technical researchers instead of just one. Around 85 people have attended one or more MIRIx workshops. There are so many promising researchers who have expressed interest in our technical research that ~25 of them have already confirmed interest and availability to attend a MIRI introductory workshop this summer, and this mostly doesn’t include people who have attended past MIRI workshops, nor have we sent out all the invites yet. Moreover, there are now several researchers we know who are plausible MIRI hires in the next 1-2 years.

I am extremely grateful to MIRI’s donors, without whom this progress would have been impossible.

The rest of this post provides a more detailed summary of our activities in 2014.

Overview of 2014 activities

  1. Technical research: We hired 2 new research fellows, launched the MIRIx program, hosted many visiting researchers, and released 14 technical papers/reports, including our new technical agenda overview.
  2. Strategic research: We published 15 analyses, 5 papers, and 54 expert interviews.
  3. Outreach: We organized some talks and media stories, and released Smarter Than Us. Yudkowsky also continued writing Harry Potter and the Methods of Rationality, which has proven to be a surprisingly effective outreach tool for MIRI’s work.
  4. Fundraising: We raised $1,237,557 in contributions in 2014. This is slightly less than we raised in 2013, but only because 2013’s numbers include a one-time, outlier donation of ~$525,000.
  5. Operations: We made many improvements to MIRI’s organizational efficiency and robustness.

 

2014 Technical Research

Two of the top three goals in our mid-2014 strategic plan were to (1) increase our technical research output and (2) invest heavily in recruiting additional technical researchers (via a prospecting -> prospect development -> hiring funnel). The third goal concerned fundraising; see below.

As for (1): this past year we released 14 technical papers/reports4 and gave a few technical talks for academic audiences. To give our staff researchers more time to write up existing results, and to focus more on recruiting, we held only one research workshop in 2014.

As for (2), in 2014 we:

  • Published our new research agenda overview, which makes it much easier for newcomers to understand what we’re doing and why.
  • Hired two new full-time research fellows. Benja Fallenstein and Nate Soares joined in April 2014. (Patrick LaVictoire joined MIRI in March 2015.)
  • Launched our MIRIx program, now with 14 active groups around the world. This program allows mathematicians and computer scientists to spend time studying and discussing MIRI’s research agenda, and is a key component of our prospecting and development pipeline.
  • Published A Guide to MIRI’s Research, which guides students through the topics and papers they should study to become familiar with each area of MIRI’s research agenda.
  • Hosted several visiting researchers to work with us on MIRI’s research problems in Berkeley for a few days or weeks at a time.5
  • Co-sponsored the SPARC 2014 camp, which trains mathematically talented youth to apply their quantitative thinking skills to their lives and the world. SPARC didn’t teach participants about MIRI’s research, but it did teach participants about effective altruism, and brought them into contact with our social circles more generally. At least one SPARC graduate has subsequently expressed serious interest in working for MIRI in the future (but mostly, they are still too early in their studies to be considered).

In addition, some of our outreach activities (described below) double as researcher prospecting activities.

In my estimation, the growth of our technical research program in 2014 fell short of my earlier goals, mostly due to insufficient staff capacity to launch new recruitment initiatives. Thankfully, this situation improved in early 2015. We are still seeking to hire additional operations staff to help us grow our technical research program,6 but in the meantime we have met some of our immediate capacity needs by (for example) contracting MIRIx participant James Cook to help us better steward and grow the MIRIx program, and contracting Jesse Galef to help us organize our summer 2015 workshops.

 

2014 Strategic Research

As planned, in 2014 we decreased our output of strategic research.7 Even still we published a sizable amount of strategic work in 2014:

  • 15 strategic analyses posted on MIRI’s blog and elsewhere.8
  • 54 expert interviews on a wide range of topics.
  • 5 papers/chapters on AI strategy topics.9

Nearly all of the interviews were begun in 2013 or early 2014, even if they were not finished and published until much later. Mid-way through 2014, we decided to de-prioritize expert interviews, due to apparent diminishing returns.

This level of strategic research output aligns closely with our earlier goals.

 

2014 Outreach

Our outreach efforts declined this year in favor of increased focus on our technical research. Our outreach efforts in 2014 included:

This level of outreach output aligns closely with our earlier goals, except that we had planned to release the ebook version of The Sequences in 2014, and this release was delayed until March 2015.

Despite MIRI’s declining outreach efforts, public outreach about MIRI’s core concerns massively increased in 2014 due mostly to the efforts of others. FHI‘s Nick Bostrom released Superintelligence, which is now the best available summary of the problem MIRI exists to solve. Several prominent figures — in particular Stephen Hawking and Elon Musk — promoted long-term AI safety concerns to the media’s attention. In addition, two new organizations, CSER and FLI, did substantial outreach on this issue. Largely as a result of these efforts, Edge.org decided to make its widely-read 2015 Annual Question about long-term AI risks. Several of the respondents expressed views basically aligned with MIRI’s thinking on the issue, including Sam Harris, Stuart Russell, Jaan Tallinn, Max Tegmark, Steve Omohundro, Nick Bostrom, and of course MIRI’s own Eliezer Yudkowsky.

 

2014 Fundraising

Originally, we set a very ambitious fundraising goal for 2014. Shortly thereafter, we decided to focus on recruiting-related efforts rather than fundraising. So while our 2014 fundraising fell far short of our original (very ambitious) goal, I think the decision to focus on recruiting rather than fundraising in 2014 was the right choice.

In 2014 we raised $1,237,557 in contributions.11 Our largest sources of funding were:

  • ~$400,000 from our summer matching challenge (this includes the $200,000 in matching funds from Jaan Tallinn, Edwin Evans, and Rick Schwall)
  • ~$200,000 from our winter matching challenge (this includes $100,000 in matching funds from the Thiel Foundation)
  • ~$175,000 from the one-day SV Gives fundraiser, ~$63,000 of which was from prizes and matching funds from donors who wouldn’t normally contribute to MIRI
  • ~$150,000 from the Thiel Foundation (in addition to the $100,000 in matching funds for the winter matching challenge)
  • ~$105,000 from Jed McCaleb.12

It is difficult to meaningfully compare MIRI’s 2013 and 2014 fundraising income to MIRI’s fundraising income in earlier years, because MIRI was a fairly different organization in 2012 and earlier.13 But I’ll show the comparison anyway:

Total donations were lower in 2014 than in 2013, but this is due to a one-time outlier donation in 2013 from Jed McCaleb, who was then a new donor. (By the way, this one donation allowed our research program to jump forward more quickly than I had originally been planning at the time.) If we set aside McCaleb’s large 2013 and 2014 gifts, MIRI’s fundraising grew slightly from 2013 to 2014.

New donor growth was strong in 2014, though this mostly came from small donations made during the SV Gives fundraiser. A significant portion of growth in returning donors can also be attributed to lapsed donors making small contributions during the SV Gives fundraiser.

This graph shows how much of our support during the past few years came from small, mid-sized, large, and very large donors. My understanding is that the distributions shown for 2012, 2013, and 2014 are fairly typical of non-profits our size. (Again, the green bar is taller in 2013 than in 2014 due to Jed McCaleb’s outlier 2013 donation.)

 

2014 Operations

Building on our 2013 organizational improvements, our operational efficiency and robustness improved substantially throughout 2014. Operations-related tasks, including the operational processes specific to our research program, now take up a smaller fraction of staff time than before, which has allowed us to divert more capacity to growing our research program. We also implemented many new policies and services that make MIRI more robust in the face of staff turnover, cyberattack, fluctuations in income, etc. I won’t go into much detail on operations in this post, but we’re typically happy to share what we’ve learned about running an efficient and robust organization when someone asks us to.

Coming Soon

Our next strategic plan post won’t be ready for another month or two, but of course we already have many projects in motion. Here’s what you can expect from MIRI over the next few months:

  • We have several technical reports and conference papers nearing completion.
  • We are running a series of workshops this summer. Many of the most promising people who applied to come to future workshops, or who are showing promise in MIRIx groups around the world, are being invited.
  • We are beginning to pay particularly productive MIRIx participants for part-time remote research on problems in MIRI’s technical agenda.
  • We are co-organizing a decision theory conference at Cambridge University in May.
  • Once again we are sponsoring SPARC‘s summer camp for mathematically talented high-schoolers.
  • We are “putting the finishing touches” on two large pieces of strategy research conducted in 2014. We will also finish running the Superintelligence reading group and then assemble the resulting Superintelligence reading guide. We will also contribute additional articles to AI Impacts, until our earmarked funding for that work runs out.14

Stay tuned for our next strategic plan, which will contain more detail about our planned programs.

  1. This year’s annual review is shorter than last year’s 5-part review of 2013, in part because 2013 was an unusually complicated focus-shifting year, and in part because, in retrospect, last year’s 5-part review simply took more effort to produce than it was worth. Also, because we recently finished switching to accrual accounting, I can now more easily provide annual reviews of each calendar year rather than of a March-through-February period. As such, this review of calendar year 2014 will overlap a bit with what was reported in the previous annual review (of March 2013 through February 2014). ↩
  2. Clearly there are forecasting and political challenges as well, and there are technical challenges related to ensuring good outcomes from nearer-term AI systems, but MIRI has chosen to specialize in the technical challenges of aligning superintelligence with human interests. See also: Friendly AI research as effective altruism and Why MIRI? ↩
  3. Obviously, the division of labor was more complex than I’ve described here. For example, FHI produced some technical research progress in 2014, and MIRI did some public outreach. ↩
  4. These were, roughly in chronological order: BotWorld, Program Equilibrium in the Prisoner’s Dilemma via Löb’s Theorem, Problems of self-reference in self-improving space-time embedded intelligence, Loudness, Distributions allowing tiling of staged subjective EU maximizers, Non-omniscience, probabilistic inference, and metamathematics, Corrigibility, UDT with known search order, Aligning superintelligence with human interests, Toward idealized decision theory, Computable probability distributions which converge…, Tiling agents in causal graphs, Concept learning for safe autonomous AI. I’m also counting Rob Bensinger’s blog post sequence on naturalized induction as one semi-technical “report.” A few of the technical agenda overview’s supporting papers weren’t announced on our blog until 2015. These aren’t counted here. For comparison’s sake, we released 10 technical papers/reports in 2013, but 7 of these were uncommonly short technical reports immediately following our December 2013 workshop. The 10 technical papers/reports from 2013 are: Scientific induction in probabilistic metamathematics, Fallenstein’s monster, Recursively-defined logical theories are well-defined, Tiling agents for self-modifying AI, and the Löbian obstacle, The procrastination paradox, A comparison of decision algorithms on Newcomblike problems, Definability of truth in probabilistic logic, The 5-and-10 problem and the tiling agents formalism, Decreasing mathematical strength in one formalization of parametric polymorphism, and An infinitely descending sequence of sound theories each proving the next consistent. ↩
  5. Visiting researchers in 2014 included Abram Demski, Scott Garrabrant, Nik Weaver, Nisan Stiennon, Vladimir Slepnev, Tsvi Benson-Tilsen, Danny Hintze, and Ilya Shpitser. ↩
  6. Hiring additional development staff will help grow our research program by freeing up more of my own time for research program work, and hiring one or more additional executives will directly add new staff capacity directed toward growing our research program. ↩
  7. This year, I have collapsed my previous categories of “strategic” and “expository” research into one category I simply call “strategic research.” ↩
  8. These were, roughly in chronological order: How Big is the Field of Artificial Intelligence?, Robust Cooperation: A Case Study in Friendly AI Research, The world’s distribution of computation, Is my view contrarian?, Exponential and non-exponential trends in information technology, How to study superintelligence strategy, Tentative tips for people engaged in an exercise that involves some form of prediction or forecasting, Groundwork for AGI safety engineering, Loosemore on AI safety and attractors, AGI outcomes and civilizational competence, The Financial Times story on MIRI, Three misconceptions in Edge.org’s conversation on “The Myth of AI”, Two mistakes about the threat from artificial intelligence, Brooks and Searle on AI volition and timelines, Davis on AI capability and motivation. ↩
  9. These were, roughly in chronological order: Embryo selection for cognitive enhancement, Why we need Friendly AI, The errors, insights, and lessons of famous AI predictions, The ethics of artificial intelligence, and Exploratory engineering in artificial intelligence. ↩
  10. HPMoR has now finished, but it wasn’t finished yet in 2014. ↩
  11. MIRI has some sources of funding besides contributions. For example, in 2014 our realized and unrealized gains, plus interest and dividends — but not including realized and unrealized gains for our cryptocurrency holdings, which are highly volatile — amounted to ~$97,000. We also made ~$7,000 from ebook sales, and ~$5,000 from Give for Free programs. ↩
  12. This donation was made in Ripple, which we eventually sold. ↩
  13. See “Comparison to past years” here. Also, during 2014 we switched to accrual accounting, which confuses the comparison to past years even further. Furthermore, the numbers in this section might not exactly match past published estimates, because every now and then we still find and correct old errors in our donor database. Finally, note that in-kind donations are not included in the numbers or graphs on this page. ↩
  14. We are happy to support such strategic research given earmarked funding and low management overhead, but otherwise we are focusing on our technical research program. ↩