2013 in Review: Operations

 |   |  MIRI Strategy

It’s been 8 months since our big strategic shift to a focus on Friendly AI research. What has worked well? What hasn’t? What have we learned? In this and several upcoming posts, I’ll provide a qualitative and personal self-review of MIRI in 2013.

This review will be mostly qualitative because we haven’t yet found quantitative metrics that do a decent job of tracking what we actually care about (see below). The review will be personal because MIRI staff have a variety of perspectives, so I will usually be speaking for myself rather than for the organization as a whole.

I’ll split my review into six posts: operations (this post), outreach, strategic and expository research, Friendly AI research, fundraising, and our mid-2014 strategic plan.

 

Operations in 2013

  1. 2013 was a big year for capacity-building at MIRI — not so much by hiring more people but by doing more with the people we have.
  2. That said, this progress cost lots of executive time, and I should have been more aggressive about finding ways to accomplish this with fewer executive hours.
  3. As a result of (2), we did not make as much progress on metrics and systematic experimentation as I would have liked to make in 2013.

Capacity-building at MIRI in 2013

Here I’ll explain how our operations capacity — our ability to efficiently execute a wide range of programs and projects — has grown in 2013.1 Our research capacity will be discussed in a later post.

In 2013, our operations capacity grew not by hiring more operations staff, but by making our processes more efficient and thereby doing more with a roughly-stable number of staff.2

Below is a partial list of operational improvements we made in 2013.3 I suspect other nonprofit executives can benefit from some of them.

  • We moved to a larger and more centrally-located office in downtown Berkeley, 2.5 blocks from the UC Berkeley campus and 0.5 blocks from a BART subway entrance. As expected, this has made it easier for people to work together on projects, easier to meet with visitors, easier to host research workshops, etc.
  • We streamlined our Board meetings and reduced their frequency by providing the Board with a monthly “snapshot” update in PDF, and by giving all but the highest-level decisions to the executives, or to committees composed of particular board members and executives. (We still need to do more of this, though.)
  • We won an exemption to the USA’s annual cap on H-1B visas. This makes it easier for us to bring in foreign talent to work with us in Berkeley, thus greatly expanding our pool of potential hires. Using this cap exemption, we won H-1B visas for Malo Bourgon and Alex Vermeer. Alex has already joined us in Berkeley and realized the expected productivity and efficiency gains. Malo will join us in Berkeley next year.
  • Many operational improvements came from automating  processes with software. For example, we tested several time-tracking solutions and settled on Harvest, which makes it easy to track staff and contractor hours, and also to estimate the time costs of each major project. We also manage some projects, especially those involving contractors, via Basecamp.
  • With the help of a few custom scripts, we now have a pretty accurate donations database (via DonorTools) that is integrated with our favorite CRM (HighRise) and some web pages (e.g. the live progress bar for our ongoing fundraiser). This makes it easier for us to track donations, and to stay in touch with donors, potential workshop participants, and other contacts.
  • After struggling to efficiently use popular, broken payroll systems like Paychex and Wells Fargo Payroll, we made the jump to ZenPayroll and discovered it to be exactly what it claims to be: a painless, modern payroll system with responsive customer support.
  • Lots of other little tweaks: we consolidated our web domains and web hosting, we record expenses by emailing smartphone photos of receipts to an email address, we use EchoSign for signing contracts online instead of faxing or mailing them, we switched to MailChimp for newsletter management and analytics, we switched to Disqus for blog comment management, etc.

Compared to mid-2012, it is noticeably easier to just get things done at MIRI. In mid-2012, I would often find that doing X required that Y and Z be in place, and to set up Y and Z we needed four different documents, and we didn’t have any of those documents yet because MIRI hadn’t previously invested much in operations, and it was hard to outsource the production of those documents because we didn’t have a central database for that kind of organizational information, so I’d have to pull it all from my own head and make phone calls to other people, and I couldn’t fill in line 6 of document 3 because the payroll system once again wouldn’t let me load that report, and I couldn’t estimate the project costs requested by document 2 because there had been no costs tracking, and… well, these kinds of tasks are much easier now, and much easier to outsource, which means that MIRI’s executives (Louie and I) can spend more time on organizational strategy, fundraising, and building the research program.

 

These operational improvements consumed too much executive time

For the past few months, these operational improvements have largely been tested and implemented by Malo Bourgon and (to a lesser extent) Alex Vermeer. But for most of 2013, these improvements required much personal effort from MIRI’s executives, which carries an especially high opportunity cost. In retrospect, I think I could have conserved executive time better:

  • I could have, by mid-2012, recognized our ongoing need for technically-savvy operations support, and recruited more aggressively for that role. That said, I’m not sure more recruiting effort here would have paid off. (See here.)
  • I could have further reduced the executive time needed to make these improvements, by giving greater responsibility to Malo Bourgon earlier, and by helping Malo outsource his other responsibilities sooner.4
  • I could have done more to free up executive time in other ways, by delegating tasks and projects more aggressively. For example, I personally ran onsite logistics for some of this year’s research workshops, to “make sure everything went well,” but I should have delegated that role more fully to others, even if it wouldn’t have been executed quite as well. That said, we did outsource quite a lot of work (2500+ contractor hours) in 2013.

 

As a result, we didn’t make much progress on metrics and experimentation

To improve MIRI’s effectiveness via tight feedback loops, we need quantitative measures of the kinds of progress we care about, and we need to run frequent experiments to test hypotheses about which strategies will work best for operational efficiency, outreach success, research progress, and fundraising.

MIRI is a quickly-evolving organization with hard-to-measure goals, and thus it requires significant high-level investment from MIRI’s executives to figure out how to measure variables we care about (instead of variables that are easy to measure), via experimentation and other means. But because so much executive time in 2013 was consumed by operational improvements, we made little progress on metrics and experimentation until very recently. Right now we’re experimenting with the “OKRs and KPIs” framework used at Intel, Google, Quixey, and other companies, but we have a lot more work to do.

I think another reason for our lack of progress on metrics and experimentation, besides the lack of executive time, was that I failed to set up deadlines for concrete progress on developing metrics and experiments. In part this was because “make progress figuring out how to measure progress” felt too abstract for useful deadlines, and in part it was because MIRI’s object-level work usually felt more urgent to me than the “meta-level” work on metrics and experimentation. But these were poor reasons to avoid putting deadlines on an important project.

 

What I’m doing to improve

Here’s what I’m doing to improve our progress on metrics and experimentation:

  1. I’m more aggressively delegating and outsourcing tasks and projects, and helping Louie, Malo, and Alex to do the same.
  2. I’m prioritizing the development of metrics and experimentation over some object-level projects that could otherwise consume staff time.
  3. I’m setting up regular deadlines for concrete increments of progress on the development of metrics and experiments.

I do not, however, intend to hire additional full-time staff to achieve our operations goals, unless we significantly expand our operations in 2014 in a way that requires more operations staff. We agree with GiveWell that growing capacity by hiring staff is difficult and risky — in this case, unnecessarily so.

In the next part of my 2013 review, I’ll review our outreach activities in 2013.

  1. Some operations work in 2013 wasn’t related to improving our operations capacity per se. For example, we also wrote and passed some important policies that are (e.g.) required to get certain insurance policies. We also setup full offsite backups, made sure our passwords are protected with 1Password, etc. For communication security, we signed up for a trial of Silent Circle (one month before the Snowden leaks), but unfortunately Silent Circle’s phone solution was buggy and their email solution never materialized. ↩
  2. Alex Vermeer became a full-time team member during 2013, and Malo Bourgon went full-time near the end of 2012, but Malo was already a near-full-time contractor before that, and Alex’s rising hours in the latter half of 2013 quantitatively substituted for Ioven’s hours from the first third of 2013, which were lost when Ioven departed in mid-March. ↩
  3. In other cases, it’s not clear that the implemented change improved our efficiency. In particular, we set up MIRIvolunteers.org (using YouTopia) to manage and incentivize volunteers, but it’s not clear whether this has been worth it. We have 450+ volunteers signed up on the site, and the clickthrough rate for volunteer task announcements is surprisingly high (19%), but there were only about 25hrs/mo of volunteer labor throughout 2013, which meant that many volunteer projects saw slow progress, or none at all. Given the time cost of setting up the YouTopia site and maintaining engagement with volunteers, it’s at least not obvious that MIRIvolunteers.org increases our efficiency. We remain curious to see how much volunteer engagement we get with the Sequences ebook project, and also whether engagement improves as the YouTopia team implements new features. ↩
  4. I couldn’t have done this so easily with Alex Vermeer, given the projects he was running at the time. ↩