But predicting AI is more difficult than many people think.

To explore these difficulties, let’s start with a 2009 bloggingheads.tv conversation between MIRI researcher Eliezer Yudkowsky and MIT computer scientist Scott Aaronson, author of the excellent Quantum Computing Since Democritus. Early in that dialogue, Yudkowsky asked:

It seems pretty obvious to me that at some point in [one to ten decades] we’re going to build an AI smart enough to improve itself, and [it will] “foom” upward in intelligence, and by the time it exhausts available avenues for improvement it will be a “superintelligence” [relative] to us. Do you feel this is obvious?

Aaronson replied:

The idea that we could build computers that are smarter than us… and that those computers could build still smarter computers… until we reach the physical limits of what kind of intelligence is possible… that we could build things that are to us as we are to ants — all of this is compatible with the laws of physics… and I can’t find a reason of principle that it couldn’t eventually come to pass…

The main thing we disagree about is the time scale… a few thousand years [before AI] seems more reasonable to me.

Those two estimates — several decades vs. “a few thousand years” — have wildly different policy implications.

If there’s a good chance that AI will replace humans at the steering wheel of history in the next several decades, then we’d better put our gloves on and get to work making sure that this event has a positive rather than negative impact. But if we can be pretty confident that AI is thousands of years away, then we needn’t worry about human-level AI for now, and we should focus on other global priorities. Thus it appears that “When will AI be created?” is a question with high value of information for our species.

Let’s take a moment to review the forecasting work that has been done, and see what conclusions we might draw about when AI will likely be created.

The challenge of forecasting AI

Expert elicitation

Maybe we can ask the experts? Astronomers are pretty good at predicting eclipses, even decades or centuries in advance. Technological development tends to be messier than astronomy, but maybe the experts can still give us a range of years during which we can expect AI to be built? This method is called expert elicitation.

Several people have surveyed experts working in AI or computer science about their AI timelines. Unfortunately, most of these surveys suffer from rather strong sampling bias, and thus aren’t very helpful for our purposes.¹

Should we expect experts to be good at predicting AI, anyway? As Armstrong & Sotala (2012) point out, decades of research on expert performance² suggest that predicting the first creation of AI is precisely the kind of task on which we should expect experts to show poor performance — e.g. because feedback is unavailable and the input stimuli are dynamic rather than static. Muehlhauser & Salamon (2013) add, “If you have a gut feeling about when AI will be created, it is probably wrong.”

That said, the experts surveyed in Michie (1973) — a more representative sample than in other surveys³ — did pretty well. When asked to estimate a timeline for “[computers] exhibiting intelligence at adult human level,” the majority of experts gave the most pessimistic answer allowed: “More than 50 years.” Assuming (as most people do) that AI will not arrive by 2023, these experts will have been correct.

Unfortunately, “more than 50 years” is a broad time frame that includes both “several decades from now” and “thousands of years from now.” So we don’t yet have any evidence that a representative survey of experts can predict AI within a few decades, and we have general reasons to suspect experts may not be capable of doing this kind of forecasting very well — although various aids (e.g. computational models; see below) may help them to improve their performance.

How else might we forecast when AI will be created?

Trend extrapolation

Many have tried to forecast the first creation of AI by extrapolating various trends. Like Kevin Drum, Vinge (1993) based his own predictions about AI on hardware trends (e.g. Moore’s Law). But in a 2003 reprint of his article, Vinge noted the insufficiency of this reasoning: even if we acquire hardware sufficient for AI, we may not have the software problem solved.⁴ As Robin Hanson reminds us, “AI takes software, not just hardware.”

Perhaps instead we could extrapolate trends in software progress?⁵ Some people estimate the time until AI by asking what proportion of human abilities today’s software can match, and how quickly machines are “catching up.”⁶ Unfortunately, it’s not clear how to divide up the space of “human abilities,” nor how much each ability matters. Moreover, software progress seems to come in fits and starts.⁷ With the possible exception of computer chess progress, I’m not aware of any trend in software progress as robust across multiple decades as Moore’s Law is in computing hardware.

On the other hand, Tetlock (2005) points out that, at least in his large longitudinal database of pundit’s predictions about politics, simple trend extrapolation is tough to beat. Consider one example from the field of AI: when David Levy asked 1989 World Computer Chess Championship participants when a chess program would defeat the human World Champion, their estimates tended to be inaccurately pessimistic,⁸ despite the fact that computer chess had shown regular and predictable progress for two decades by that time. Those who forecasted this event with naive trend extrapolation (e.g. Kurzweil 1990) got almost precisely the correct answer (1997).

Hence, it may be worth searching for a measure for which (a) progress is predictable enough to extrapolate, and for which (b) a given level of performance on that measure robustly implies the arrival of human-level AI. But to my knowledge, this has not yet been done, and it’s not clear that trend extrapolation can tell us much about AI timelines until such an argument is made, and made well.

Disruptions

Worse, several events could significantly accelerate or decelerate our progress toward AI, and we don’t know which of these events will occur, nor in what order. For example:

• An end to Moore’s Law. The “serial speed” version of Moore’s Law broke down in 2004, requiring a leap to parallel processors, which raises substantial new difficulties for software developers (Fuller & Millett 2011). The most economically relevant formulation of Moore’s law, computations per dollar, has been maintained thus far,⁹ but it remains unclear as to whether this will continue much longer (Mack 2011; Esmaeilzadeh et al. 2012).
• Depletion of low-hanging fruit. Progress is not only a function of effort but also of the difficulty of the progress. Some fields see a pattern of increasing difficulty with each successive discovery (Arbesman 2011). AI may prove to be a field in which new progress requires far more effort than earlier progress. That is clearly the case for many parts of AI already, for example natural language processing (Davis 2012).
• Societal collapse. Political, economic, technological, or natural disasters may cause a societal collapse during which progress in AI would be essentially stalled (Posner 2004; Bostrom and Ćirković 2008).
• Disinclination. Chalmers (2010) and Hutter (2012a) think the most likely “speed bump” in our progress toward AI will be disinclination. As AI technologies become more powerful, humans may question whether it is wise to create machines more powerful than themselves.
• A breakthrough in cognitive neuroscience. It is difficult, with today’s tools, to infer the cognitive algorithms behind human intelligence (Trappenberg 2009). New tools and methods, however, might enable cognitive neuroscientists to decode how the human brain achieves its own intelligence, which might allow AI scientists to replicate that approach in silicon.
• Human enhancement. Human enhancement technologies may make scientists more effective via cognitive enhancement pharmaceuticals (Bostrom and Sandberg 2009), brain-computer interfaces (Groß 2009), and genetic selection or engineering for cognitive enhancement.¹⁰
• Quantum computing. Quantum computing has overcome some of its early hurdles (Rieffel and Polak 2011), but it remains difficult to predict whether quantum computing will contribute significantly to the development of machine intelligence. Progress in quantum computing depends on particularly unpredictable breakthroughs. Furthermore, it seems likely that even if built, a quantum computer would provide dramatic speedups only for specific applications (e.g. searching unsorted databases).
• A tipping point in development incentives. The launch of Sputnik in 1957 demonstrated the possibility of space flight to the public. This event triggered a space race between the United States and the Soviet Union, and led to long-term funding for space projects from both governments. If there is a “Sputnik moment” for AI that makes it clear to the public and to governments that smarter-than-human AI is inevitable, a race to human-level AI may ensue, especially since the winner of the AI race might reap extraordinary economic, technological and geopolitical advantage.¹¹

Great uncertainty

Given these considerations, I think the most appropriate stance on the question “When will AI be created?” is something like this:

We can’t be confident AI will come in the next 30 years, and we can’t be confident it’ll take more than 100 years, and anyone who is confident of either claim is pretending to know too much.

How confident is “confident”? Let’s say 70%. That is, I think it is unreasonable to be 70% confident that AI is fewer than 30 years away, and I also think it’s unreasonable to be 70% confident that AI is more than 100 years away.

This statement admits my inability to predict AI, but it also constrains my probability distribution over “years of AI creation” quite a lot.

I think the considerations above justify these constraints on my probability distribution, but I haven’t spelled out my reasoning in great detail. That would require more analysis than I can present here. But I hope I’ve at least summarized the basic considerations on this topic, and those with different probability distributions than mine can now build on my work here to try to justify them.

How to reduce our ignorance

But let us not be satisfied with a declaration of ignorance. Admitting our ignorance is an important step, but it is only the first step. Our next step should be to reduce our ignorance if we can, especially for high-value questions that have large strategic implications concerning the fate of our entire species.

How can we improve our long-term forecasting performance? Horowitz & Tetlock (2012), based on their own empirical research and prediction training, offer some advice on the subject:

• Explicit quantification: “The best way to become a better-calibrated appraiser of long-term futures is to get in the habit of making quantitative probability estimates that can be objectively scored for accuracy over long stretches of time. Explicit quantification enables explicit accuracy feedback, which enables learning.”
• Signposting the future: Thinking through specific scenarios can be useful if those scenarios “come with clear diagnostic signposts that policymakers can use to gauge whether they are moving toward or away from one scenario or another… Falsifiable hypotheses bring high-flying scenario abstractions back to Earth.”
• Leveraging aggregation: “the average forecast is often more accurate than the vast majority of the individual forecasts that went into computing the average…. [Forecasters] should also get into the habit that some of the better forecasters in [an IARPA forecasting tournament called ACE] have gotten into: comparing their predictions to group averages, weighted-averaging algorithms, prediction markets, and financial markets.

Many forecasting experts add that when making highly uncertain predictions, it usually helps to decompose the phenomena into many parts and make predictions about each of the parts.¹² As Raiffa (1968) succinctly put it, our strategy should be to “decompose a complex problem into simpler problems, get one’s thinking straight [on] these simpler problems, paste these analyses together with a logical glue, and come out with a program for action for the complex problem” (p. 271). MIRI’s The Uncertain Future is a simple toy model of this kind, but more sophisticated computational models could be produced, and integrated with other prediction techniques.

We should expect AI forecasting to be difficult, but we need not be as ignorant about AI timelines as we are today.

Notes

My thanks to Carl Shulman, Ernest Davis, Louie Helm, Scott Aaronson, and Jonah Sinick for their helpful feedback on this post.

¹ First, Sandberg & Bostrom (2011) gathered the AI timeline predictions of 35 participants at a 2011 academic conference on human-level machine intelligence. Participants was asked by what year they thought there is a 10%, 50%, and 90% chance that AI will have been built, assuming that “no global catastrophe halts progress.” Five of the 35 respondents expressed varying degrees of confidence that human-level AI would never be achieved. The median figures, calculated from the views of the other 30 respondents, were: 2028 for “10% chance,” 2050 for “50% chance,” and 2150 for “90% chance.” Second, Baum et al. (2011) surveyed 21 participants at a 2009 academic conference on machine intelligence, and found estimates similar to those in Sandberg & Bostrom (2011). Third, Kruel (2012) has, as of May 7th, 2013, interviewed 34 people about AI timelines and risks via email, 33 of whom could be considered “experts” of one kind or another in AI or computer science (Richard Carrier is a historian). Of those 33 experts, 19 provided full, quantitative answers to Kruel’s question about AI timelines: “Assuming beneficial political and economic development and that no global catastrophe halts progress, by what year would you assign a 10%/50%/90% chance of the development of artificial intelligence that is roughly as good as humans (or better, perhaps unevenly) at science, mathematics, engineering and programming?” For those 19 experts, the median estimates for 10%, 50%, and 90% were 2025, 2035, and 2070, respectively (spreadsheet here). Fourth, Bainbridge (2005), surveying participants of 3 conferences on “Nano-Bio-Info-Cogno” technological convergence, found a median estimate of 2085 for “the computing power and scientific knowledge will exist to build machines that are functionally equivalent to the human brain.” However, the participants in these four surveys were disproportionately HLAI enthusiasts, and this introduces a significant sampling bias. The database of AI forecasts discussed in Armstrong & Sotala (2012) probably suffers from a similar problem: individuals who thought AI was imminent rather than distant were more likely to make public predictions of AI.

² Shanteau (1992); Kahneman and Klein (2009).

³ Another survey was taken at the AI@50 conference in 2006. When participants were asked “When will computers be able to simulate every aspect of human intelligence?”, 41% said “More than 50 years” and another 41% said “Never.” Unfortunately, many of the survey participants were not AI experts but instead college students who were attending the conference. Moreover, the phrasing of the question may have introduced a bias. The “Never” answer may have been given as often as it was because some participants took “every aspect of human intelligence” to include consciousness, and many people have philosophical objections to the idea that machines could be conscious. Had they instead been asked “When will AIs replace humans in almost all jobs?”, I suspect the “Never” answer would have been far less common. As for myself, I don’t accept any of the in-principle objections to the possibility of AI. For replies to the most common of these objections, see Chalmers (1996), ch. 9, and Chalmers (2012).

⁴ Though, Muehlhauser & Salamon (2013) point out that “Hardware extrapolation may be a more useful method in a context where the intelligence software is already written: whole brain emulation [WBE]. Because WBE seems to rely mostly on scaling up existing technologies like microscopy and large-scale cortical simulation, WBE may be largely an “engineering” problem, and thus the time of its arrival may be more predictable than is the case for other kinds of AI.” However, it is especially difficult to forecast WBE while we do not even have a proof of concept via a simple organism like C. elegans (David Dalrymple is working on this). Moreover, much progress in neuroscience will be required (Sandberg & Bostrom 2011), and such progress is probably less predictable than hardware extrapolation.

⁵ I’m not sure what a general measure of software progress would look like, though we can certainly identify local examples of software progress. For example, Holdren et al. (2010) notes: “in many areas, performance gains due
to improvements in algorithms have vastly exceeded even the dramatic performance gains due to increased
processor speed… [For example] Martin Grötschel…, an expert in optimization, observes that a benchmark production planning model solved using linear programming would have taken 82 years to solve in 1988, using the computers and the linear programming algorithms of the day. Fifteen years later – in 2003 – this same model could be solved in roughly 1 minute, an improvement by a factor of roughly 43 million. Of this, a factor of roughly 1,000 was due to increased processor speed, whereas a factor of roughly 43,000 was due to improvements in algorithms! Grötschel also cites an algorithmic improvement of roughly 30,000 for mixed integer programming between 1991 and 2008.” Muehlhauser & Salamon (2013) give another example: “For example, IBM’s Deep Blue played chess at the level of world champion Garry Kasparov in 1997 using about 1.5 trillion instructions per second (TIPS), but a program called Deep Junior did it in 2003 using only 0.015 TIPS. Thus, the computational efficiency of the chess algorithms increased by a factor of 100 in only six years (Richards and Shaw 2004).” A third example is Setty et al. (2012), which improved the efficiency of a probabilistically checkable proof method by 20 orders of magnitude with a single breakthrough. On the other hand, one can easily find examples of very slow progress, too (Davis 2012).

⁶ For example, see Good (1970).

⁷ As I wrote earlier: “Increases in computing power are pretty predictable, but for AI you probably need fundamental mathematical insights, and it’s damn hard to predict those. In 1900, David Hilbert posed 23 unsolved problems in mathematics. Imagine trying to predict when those would be solved.” Some of these problems were solved quickly, some of them required several decades to solve, and many of them remain unsolved. Even the order in which Hilbert’s problems would be solved was hard to predict. According to Erdős & Graham (1980), p. 7, “Hilbert lectured in the early 1920′s on problems in mathematics and said something like this: probably all of us will see the proof of the Riemann hypothesis, some of us… will see the proof of Fermat’s last theorem, but none of us will see the proof that √2^√2 is transcendental.” In fact, these results came in the reverse order: the last was proved by Kusmin a few years later, Fermat’s last theorem was proved by Wiles in 1994, and the Riemann hypothesis still has not been proved or disproved.

⁸ According to Levy & Newborn (1991), one participant guessed the correct year (1997), thirteen participants guessed years from 1992-1995, twenty-eight participants guessed years from 1998-2056, and one participant guessed “Never.” Of the twenty-eight who guessed years from 1998-2056, eleven guessed year 2010 or later.

⁹ As Fuller & Millett (2011, p. 81) note, “When we talk about scaling computing performance, we implicitly mean to increase the computing performance that we can buy for each dollar we spend.” Most of us don’t really care whether our new computer has more transistors or some other structure; we just want it to do more stuff, more cheaply. Kurzweil (2012), ch. 10, footnote 10 shows “calculations per second per $1,000″ growing exponentially from 1900 through 2010, including several data points after the serial speed version of Moore’s Law broke down in 2004. The continuation of this trend is confirmed by “instructions per second per dollar” data for 2006-2011, gathered from Intel and other sources by Chris Hallquist (spreadsheet here). Thus it seems that the computations per dollar form of Moore’s Law has continued unabated, at least for now.

¹⁰ One possible breakthrough here may be iterated embryo selection. See Miller (2012, ch. 9) for more details.

¹¹ It is interesting, however, that the United States did not pursue extraordinary economic, technological and geopolitical advantage in the period during which it was the sole possessor of nuclear weapons. Also, it is worth noting that violence and aggression have steadily declined throughout human history (Pinker 2012).

¹² E.g. Armstrong & Sotala (2012); MacGregor (2001); Lawrence et al. (2006).

The post When Will AI Be Created? appeared first on Machine Intelligence Research Institute.

If you sign up, we will occasionally ask you questions, or send you early drafts of upcoming writings for feedback.

We don’t always want technical advice (“Well, you can do that with a relativized arithmetical hierarchy…”); often, we just want to understand how different groups of experts respond to our writing (“The tone of this paragraph rubs me the wrong way because…”).

At the moment, we are most in need of advisors on the following subjects:

• Mathematical logic (especially computability theory and proof theory, more-especially provability logic, and most-especially the combination of logic and probabilism)
• Theoretical computer science (especially computability theory, computational complexity theory, and AIXI)
• Artificial intelligence (especially machine learning, agent architectures, and probabilistic graphical models)
• Economics (especially endogenous growth theory and the economics of innovation)
• Game theory (especially program equilibrium and normative decision theory)

Even if you don’t have much time to help, please sign up! We will of course respect your own limits on availability.

The post Advise MIRI with Your Domain-Specific Expertise appeared first on Machine Intelligence Research Institute.

This, and a number of other basic strategic views, can be summed up as a consequence of 5 theses about purely factual questions about AI, and 2 lemmas we think are implied by them, as follows:

Intelligence explosion thesis. A sufficiently smart AI will be able to realize large, reinvestable cognitive returns from things it can do on a short timescale, like improving its own cognitive algorithms or purchasing/stealing lots of server time. The intelligence explosion will hit very high levels of intelligence before it runs out of things it can do on a short timescale. See: Chalmers (2010); Muehlhauser & Salamon (2013); Yudkowsky (2013).

Orthogonality thesis. Mind design space is huge enough to contain agents with almost any set of preferences, and such agents can be instrumentally rational about achieving those preferences, and have great computational power. For example, mind design space theoretically contains powerful, instrumentally rational agents which act as expected paperclip maximizers and always consequentialistically choose the option which leads to the greatest number of expected paperclips. See: Bostrom (2012); Armstrong (2013).

Convergent instrumental goals thesis. Most utility functions will generate a subset of instrumental goals which follow from most possible final goals. For example, if you want to build a galaxy full of happy sentient beings, you will need matter and energy, and the same is also true if you want to make paperclips. This thesis is why we’re worried about very powerful entities even if they have no explicit dislike of us: “The AI does not love you, nor does it hate you, but you are made of atoms it can use for something else.” Note though that by the Orthogonality Thesis you can always have an agent which explicitly, terminally prefers not to do any particular thing — an AI which does love you will not want to break you apart for spare atoms. See: Omohundro (2008); Bostrom (2012).

Complexity of value thesis. It takes a large chunk of Kolmogorov complexity to describe even idealized human preferences. That is, what we ‘should’ do  is a computationally complex mathematical object even after we take the limit of reflective equilibrium (judging your own thought processes) and other standard normative theories. A superintelligence with a randomly generated utility function would not do anything we see as worthwhile with the galaxy, because it is unlikely to accidentally hit on final preferences for having a diverse civilization of sentient beings leading interesting lives. See: Yudkowsky (2011); Muehlhauser & Helm (2013).

Fragility of value thesis. Getting a goal system 90% right does not give you 90% of the value, any more than correctly dialing 9 out of 10 digits of my phone number will connect you to somebody who’s 90% similar to Eliezer Yudkowsky. There are multiple dimensions for which eliminating that dimension of value would eliminate almost all value from the future. For example an alien species which shared almost all of human value except that their parameter setting for “boredom” was much lower, might devote most of their computational power to replaying a single peak, optimal experience over and over again with slightly different pixel colors (or the equivalent thereof). Friendly AI is more like a satisficing threshold than something where we’re trying to eke out successive 10% improvements. See: Yudkowsky (2009, 2011).

These five theses seem to imply two important lemmas:

Indirect normativity. Programming a self-improving machine intelligence to implement a grab-bag of things-that-seem-like-good-ideas will lead to a bad outcome, regardless of how good the apple pie and motherhood sounded. E.g., if you give the AI a final goal to “make people happy” it’ll just turn people’s pleasure centers up to maximum. “Indirectly normative” is Bostrom’s term for an AI that calculates the ‘right’ thing to do via, e.g., looking at human beings and modeling their decision processes and idealizing those decision processes (e.g. what you would-want if you knew everything the AI knew and understood your own decision processes, reflective equilibria, ideal advisior theories, and so on), rather than being told a direct set of ‘good ideas’ by the programmers. Indirect normativity is how you deal with Complexity and Fragility. If you can succeed at indirect normativity, then small variances in essentially good intentions may not matter much — that is, if two different projects do indirect normativity correctly, but one project has 20% nicer and kinder researchers, we could still hope that the end results would be of around equal expected value. See: Muehlhauser & Helm (2013).

Large bounded extra difficulty of Friendliness. You can build a Friendly AI (by the Orthogonality Thesis), but you need a lot of work and cleverness to get the goal system right. Probably more importantly, the rest of the AI needs to meet a higher standard of cleanness in order for the goal system to remain invariant through a billion sequential self-modifications. Any sufficiently smart AI to do clean self-modification will tend to do so regardless, but the problem is that intelligence explosion might get started with AIs substantially less smart than that — for example, with AIs that rewrite themselves using genetic algorithms or other such means that don’t preserve a set of consequentialist preferences. In this case, building a Friendly AI could mean that our AI has to be smarter about self-modification than the minimal AI that could undergo an intelligence explosion. See: Yudkowsky (2008) and Yudkowsky (2013).

These lemmas in turn have two major strategic implications:

1. We have a lot of work to do on things like indirect normativity and stable self-improvement. At this stage a lot of this work looks really foundational — that is, we can’t describe how to do these things using infinite computing power, let alone finite computing power.  We should get started on this work as early as possible, since basic research often takes a lot of time.
2. There needs to be a Friendly AI project that has some sort of boost over competing projects which don’t live up to a (very) high standard of Friendly AI work — a project which can successfully build a stable-goal-system self-improving AI, before a less-well-funded project hacks together a much sloppier self-improving AI.  Giant supercomputers may be less important to this than being able to bring together the smartest researchers (see the open question posed in Yudkowsky 2013) but the required advantage cannot be left up to chance.  Leaving things to default means that projects less careful about self-modification would have an advantage greater than casual altruism is likely to overcome.

The post Five theses, two lemmas, and a couple of strategic implications appeared first on Machine Intelligence Research Institute.

AGI impacts experts develop skills related to predicting technological development (e.g. building computational models of AI development or reasoning about intelligence explosion microeconomics), predicting AGI’s likely impacts on society, and identifying which interventions are most likely to increase humanity’s chances of safely navigating the creation of AGI. For overviews, see Bostrom & Yudkowsky (2013); Muehlhauser & Salamon (2013).

Friendly AI experts develop skills useful for the development of mathematical architectures that can enable AGIs to be trustworthy (or “human-friendly”). This work is carried out at MIRI research workshops and in various publications, e.g. Christiano et al. (2013); Hibbard (2013). Note that the term “Friendly AI” was selected (in part) to avoid the suggestion that we understand the subject very well — a phrase like “Ethical AI” might sound like the kind of thing one can learn a lot about by looking it up in an encyclopedia, but our present understanding of trustworthy AI is too impoverished for that.

Now, what do we mean by “expert”?

Reliably superior performance on representative tasks

An expert is “a person with a high degree of skill in or knowledge of a certain subject.” Some domains (e.g. chess) provide objective measures of expertise, while other domains rely on peer recognition (e.g. philosophy). However, as Ericsson (2006) notes:

people recognized by their peers as experts do not always display superior performance on domain-related tasks. Sometimes they are no better than novices even on tasks that are central to the expertise, such as selecting stocks with superior future value, treatment of psychotherapy patients, and forecasts.

Thus, we should specify that the kind of expertise we want in AGI impacts experts and Friendly AI experts is what Ericsson (2006) calls “Expertise as Reliably Superior Performance on Representative Tasks” (RSPRT). It won’t do humanity much good to have a bunch of peer-credentialed “AGI impacts experts” who aren’t really any better than laypeople at predicting AGI outcomes, or a bunch of “Friendly AI experts” who aren’t much good at generating new FAI-relevant math results.

As an example of expertise as RSPRT, consider chess. Do chess ratings reliably track with RSPRT? Yes they do. For example, chess ratings are highly correlated with the ability to select the best move for presented chess positions (de Groot 1978; Ericsson & Lehmann 1996; Van der Maas & Wagenmakers 2005).

Similar methods have been used to confirm “expertise as RSPRT” in medicine (Ericsson 2004, 2007), sport (Côté et al. 2012), Scrabble (Tuffiash et al. 2007), and music (Lehmann & Grüber 2006).

So, what are some “representative tasks” for which AGI impacts experts and FAI experts should demonstrate superior performance?

Scholastic expertise

At the very least, we’d hope AGI impacts experts and Friendly AI experts would have a kind of scholastic expertise in AGI impacts and Friendly AI. That is, they should know what the basic debates are about, which arguments and counter-arguments are often given, and who gives them. Generally, experts in everything from Zoroastrian theology to theoretical time travel at least have this kind of expertise.

For example, Nick Bostrom has researched AGI impacts on and off for more than a decade, and has written extensively on the subject. Both in conversation and through his writings, Bostrom demonstrates pretty solid scholastic expertise in AGI impacts.

AI researchers, in contrast, do not tend to be familiar with the basic debates, arguments, and counterarguments related to AGI impacts. (Why would they be? That’s not their job.) Thus, it’s hard to see much value in, say, the projections about AGI impacts from the AAAI Presidential Panel on Long-Term AI Futures, which included no participants with known scholastic expertise in AGI impacts — and only one participant who is (barely) involved in the broader machine ethics community (Alan Mackworth).

But maybe we shouldn’t place any value on the opinions of those who do have scholastic expertise in the subject, either. Maybe those with scholastic expertise can’t reliably demonstrate superior performance on anything more practical than merely knowing which arguments and counterarguments are in play.

Ideally, AGI impacts experts and FAI experts should do more than demonstrate scholastic expertise. What other examples of RSPRT expertise should be relevant to both AGI impacts experts and FAI experts?

Sensitivity to evidence

In general, humans don’t accurately update their beliefs in response to medium-sized bits of evidence, like a perfectly rational agent would. That’s why we need science, where our method is to “amass such an enormous mountain of evidence* that… scientists cannot ignore it.”

But there usually aren’t “mountains” of evidence available when testing hypotheses about the design of future technologies and their likely impacts. As explained elsewhere: “The less evidence you have, or the harder it is to interpret, the more rationality you need to get the right answer. (As likelihood ratios get smaller, your priors need to be better and your updates more accurate.)”

Can human rationality be improved? Based on a couple decades of “debiasing” research (Larrick 2004), my guess is that we probably can, but we haven’t tried very hard yet.

Why think there is low-hanging fruit in the field of rationality training? Very few people, if any, put as much effort into improving their rationality as our best musicians and athletes put into improving their musical and athletic abilities. The best musicians practice 4 hours per day over many years (Ericsson et al. 1993); champion swimmer Michael Phelps spent 3-6 hours per day in the pool; Sun Microsystems co-founder Bill Joy practiced programming 10 hours per day in college (Gladwell 2008, p. 46); and during one period, chess champion Bobby Fischer reportedly practiced chess 14 hours a day. But who spends 4-10 hours per day doing calibration training or building up good rationality habits?

Ideally, both AGI impacts experts and Friendly AI experts would train good rationality habits so as to increase their sensitivity to evidence, so that they can reason productively about future technologies without first needing to amass (unavailable) “mountains of evidence.”

What might an FAI expert look like?

Next, let’s look at the specific skills needed for FAI expertise in particular. Clearly, such experts must be able to generate new results in math. And luckily, math research skill is more easily measurable and “objective” than, say, psychology or philosophy research skill.

What other kinds of expertise might we want in an FAI expert?

Yudkowsky described an FAI expert like this:

A Friendly AI [expert] is somebody who specializes in seeing the correspondence of mathematical structures to What Happens in the Real World. It’s somebody who looks at Hutter’s specification of AIXI and reads the actual equations… and sees, “Oh, this AI will try to gain control of its reward channel,” as well as numerous subtler issues like, “This AI presumes a Cartesian boundary separating itself from the environment; it may drop an anvil on its own head.” Similarly, working on TDT means e.g. looking at a mathematical specification of decision theory, and seeing “Oh, this is vulnerable to blackmail” and coming up with a mathematical counter-specification of an AI that isn’t so vulnerable to blackmail.

…If you want to have a sensible discussion about which AI designs are safer, there are specialized skills you can apply to that discussion, [such as the skill described above,] as built up over years of study and practice by someone who specializes in answering that sort of question.

Let me give some examples of people who, as Yudkowsky put it, “specialize in seeing the correspondence of mathematical structures to What Happens in the Real World.” (In particular, we’re interested in the consequences of mathematical objects with a kind of “general intelligence,” not so much the real world consequences of narrow-domain algorithms like Stuxnet.) To the extent that AGI behavior can be modeled with mathematics, this is a crucial skill.

Yudkowsky read Hutter’s specification of AIXI and saw “Oh, this AI will try to gain control of its reward channel” and “the AI presumes a Cartesian boundary separating itself from the environment; it may drop an anvil on its own head,” but he didn’t write down technical demonstrations of these facts.

That honor goes to Laurent Orseau (AgroParisTech) and Mark Ring (IDSIA), who discovered those problems independently and demonstrated them in Ring & Orseau (2011) and Orseau & Ring (2011). They also worked toward formalizing the latter problem in Orseau & Ring (2012), as did Bill Hibbard (University of Wisconsin) in Hibbard (2012).

Examples of this kind of work from MIRI’s research fellows or research associates include Dewey (2011), de Blanc (2011), and Yudkowsky (2010).

This skill may be difficult to measure objectively, but that is true of many of the skills that university administrators (or recruiters for hedge funds and technology companies) try to identify in mathematical researchers. And yet these groups have much success in locating the best and brightest. So perhaps there is some hope for identifying people with this skill.

There are other tasks on which FAI experts should demonstrate “reliably superior performance.” For example, they must be able to formalize philosophical concepts. Here again there is no standard measure for the skill, but we have many past examples from which to learn. The last century was a pretty productive one for turning previously mysterious philosophical concepts into formal ones. See Kolmogorov (1965) on complexity and simplicity, Solomonoff (1964a, 1964b) on induction, Von Neumann and Morgenstern (1947) on rationality, Shannon (1948) on information, and Tennenholtz’s development of “program equilibrium” (for an overview, see Wooldridge 2012).

Readers interested in developing Friendly AI expertise should consider taking the courses (or reading the textbooks) listed in Course Recommendations for MIRI Researchers.

What might an AGI impacts expert look like?

To begin, AGI impacts experts should demonstrate reliably superior performance at forecasting technological progress, especially AI progress.

Unfortunately, we haven’t yet discovered reliable methods for successful long-term technological forecasting (Muehlhauser & Salamon 2012), and both experts and laypeople are particularly bad at predicting AI (Armstrong & Sotala 2012). The price-performance formulation of Moore’s Law has been surprisingly robust across time, but one cannot predict specific technologies from this trend without making additional assumptions that are (in most cases) less robust than Moore’s Law. Famous tech forecaster Ray Kurzweil claims good accuracy, but these claims are probably overstated.

None of this should be surprising: good forecasting performance seems to depend (among other things) on regular feedback on one’s predictions, and quick feedback isn’t available when making long-term forecasts.

Luckily, there are many opportunities for forecasters to improve their performance. Horowitz & Tetlock (2012), based on their own empirical research and prediction training, offer some advice on the subject:

• Explicit quantification: “The best way to become a better-calibrated appraiser of long-term futures is to get in the habit of making quantitative probability estimates that can be objectively scored for accuracy over long stretches of time. Explicit quantification enables explicit accuracy feedback, which enables learning.”
• Signposting the future: Thinking through specific scenarios can be useful if those scenarios “come with clear diagnostic signposts that policymakers can use to gauge whether they are moving toward or away from one scenario or another… Falsifiable hypotheses bring high-flying scenario abstractions back to Earth.”
• Leveraging aggregation: “the average forecast is often more accurate than the vast majority of the individual forecasts that went into computing the average…. [Forecasters] should also get into the habit that some of the better forecasters in [an IARPA forecasting tournament called ACE] have gotten into: comparing their predictions to group averages, weighted-averaging algorithms, prediction markets, and financial markets.

Armstrong & Sotala (2012) add that it can be helpful to decompose the phenomena into many parts and make predictions about each of the parts, as feedback may be available for at least some of the parts. This is the approach to AI prediction taken by The Uncertain Future (Rayhawk et al. 2009).

Armstrong & Sotala also make a distinction between “grind” — lots of hard work and money — and “insight” — entirely new unexpected ideas. Grind is moderately easy to predict, while insight is difficult to forecast. Grind predictions could be more reliable to the extent that a phenomenon is mostly about grind, and doesn’t require new conceptual breakthroughs. For example, while AI appears to be an “insight” technology, whole brain emulation may be largely a “grind” technology, and therefore more easily predicted.

There is much more to say about the skills needed for AGI impacts expertise, but for now I leave the reader with the examples above, and also the following passage from Bostrom (1997):

[Recently] a starring role has developed on the intellectual stage for which the actor is still wanting. This is the role of the generalised scientist, or the polymath, who has insights into many areas of science and the ability to use these insights to work out solutions to those more complicated problems which are usually considered too difficult for scientists and are therefore either consigned to politicians and the popular press, or just ignored. The sad thing is that ignoring these problems won’t make them go away, and… some of them are challenges to the very survival of intelligent life.

[One such problem is] superintelligence… [which] takes on practical urgency when many experts think that we will soon have the ability to create superintelligence.

What questions could [this discipline] deal with? Well, questions like: How much would the predictive power for various fields increase if we increase the processing speed of a human-like mind a million times? If we extend the short-term or long-term memory? If we increase the neural population and the connection density? What other capacities would a superintelligence have? …Can we know anything about the motivation of a superintelligence? Would it be feasible to preprogram it to be good or philanthropic, or would such rules be hard to reconcile with the flexibility of its cognitive processes? Would a superintelligence, given the desire to do so, be able to outwit humans into promoting its own aims even if we had originally taken strict precautions to avoid being manipulated? Could one use one superintelligence to control another? …How would our human self-perception and aspirations change if were forced to abdicate the throne of wisdom…? How would we individuate between superminds if they could communicate and fuse and subdivide with enormous speed? Will a notion of personal identity still apply to such interconnected minds? …Could we then be able to compete with the superintelligences, if we were accelerated and augmented with extra memory etc., or would such profound reorganisation be necessary that we would no longer feel we were humans? Would that matter?

Maybe these are not the right questions to ask, but they are at least a start.

Concluding thoughts

MIRI exists entirely to host such experts and enable their research, and FHI and CSER share that focus among others. All three organizations are funding-limited, but to some degree they are also person-limited, because there are so few people in the world actively developing AGI impacts expertise or Friendly AI expertise. The goal of this post is to light the path for those who may want to contribute to this important research program.

Notes

My thanks to Carl Shulman, Kaj Sotala, Eliezer Yudkowsky, Louie Helm, and Benjamin Noble for their helpful comments.

The post AGI Impacts Experts and Friendly AI Experts appeared first on Machine Intelligence Research Institute.

I. J. Good’s thesis of the ‘intelligence explosion’ is that a sufficiently advanced machine intelligence could build a smarter version of itself, which could in turn build an even smarter version of itself, and that this process could continue enough to vastly exceed human intelligence. As Sandberg (2010) correctly notes, there are several attempts to lay down return-on-investment formulas intended to represent sharp speedups in economic or technological growth, but very little attempt has been made to deal formally with I. J. Good’s intelligence explosion thesis as such.

I identify the key issue as returns on cognitive reinvestment – the ability to invest more computing power, faster computers, or improved cognitive algorithms to yield cognitive labor which produces larger brains, faster brains, or better mind designs. There are many phenomena in the world which have been argued as evidentially relevant to this question, from the observed course of hominid evolution, to Moore’s Law, to the competence over time of machine chess-playing systems, and many more. I go into some depth on the sort of debates which then arise on how to interpret such evidence. I propose that the next step forward in analyzing positions on the intelligence explosion would be to formalize return-on-investment curves, so that each stance can say formally which possible microfoundations they hold to be falsified by historical observations already made. More generally, I pose multiple open questions of ‘returns on cognitive reinvestment’ or ‘intelligence explosion microeconomics’. Although such questions have received little attention thus far, they seem highly relevant to policy choices affecting the outcomes for Earth-originating intelligent life.

The preferred place for public discussion of this research is here. There is also a private mailing list for technical discussants, which you can apply to join here.

The post “Intelligence Explosion Microeconomics” Released appeared first on Machine Intelligence Research Institute.

The book contains 20 chapters about the prospect of machine superintelligence, including 4 chapters by MIRI researchers and research associates.

“Intelligence Explosion: Evidence and Import” (pdf) by Luke Muehlhauser and (previous MIRI researcher) Anna Salamon reviews

the evidence for and against three claims: that (1) there is a substantial chance we will create human-level AI before 2100, that (2) if human-level AI is created, there is a good chance vastly superhuman AI will follow via an “intelligence explosion,” and that (3) an uncontrolled intelligence explosion could destroy everything we value, but a controlled intelligence explosion would benefit humanity enormously if we can achieve it. We conclude with recommendations for increasing the odds of a controlled intelligence explosion relative to an uncontrolled intelligence explosion.

“Intelligence Explosion and Machine Ethics” (pdf) by Luke Muehlhauser and Louie Helm discusses the challenges of formal value systems for use in AI:

Many researchers have argued that a self-improving artificial intelligence (AI) could become so vastly more powerful than humans that we would not be able to stop it from achieving its goals. If so, and if the AI’s goals differ from ours, then this could be disastrous for humans. One proposed solution is to program the AI’s goal system to want what we want before the AI self-improves beyond our capacity to control it. Unfortunately, it is difficult to specify what we want. After clarifying what we mean by “intelligence,” we offer a series of “intuition pumps” from the field of moral philosophy for our conclusion that human values are complex and difficult to specify. We then survey the evidence from the psychology of motivation, moral psychology, and neuroeconomics that supports our position. We conclude by recommending ideal preference theories of value as a promising approach for developing a machine ethics suitable for navigating an intelligence explosion or “technological singularity.”

“Friendly Artificial Intelligence” by Eliezer Yudkowsky is a shortened version of Yudkowsky (2008).

Finally, “Artificial General Intelligence and the Human Mental Model” (pdf) by Roman Yampolskiy and (MIRI research associate) Joshua Fox  reviews the dangers of anthropomorphizing machine intelligences:

When the first artificial general intelligences are built, they may improve themselves to far-above-human levels. Speculations about such future entities are already affected by anthropomorphic bias, which leads to erroneous analogies with human minds. In this chapter, we apply a goal-oriented understanding of intelligence to show that humanity occupies only a tiny portion of the design space of possible minds. This space is much larger than what we are familiar with from the human example; and the mental architectures and goals of future superintelligences need not have most of the properties of human minds. A new approach to cognitive science and philosophy of mind, one not centered on the human example, is needed to help us understand the challenges which we will face when a power greater than us emerges.

The book also includes brief, critical responses to most chapters, including responses written by Eliezer Yudkowsky and (previous MIRI staffer) Michael Anissimov.

The post “Singularity Hypotheses” Published appeared first on Machine Intelligence Research Institute.