“Singularity Hypotheses” Published

 |   |  Papers

singularity hypothesesSingularity Hypotheses: A Scientific and Philosophical Assessment has now been published by Springer, in hardcover and ebook forms.

The book contains 20 chapters about the prospect of machine superintelligence, including 4 chapters by MIRI researchers and research associates.

“Intelligence Explosion: Evidence and Import” (pdf) by Luke Muehlhauser and (previous MIRI researcher) Anna Salamon reviews

the evidence for and against three claims: that (1) there is a substantial chance we will create human-level AI before 2100, that (2) if human-level AI is created, there is a good chance vastly superhuman AI will follow via an “intelligence explosion,” and that (3) an uncontrolled intelligence explosion could destroy everything we value, but a controlled intelligence explosion would benefit humanity enormously if we can achieve it. We conclude with recommendations for increasing the odds of a controlled intelligence explosion relative to an uncontrolled intelligence explosion.

“Intelligence Explosion and Machine Ethics” (pdf) by Luke Muehlhauser and Louie Helm discusses the challenges of formal value systems for use in AI:

Many researchers have argued that a self-improving artificial intelligence (AI) could become so vastly more powerful than humans that we would not be able to stop it from achieving its goals. If so, and if the AI’s goals differ from ours, then this could be disastrous for humans. One proposed solution is to program the AI’s goal system to want what we want before the AI self-improves beyond our capacity to control it. Unfortunately, it is difficult to specify what we want. After clarifying what we mean by “intelligence,” we offer a series of “intuition pumps” from the field of moral philosophy for our conclusion that human values are complex and difficult to specify. We then survey the evidence from the psychology of motivation, moral psychology, and neuroeconomics that supports our position. We conclude by recommending ideal preference theories of value as a promising approach for developing a machine ethics suitable for navigating an intelligence explosion or “technological singularity.”

“Friendly Artificial Intelligence” by Eliezer Yudkowsky is a shortened version of Yudkowsky (2008).

Finally, “Artificial General Intelligence and the Human Mental Model” (pdf) by Roman Yampolskiy and (MIRI research associate) Joshua Fox  reviews the dangers of anthropomorphizing machine intelligences:

When the first artificial general intelligences are built, they may improve themselves to far-above-human levels. Speculations about such future entities are already affected by anthropomorphic bias, which leads to erroneous analogies with human minds. In this chapter, we apply a goal-oriented understanding of intelligence to show that humanity occupies only a tiny portion of the design space of possible minds. This space is much larger than what we are familiar with from the human example; and the mental architectures and goals of future superintelligences need not have most of the properties of human minds. A new approach to cognitive science and philosophy of mind, one not centered on the human example, is needed to help us understand the challenges which we will face when a power greater than us emerges.

The book also includes brief, critical responses to most chapters, including responses written by Eliezer Yudkowsky and (previous MIRI staffer) Michael Anissimov.

  • paul

    quick comment about one of the things i notice a lot in talking about singularity on this site.

    “if the AI’s goals differ from ours, then this could be disastrous for humans”.

    My comment is that it seems odd that we are using human goals as the correct goal. i may be missing something but it seems often to be talked about in this way. It seems odd that we are talking about human goals and not about goals of those creatures that matter (to the extent that they matter, and this is ever changing, and must be interpreted by a mind or group of minds). Right now it is ok to use this as a general rule but i would hate for an A.I. to think that human goals are important over other ways of looking at goals. It seems if A.I. intelligence continues it may likely find other things that are much more important than human goals (not referring to god) in the future that either exist or come into existence. Granted i see the reason to use the concept of human goals for marketing and for a general guideline to help people understand the project but i can only imagine if a creature was created or born with a much higher amount of value experiencing ability (say 500 billion times the highest goals of humans) we would want to maximize that or at least not think it was more important than something so much smaller

    just figured i would say this as it has been bothering me about the way some of the ideas on this site are talked about.

    I am a huge fan of Luke and the great podcasts and writing he has done! thanks for all you work Luke. CFAPBD podcast was one of my all time favorite podcasts. – Paul

    • Luke Muehlhauser

      Thanks for the kind words, Paul.

      In fact, we don’t think human goals are the right goals for a machine superintelligence, but this brings up lots of tricky issues in metaethics and language. For more detail, please see section 6 of my paper with Louie Helm, Intelligence Explosion and Machine Ethics.

      • PAUL

        thanks for the link Luke.

        I understand why you are saying that current human goals are not what we want but we want “what one’s values would be under more ideal circumstances (e.g. of full information, value coherence)”. If you are thinking of full information and value coherence about all creatures that have goals i think i do agree. I just do not see a reason to ultimately care about human goals (even when they are optimized with full information, value coherence) when thinking about all the mind designs that maybe taken into consideration in the future by a the “seed AI”. I can easily imagine a thought experiment where it would be better to kill off all humans if that gave another creature that had a much higher amount of experiencing ability of value some amount of pleasure greater than what is lost by humans. In this case human values are taken into consideration but are not worth optimizing for as something else has much more important or stronger goals or value experianceing ability. You may be taking this into account with your “full information, value coherence” and if so it think this is a great way of looking at it … but humans are not what are important, we just happen to be really important at this point in time (as far as i know).

        I think i have seen the idea of human values used other places on this site. i understand this in comparison to an AI optimized to make paper clips and has that as a goal but experiences no value and takes a way value from humans in its goal to find all paper clip materials. This is clearly not “value coherence”. i am thinking other mind designs will become important (have stronger goals and ability to experiance value) over time and i would hate for the seed AI to care about humans at the expense of something much more important.

        • Luke Muehlhauser

          I think the concern you’re expressing is correct. We usually try to say “humane” values, which is nicely vague, in a way that reflects the complexities here. Would the values of agents converge under any reasonable extrapolation procedure? What could make one extrapolation procedure more reasonable than another? How come none of the options in population ethics look remotely good (see: Arrhenius)? Given this normative uncertainty, what should we do (see: Crouch, Sepielli)? These are all open questions, and MIRI doesn’t pretend to have them answered. The normative uncertainty part of all this looks the most tractable to current humans — if you’re interested, I’d start with Crouch’s bachelor’s dissertation, which you should be able to Google.

  • paul

    Luke, I feel that i have an answere to the population ethics or repugnant conclusion and the normative ethics. Well, the keyword is probably “feel” as i know that if others have not figured it out i probably did not. This is a relatively new problem for me so i have not had as much time to think about it as others and may be missing some main aspects. I had to look up “population ethics” this weekend and was able to learn a bit about the repugnant conclusion. (from what i can tell these are the same thing).

    It seems as if in the repugnant conclusion you can be fixed if we admit that we are making guesses about how we add the quantify life.

    1. Our intuition that low quality life at large amounts is worse could be wrong. in this case it is better to have lots of low quality life and we are just instinctively bias.

    2. we may be missing that the “whole is greater then the sum of its parts” and that addition is not the only way of looking at it or the correct way to get value.

    I would look at guessing the quality of life and how they interact with each other as a clue and not “the end all” in understanding value in a population.

    Here is an example of how i think a seed AI should make guesses about the future. This has images so i made a quick webpage. http://enjoy2.net/pop.html

    I still have a lot to learn on the subject. thanks for the resources you sent. I have looked at parts of some them but still have a lot more to read.