All MIRI Publications
Articles
Journal Article Conference Paper Book Chapter Technical Report/Working Paper
Recent publications
Nate Soares and Benja Fallenstein. 2015. “Aligning Superintelligence with Human Interests: A Technical Research Agenda.” Forthcoming in The Technological Singularity: Managing the Journey. Springer. Previously published as technical report 2014–8.
Benja Fallenstein, Jessica Taylor, and Paul Christiano. 2015. “Reflective Oracles: A Foundation for Classical Game Theory.” Paper accepted to LORI-V. Previously published as technical report 2015–8.
Katja Grace. 2015. “The Asilomar Conference: A Case Study in Risk Mitigation.” Technical report 2015–9. MIRI.Nate Soares and Benja Fallenstein. 2015. “Two Attempts to Formalize Counterpossible Reasoning in Deterministic Settings.” Paper accepted to AGI 2015.Benja Fallenstein, Nate Soares, and Jessica Taylor. 2015. “Reflective Variants of Solomonoff Induction and AIXI.” Paper accepted to AGI 2015. Previously published as technical report 2015–7.
2015
Benja Fallenstein and Nate Soares. 2015. “Vingean Reflection: Reliable Reasoning for Self-Improving Agents.” Technical report 2015–2. MIRI.Patrick LaVictoire. 2015. “An Introduction to Löb’s Theorem in MIRI Research.” Technical report 2015–6. MIRI.Nate Soares. 2015. “Aligning Superintelligence with Human Interests: An Annotated Bibliography.” Technical report 2015–5. MIRI.Nate Soares. 2015. “Formalizing Two Problems of Realistic World-Models.” Technical report 2015–3. MIRI.Nate Soares. 2015. “The Value Learning Problem.” Technical report 2015–4. MIRI.Nate Soares and Benja Fallenstein. 2015. “Questions of Reasoning under Logical Uncertainty.” Technical report 2015–1. MIRI.Nate Soares and Benja Fallenstein. 2015. “Toward Idealized Decision Theory.” arXiv:1507.01986. Previously published as technical report 2014–7.Nate Soares, Benja Fallenstein, Eliezer Yudkowsky, and Stuart Armstrong. 2015. “Corrigibility.” Paper presented at AAAI 2015. Previously published as technical report 2014–6.Kaj Sotala. 2015. “Concept Learning for Safe Autonomous AI.” Paper presented at AAAI 2015.
2014
Stuart Armstrong, Kaj Sotala, and Seán Ó hÉigeartaigh. 2014. “The Errors, Insights and Lessons of Famous AI Predictions – and What They Mean for the Future.” Journal of Experimental & Theoretical Artificial Intelligence 26 (3): 317–342. 
Mihály Bárász, Paul Christiano, Benja Fallenstein, Marcello Herreshoff, Patrick LaVictoire, and Eliezer Yudkowsky. 2014. “Robust Cooperation on the Prisoner’s Dilemma: Program Equilibrium via Provability Logic.” arXiv:1401.5577 [cs.GT].Tsvi Benson-Tilsen. 2014. “UDT with Known Search Order.” Technical report 2014–4. MIRI.Nick Bostrom and Eliezer Yudkowsky. 2014. “The Ethics of Artificial Intelligence.” In The Cambridge Handbook of Artificial Intelligence. Cambridge University Press.Paul Christiano. 2014. “Non-Omniscience, Probabilistic Inference, and Metamathematics.” Technical report 2014–3. MIRI.Benja Fallenstein. 2014. “Procrastination in Probabilistic Logic.” Working paper.
Benja Fallenstein and Nate Soares. 2014. “Problems of Self-Reference in Self-Improving Space-Time Embedded Intelligence.” In Proceedings of AGI 2014. Springer.Benja Fallenstein and Nisan Stiennon. 2014. “‘Loudness’: On Priors over Preference Relations.” Brief technical note.Patrick Lavictoire, Benja Fallenstein, Eliezer Yudkowsky, Mihály Bárász, Paul Christiano and Marcello Herreshoff. 2014. “Program Equilibrium in the Prisoner’s Dilemma via Löb’s Theorem.” Paper presented at AAAI 2014.Luke Muehlhauser and Nick Bostrom. 2014. “Why We Need Friendly AI.” Think 13 (36): 42–47.Luke Muehlhauser and Bill Hibbard. 2014. “Exploratory Engineering in AI.” Communications of the ACM 57 (9): 32–34. Carl Shulman and Nick Bostrom. 2014. “Embryo Selection for Cognitive Enhancement: Curiosity or Game-Changer?” Global Policy 5 (1): 85–92.Nate Soares. 2014. “Tiling Agents in Causal Graphs.” Technical report 2014–5. MIRI.Nate Soares and Benja Fallenstein. 2014. “Botworld 1.0.” Technical report 2014–2. MIRI.Eliezer Yudkowsky. 2014. “Distributions Allowing Tiling of Staged Subjective EU Maximizers.” Technical report 2014–1. MIRI.
2013
Alex Altair. 2013. “A Comparison of Decision Algorithms on Newcomblike Problems.” Working paper. MIRI.Stuart Armstrong, Nick Bostrom, and Carl Shulman. 2013. “Racing to the Precipice: A Model of Artificial Intelligence Development.” Technical report 2013–1. Oxford, UK: Future of Humanity Institute.
Paul Christiano, Eliezer Yudkowsky, Marcello Herreshoff, and Mihály Bárász. 2013. “Definability of “Truth” in Probabilistic Logic.” Draft. MIRI.Benja Fallenstein. 2013. “The 5-and-10 Problem and the Tiling Agents Formalism.” Technical report 2013–9. MIRI.Benja Fallenstein. 2013. “Decreasing Mathematical Strength in One Formalization of Parametric Polymorphism.” Brief technical note. MIRI.Benja Fallenstein. 2013. “An Infinitely Descending Sequence of Sound Theories Each Proving the Next Consistent.” Technical report 2013–6. MIRI.Benja Fallenstein and Alex Mennen. 2013. “Predicting AGI: What Can We Say When We Know So Little?” Working paper. MIRI.Katja Grace. 2013. “Algorithmic Progress in Six Domains.” Technical report 2013–3. MIRI.Jeremy Hahn. 2013. “Scientific Induction in Probabilistic Metamathematics.” Technical report 2013–4. MIRI.
Luke Muehlhauser and Louie Helm. 2013. “Intelligence Explosion and Machine Ethics.” In Singularity Hypotheses. Springer.
Luke Muehlhauser and Anna Salamon. 2013. “Intelligence Explosion: Evidence and Import.” In Singularity Hypotheses. Springer. (Español) (Français) (Italiano)Luke Muehlhauser and Chris Williamson. 2013. “Ideal Advisor Theories and Personal CEV.” Working paper. MIRI.Will Sawin and Abram Demski. 2013. “Computable Probability Distributions Which Converge on Believing True Π1 Sentences Will Disbelieve True Π2 Sentences.” Technical Report 2013–10. MIRI.Nate Soares. 2013. “Fallenstein’s Monster.” Technical report 2013–7. MIRI.Kaj Sotala and Roman Yampolskiy. 2013. “Responses to Catastrophic AGI Risk: A Survey.” Technical report 2013–2. MIRI.Nisan Stiennon. 2013. “Recursively-Defined Logical Theories Are Well-Defined.” Technical report 2013–8. MIRI.Roman Yampolskiy and Joshua Fox. 2013. “Artificial General Intelligence and the Human Mental Model.” In Singularity Hypotheses. Springer.Roman Yampolskiy and Joshua Fox. 2013. “Safety Engineering for Artificial General Intelligence.” Topoi 32 (2): 217–226.
Eliezer Yudkowsky. 2013. “Intelligence Explosion Microeconomics.” Technical report 2013–1. MIRI. Eliezer Yudkowsky. 2013. “The Procrastination Paradox.” Brief technical note. MIRI.Eliezer Yudkowsky and Marcello Herreshoff. 2013. “Tiling Agents for Self-Modifying AI, and the Löbian Obstacle.” Draft. MIRI.
2012
Stuart Armstrong and Kaj Sotala. 2012. “How We’re Predicting AI – or Failing To.” In Beyond AI: Artificial Dreams. Pilsen: University of West Bohemia.Bill Hibbard. 2012. “Avoiding Unintended AI Behaviors.” In Proceedings of AGI 2012. Springer.Bill Hibbard. 2012. “Decision Support for Safe AI Design.” In Proceedings of AGI 2012. Springer.Luke Muehlhauser. 2012. “AI Risk Bibliography 2012.” Working paper. MIRI.Anna Salamon and Luke Muehlhauser. 2012. “Singularity Summit 2011 Workshop Report.” Working paper. MIRI.Carl Shulman and Nick Bostrom. 2012. “How Hard Is Artificial Intelligence? Evolutionary Arguments and Selection Effects.” Journal of Consciousness Studies 19 (7–8): 103–130.Kaj Sotala. 2012. “Advantages of Artificial Intelligences, Uploads, and Digital Minds.” International Journal of Machine Consciousness 4 (1): 275-291.Kaj Sotala and Harri Valpola. 2012. “Coalescing Minds: Brain Uploading-Related Group Mind Scenarios.” International Journal of Machine Consciousness 4 (1): 293–312.
2011
Peter de Blanc. 2011. “Ontological Crises in Artificial Agents’ Value Systems.” Working paper. MIRI.Daniel Dewey. 2011. “Learning What to Value.” In Proceedings of AGI 2011. Springer.Eliezer Yudkowsky. 2011. “Complex Value Systems Are Required to Realize Valuable Futures.” In Proceedings of AGI 2011. Springer.
2010
Joshua Fox and Carl Shulman. 2010. “Superintelligence Does Not Imply Benevolence.” In Proceedings of ECAP 2010. Verlag Dr. Hut.Steven Kaas, Steve Rayhawk, Anna Salamon, and Peter Salamon. 2010. “Economic Implications of Software Minds.” Working paper. MIRI.Anna Salamon, Steve Rayhawk, and János Kramár. 2010. “How Intelligible Is Intelligence?” In Proceedings of ECAP 2010. Verlag Dr. Hut.Carl Shulman. 2010. “Omohundro’s ‘Basic AI Drives’ and Catastrophic Risks.” Working paper. MIRI.Carl Shulman. 2010. “Whole Brain Emulation and the Evolution of Superorganisms.” Working paper. MIRI.Carl Shulman and Anders Sandberg. 2010. “Implications of a Software-Limited Singularity.” In Proceedings of ECAP 2010. Verlag Dr. Hut.Kaj Sotala. 2010. “From Mostly Harmless to Civilization-Threatening.” In Proceedings of ECAP 2010. Verlag Dr. Hut.Nick Tarleton. 2010. “Coherent Extrapolated Volition: A Meta-Level Approach to Machine Ethics.” Working paper. MIRI.Eliezer Yudkowsky. 2010. “Timeless Decision Theory.” Working paper. MIRI.Eliezer Yudkowsky, Carl Shulman, Anna Salamon, Rolf Nelson, Steven Kaas, Steve Rayhawk, and Tom McCabe. 2010. “Reducing Long-Term Catastrophic Risks from Artificial Intelligence.” Working paper. MIRI.
2001-2009
Peter de Blanc. 2009. “Convergence of Expected Utility for Universal Artificial Intelligence.” Working paper. MIRI.Steve Rayhawk, Anna Salamon, Michael Anissimov, Thomas McCabe, and Rolf Nelson. 2009. “Changing the Frame of AI Futurism: From Storytelling to Heavy-Tailed, High-Dimensional Probability Distributions.” Paper presented at ECAP 2009.Carl Shulman and Stuart Armstrong. 2009. “Arms Control and Intelligence Explosions.” Paper presented at ECAP 2009.Carl Shulman, Henrik Jonsson, and Nick Tarleton. 2009. “Machine Ethics and Superintelligence.” In Proceedings of AP-CAP 2009. University of Tokyo.Carl Shulman, Nick Tarleton, and Henrik Jonsson. 2009. “Which Consequentialism? Machine Ethics and Moral Divergence.” In Proceedings of AP-CAP 2009. University of Tokyo.Eliezer Yudkowsky. 2008. “Cognitive Biases Potentially Affecting Judgement of Global Risks.” In Global Catastrophic Risks. Oxford University Press. (Italiano) (Pу́сский) (Portuguese)Eliezer Yudkowsky. 2007. “Levels of Organization in General Intelligence.” In
Artificial General Intelligence (Cognitive Technologies). Springer.Eliezer Yudkowsky. 2004. “Coherent Extrapolated Volition.” Working paper. MIRI.Eliezer Yudkowsky. 2001. “Creating Friendly AI 1.0: The Analysis and Design of Benevolent Goal Architectures.” Working paper. MIRI. Books
Rationality: From AI to Zombies
Eliezer Yudkowsky (2015)
What does it actually mean to be rational? Not Hollywood-style “rational,” where one rejects all human feeling to embrace Cold Hard Logic — real rationality, of the sort studied by psychologists, social scientists, and mathematicians. The kind of rationality where you make good decisions, even when it’s hard; where you reason well, even in the face of massive uncertainty.
In Rationality: From AI to Zombies, Eliezer Yudkowsky explains the findings of cognitive science, and the ideas of naturalistic philosophy, which help to motivate why MIRI’s research program exists.
Smarter Than Us: The Rise of Machine Intelligence
Stuart Armstrong (2014)
What happens when machines become smarter than humans? Humans steer the future not because we’re the strongest or the fastest but because we’re the smartest. When machines become smarter than humans, we’ll be handing them the steering wheel. What promises—and perils—will these powerful machines present? Stuart Armstrong’s new book navigates these questions with clarity and wit.
Facing the Intelligence Explosion
Luke Muehlhauser (2013)
Sometime this century, machines will surpass human levels of intelligence and ability. This event—the “intelligence explosion”—will be the most important event in our history, and navigating it wisely will be the most important thing we can ever do.
Luminaries from Alan Turing and I. J. Good to Bill Joy and Stephen Hawking have warned us about this. Why do we think Hawking and company are right, and what can we do about it?
Facing the Intelligence Explosion is Muehlhauser’s attempt to answer these questions.
The Hanson-Yudkowsky AI-Foom Debate
Robin Hanson and Eliezer Yudkowsky (2013)
In late 2008, economist Robin Hanson and AI theorist Eliezer Yudkowsky conducted an online debate about the future of artificial intelligence, and in particular about whether generally intelligent AIs will be able to improve their own capabilities very quickly (a.k.a. “foom”). James Miller and Carl Shulman also contributed guest posts to the debate.
The original debate took place in a long series of blog posts, which are collected here. This book also includes a transcript of a 2011 in-person debate between Hanson and Yudkowsky on this subject, a summary of the debate written by Kaj Sotala, and a 2013 technical report on AI takeoff dynamics (“intelligence explosion microeconomics”) written by Yudkowsky.
Other
MIRI’s Blog
- Analyses – MIRI publishes some of its most substantive research to its blog.
- Conversations – MIRI interviews a diverse array of researchers and intellectuals on topics related to its research.
Resources for Researchers
- A public BibLaTeX file.
- A public Mendeley group.
- A public GitHub repo.
