All MIRI Publications
Articles
Journal Article Conference/Workshop Paper Book Chapter Technical Report/Working Paper
Recent and Forthcoming Publications
Nate Soares and Ben Levinstein. Forthcoming. “Cheating Death in Damascus.” Paper accepted to the 14th Annual Formal Epistemology Workshop.
Andrew Critch. 2017. “Toward Negotiable Reinforcement Learning: Shifting Priorities in Pareto Optimal Sequential Decision-Making.” arXiv:1701.01302 [cs.AI].Vadim Kosoy. 2016. “Optimal Polynomial-Time Estimators: A Bayesian Notion of Approximation Algorithm.” arXiv:1608.04112 [cs.CC].Scott Garrabrant, Tsvi Benson-Tilsen, Andrew Critch, Nate Soares, and Jessica Taylor. 2016. “Logical Induction (Abridged).” MIRI technical report 2016–2.
Nate Soares and Benya Fallenstein. Forthcoming. “Agent Foundations for Aligning Machine Intelligence with Human Interests: A Technical Research Agenda.” Forthcoming in The Technological Singularity: Managing the Journey. Springer. Previously published as MIRI technical report 2014–8 under the name “Aligning Superintelligence with Human Interests: A Technical Research Agenda.”
2016
Tsvi Benson-Tilsen and Nate Soares. 2016. “Formalizing Convergent Instrumental Goals.” Paper presented at the AAAI 2016 AI, Ethics and Society Workshop.Andrew Critch. 2016. “Parametric Bounded Löb’s Theorem and Robust Cooperation of Bounded Agents.” arXiv:1602.04184 [cs:GT].Scott Garrabrant, Tsvi Benson-Tilsen, Andrew Critch, Nate Soares, and Jessica Taylor. 2016. “Logical Induction.” arXiv:1609.03543 [cs.AI].Scott Garrabrant, Siddharth Bhaskar, Abram Demski, Joanna Garrabrant, George Koleszarik, and Evan Lloyd. 2016. “Asymptotic Logical Uncertainty and The Benford Test.” arXiv:1510.03370 [cs.LG]. Paper presented at the Ninth Conference on Artificial General Intelligence. Previously published as MIRI technical report 2015–11.Scott Garrabrant, Benya Fallenstein, Abram Demski, and Nate Soares. 2016. “Inductive Coherence.” arXiv:1604.05288 [cs:AI]. Previously published as “Uniform Coherence.”Scott Garrabrant, Nate Soares, and Jessica Taylor. 2016. “Asymptotic Convergence in Online Learning with Unbounded Delays.” arXiv:1604.05280 [cs:LG].Jan Leike, Jessica Taylor, and Benya Fallenstein. 2016. “A Formal Solution to the Grain of Truth Problem.” Paper presented at the 32nd Conference on Uncertainty in Artificial Intelligence.Laurent Orseau and Stuart Armstrong. 2016. “Safely Interruptible Agents.” Paper presented at the 32nd Conference on Uncertainty in Artificial Intelligence.Nate Soares. 2016. “The Value Learning Problem.” Paper presented at the IJCAI 2016 Ethics for Artificial Intelligence workshop. Previously published as MIRI technical report 2015–4.Kaj Sotala. 2016. “Defining Human Values for Value Learners.” Paper presented at the AAAI 2016 AI, Ethics and Society Workshop.Jessica Taylor. 2016. “Quantilizers: A Safer Alternative to Maximizers for Limited Optimization.” Paper presented at the AAAI 2016 AI, Ethics and Society Workshop.Jessica Taylor, Eliezer Yudkowsky, Patrick LaVictoire, and Andrew Critch. 2016. “Alignment for Advanced Machine Learning Systems.” MIRI technical report 2016–1.
2015
Stuart Armstrong, Nick Bostrom, and Carl Shulman. 2015. “Racing to the Precipice: A Model of Artificial Intelligence Development.” AI & Society (DOI 10.1007/s00146-015-0590-7): 1-6. Previously published as Future of Humanity Institute technical report 2013–1.Benja Fallenstein and Ramana Kumar. 2015. “Proof-Producing Reflection for HOL: With an Application to Model Polymorphism.” In Interactive Theorem Proving: 6th International Conference, ITP 2015, Nanjing, China, August 24-27, 2015, Proceedings. Springer.Benja Fallenstein and Nate Soares. 2015. “Vingean Reflection: Reliable Reasoning for Self-Improving Agents.” MIRI technical report 2015–2.Benja Fallenstein, Nate Soares, and Jessica Taylor. 2015. “Reflective Variants of Solomonoff Induction and AIXI.” In Proceedings of AGI 2015. Springer. Previously published as MIRI technical report 2015–8.Benja Fallenstein, Jessica Taylor, and Paul Christiano. 2015. “Reflective Oracles: A Foundation for Classical Game Theory.” arXiv:1508.04145 [cs.AI]. Previously published as MIRI technical report 2015–7. Published in abridged form as “Reflective Oracles: A Foundation for Game Theory in Artificial Intelligence” in Proceedings of LORI 2015.Katja Grace. 2015. “The Asilomar Conference: A Case Study in Risk Mitigation.” MIRI technical report 2015–9.Katja Grace. 2015. “Leó Szilárd and the Danger of Nuclear Weapons: A Case Study in Risk Mitigation.” MIRI technical report 2015–10.Patrick LaVictoire. 2015. “An Introduction to Löb’s Theorem in MIRI Research.” MIRI technical report 2015–6.Nate Soares. 2015. “Aligning Superintelligence with Human Interests: An Annotated Bibliography.” MIRI technical report 2015–5.Nate Soares. 2015. “Formalizing Two Problems of Realistic World-Models.” MIRI technical report 2015–3.Nate Soares and Benja Fallenstein. 2015. “Questions of Reasoning under Logical Uncertainty.” MIRI technical report 2015–1.Nate Soares and Benja Fallenstein. 2015. “Toward Idealized Decision Theory.” arXiv:1507.01986 [cs.AI]. Previously published as MIRI technical report 2014–7. Published in abridged form as “Two Attempts to Formalize Counterpossible Reasoning in Deterministic Settings” in Proceedings of AGI 2015.Nate Soares, Benja Fallenstein, Eliezer Yudkowsky, and Stuart Armstrong. 2015. “Corrigibility.” Paper presented at the AAAI 2015 Ethics and Artificial Intelligence Workshop. Previously published as MIRI technical report 2014–6.Kaj Sotala. 2015. “Concept Learning for Safe Autonomous AI.” Paper presented at the AAAI 2015 Ethics and Artificial Intelligence Workshop.
2014
Stuart Armstrong, Kaj Sotala, and Seán Ó hÉigeartaigh. 2014. “The Errors, Insights and Lessons of Famous AI Predictions – and What They Mean for the Future.” Journal of Experimental & Theoretical Artificial Intelligence 26 (3): 317–342. 
Mihály Bárász, Paul Christiano, Benja Fallenstein, Marcello Herreshoff, Patrick LaVictoire, and Eliezer Yudkowsky. 2014. “Robust Cooperation on the Prisoner’s Dilemma: Program Equilibrium via Provability Logic.” arXiv:1401.5577 [cs.GT].Tsvi Benson-Tilsen. 2014. “UDT with Known Search Order.” MIRI technical report 2014–4.Nick Bostrom and Eliezer Yudkowsky. 2014. “The Ethics of Artificial Intelligence.” In The Cambridge Handbook of Artificial Intelligence. Cambridge University Press.Paul Christiano. 2014. “Non-Omniscience, Probabilistic Inference, and Metamathematics.” MIRI technical report 2014–3.Benja Fallenstein. 2014. “Procrastination in Probabilistic Logic.” Working paper.
Benja Fallenstein and Nate Soares. 2014. “Problems of Self-Reference in Self-Improving Space-Time Embedded Intelligence.” In Proceedings of AGI 2014. Springer.Benja Fallenstein and Nisan Stiennon. 2014. “‘Loudness’: On Priors over Preference Relations.” Brief technical note.Patrick LaVictoire, Benja Fallenstein, Eliezer Yudkowsky, Mihály Bárász, Paul Christiano and Marcello Herreshoff. 2014. “Program Equilibrium in the Prisoner’s Dilemma via Löb’s Theorem.” Paper presented at the AAAI 2014 Multiagent Interaction without Prior Coordination Workshop.Luke Muehlhauser and Nick Bostrom. 2014. “Why We Need Friendly AI.” Think 13 (36): 42–47.Luke Muehlhauser and Bill Hibbard. 2014. “Exploratory Engineering in AI.” Communications of the ACM 57 (9): 32–34. Carl Shulman and Nick Bostrom. 2014. “Embryo Selection for Cognitive Enhancement: Curiosity or Game-Changer?” Global Policy 5 (1): 85–92.Nate Soares. 2014. “Tiling Agents in Causal Graphs.” MIRI technical report 2014–5.Nate Soares and Benja Fallenstein. 2014. “Botworld 1.1.” MIRI technical report 2014–2.Kaj Sotala and Roman Yampolskiy. 2014. “Responses to Catastrophic AGI Risk: A Survey.” Physica Scripta 90 (1): 1-33. Previously published as MIRI technical report 2013–2.Eliezer Yudkowsky. 2014. “Distributions Allowing Tiling of Staged Subjective EU Maximizers.” MIRI technical report 2014–1.
2013
Alex Altair. 2013. “A Comparison of Decision Algorithms on Newcomblike Problems.” Working paper. MIRI.
Paul Christiano, Eliezer Yudkowsky, Marcello Herreshoff, and Mihály Bárász. 2013. “Definability of “Truth” in Probabilistic Logic.” Draft. MIRI.Benja Fallenstein. 2013. “The 5-and-10 Problem and the Tiling Agents Formalism.” MIRI technical report 2013–9.Benja Fallenstein. 2013. “Decreasing Mathematical Strength in One Formalization of Parametric Polymorphism.” Brief technical note. MIRI.Benja Fallenstein. 2013. “An Infinitely Descending Sequence of Sound Theories Each Proving the Next Consistent.” MIRI technical report 2013–6.Benja Fallenstein and Alex Mennen. 2013. “Predicting AGI: What Can We Say When We Know So Little?” Working paper. MIRI.Katja Grace. 2013. “Algorithmic Progress in Six Domains.” MIRI technical report 2013–3.Jeremy Hahn. 2013. “Scientific Induction in Probabilistic Metamathematics.” MIRI technical report 2013–4.
Luke Muehlhauser and Louie Helm. 2013. “Intelligence Explosion and Machine Ethics.” In Singularity Hypotheses. Springer.
Luke Muehlhauser and Anna Salamon. 2013. “Intelligence Explosion: Evidence and Import.” In Singularity Hypotheses. Springer. (Español) (Français) (Italiano)Luke Muehlhauser and Chris Williamson. 2013. “Ideal Advisor Theories and Personal CEV.” Working paper. MIRI.Will Sawin and Abram Demski. 2013. “Computable Probability Distributions Which Converge on Believing True Π1 Sentences Will Disbelieve True Π2 Sentences.” MIRI technical Report 2013–10.Nate Soares. 2013. “Fallenstein’s Monster.” MIRI technical report 2013–7.Nisan Stiennon. 2013. “Recursively-Defined Logical Theories Are Well-Defined.” MIRI technical report 2013–8.Roman Yampolskiy and Joshua Fox. 2013. “Artificial General Intelligence and the Human Mental Model.” In Singularity Hypotheses. Springer.Roman Yampolskiy and Joshua Fox. 2013. “Safety Engineering for Artificial General Intelligence.” Topoi 32 (2): 217–226.
Eliezer Yudkowsky. 2013. “Intelligence Explosion Microeconomics.” MIRI technical report 2013–1.Eliezer Yudkowsky. 2013. “The Procrastination Paradox.” Brief technical note. MIRI.Eliezer Yudkowsky and Marcello Herreshoff. 2013. “Tiling Agents for Self-Modifying AI, and the Löbian Obstacle.” Draft. MIRI.
2012
Stuart Armstrong and Kaj Sotala. 2012. “How We’re Predicting AI – or Failing To.” In Beyond AI: Artificial Dreams. Pilsen: University of West Bohemia.Bill Hibbard. 2012. “Avoiding Unintended AI Behaviors.” In Proceedings of AGI 2012. Springer.Bill Hibbard. 2012. “Decision Support for Safe AI Design.” In Proceedings of AGI 2012. Springer.Luke Muehlhauser. 2012. “AI Risk Bibliography 2012.” Working paper. MIRI.Anna Salamon and Luke Muehlhauser. 2012. “Singularity Summit 2011 Workshop Report.” Working paper. MIRI.Carl Shulman and Nick Bostrom. 2012. “How Hard Is Artificial Intelligence? Evolutionary Arguments and Selection Effects.” Journal of Consciousness Studies 19 (7–8): 103–130.Kaj Sotala. 2012. “Advantages of Artificial Intelligences, Uploads, and Digital Minds.” International Journal of Machine Consciousness 4 (1): 275-291.Kaj Sotala and Harri Valpola. 2012. “Coalescing Minds: Brain Uploading-Related Group Mind Scenarios.” International Journal of Machine Consciousness 4 (1): 293–312.
2011
Peter de Blanc. 2011. “Ontological Crises in Artificial Agents’ Value Systems.” arXiv:1105.3821 [cs.AI] Daniel Dewey. 2011. “Learning What to Value.” In Proceedings of AGI 2011. Springer.Eliezer Yudkowsky. 2011. “Complex Value Systems Are Required to Realize Valuable Futures.” In Proceedings of AGI 2011. Springer.
2010
Joshua Fox and Carl Shulman. 2010. “Superintelligence Does Not Imply Benevolence.” In Proceedings of ECAP 2010. Verlag Dr. Hut.Steven Kaas, Steve Rayhawk, Anna Salamon, and Peter Salamon. 2010. “Economic Implications of Software Minds.” In Proceedings of ECAP 2010. Verlag Dr. Hut.Anna Salamon, Steve Rayhawk, and János Kramár. 2010. “How Intelligible Is Intelligence?” In Proceedings of ECAP 2010. Verlag Dr. Hut.Carl Shulman. 2010. “Omohundro’s ‘Basic AI Drives’ and Catastrophic Risks.” Working paper. MIRI.Carl Shulman. 2010. “Whole Brain Emulation and the Evolution of Superorganisms.” Working paper. MIRI.Carl Shulman and Anders Sandberg. 2010. “Implications of a Software-Limited Singularity.” In Proceedings of ECAP 2010. Verlag Dr. Hut.Kaj Sotala. 2010. “From Mostly Harmless to Civilization-Threatening.” In Proceedings of ECAP 2010. Verlag Dr. Hut.Nick Tarleton. 2010. “Coherent Extrapolated Volition: A Meta-Level Approach to Machine Ethics.” Working paper. MIRI.Eliezer Yudkowsky. 2010. “Timeless Decision Theory.” Working paper. MIRI.Eliezer Yudkowsky, Carl Shulman, Anna Salamon, Rolf Nelson, Steven Kaas, Steve Rayhawk, and Tom McCabe. 2010. “Reducing Long-Term Catastrophic Risks from Artificial Intelligence.” Working paper. MIRI.
2001-2009
Peter de Blanc. 2009. “Convergence of Expected Utility for Universal Artificial Intelligence.” arXiv:0907.5598 [cs.AI].Steve Rayhawk, Anna Salamon, Michael Anissimov, Thomas McCabe, and Rolf Nelson. 2009. “Changing the Frame of AI Futurism: From Storytelling to Heavy-Tailed, High-Dimensional Probability Distributions.” Paper presented at ECAP 2009.Carl Shulman and Stuart Armstrong. 2009. “Arms Control and Intelligence Explosions.” Paper presented at ECAP 2009.Carl Shulman, Henrik Jonsson, and Nick Tarleton. 2009. “Machine Ethics and Superintelligence.” In Proceedings of AP-CAP 2009. University of Tokyo.Carl Shulman, Nick Tarleton, and Henrik Jonsson. 2009. “Which Consequentialism? Machine Ethics and Moral Divergence.” In Proceedings of AP-CAP 2009. University of Tokyo.Eliezer Yudkowsky. 2008. “Cognitive Biases Potentially Affecting Judgement of Global Risks.” In Global Catastrophic Risks. Oxford University Press. (Italiano) (Pу́сский) (Portuguese)Eliezer Yudkowsky. 2007. “Levels of Organization in General Intelligence.” In
Artificial General Intelligence (Cognitive Technologies). Springer.Eliezer Yudkowsky. 2004. “Coherent Extrapolated Volition.” Working paper. MIRI.Eliezer Yudkowsky. 2001. “Creating Friendly AI 1.0: The Analysis and Design of Benevolent Goal Architectures.” Working paper. MIRI. Sign up to get updates on new MIRI technical results
Get notified every time a new technical paper is published.
Books
Rationality: From AI to Zombies
Eliezer Yudkowsky (2015)
What does it actually mean to be rational? Not Hollywood-style “rational,” where one rejects all human feeling to embrace Cold Hard Logic — real rationality, of the sort studied by psychologists, social scientists, and mathematicians. The kind of rationality where you make good decisions, even when it’s hard; where you reason well, even in the face of massive uncertainty.
In Rationality: From AI to Zombies, Eliezer Yudkowsky explains the findings of cognitive science, and the ideas of naturalistic philosophy, which help to motivate why MIRI’s research program exists.
Smarter Than Us: The Rise of Machine Intelligence
Stuart Armstrong (2014)
What happens when machines become smarter than humans? Humans steer the future not because we’re the strongest or the fastest but because we’re the smartest. When machines become smarter than humans, we’ll be handing them the steering wheel. What promises—and perils—will these powerful machines present? Stuart Armstrong’s new book navigates these questions with clarity and wit.
Facing the Intelligence Explosion
Luke Muehlhauser (2013)
Sometime this century, machines will surpass human levels of intelligence and ability. This event—the “intelligence explosion”—will be the most important event in our history, and navigating it wisely will be the most important thing we can ever do.
Luminaries from Alan Turing and I. J. Good to Bill Joy and Stephen Hawking have warned us about this. Why do we think Hawking and company are right, and what can we do about it?
Facing the Intelligence Explosion is Muehlhauser’s attempt to answer these questions.
The Hanson-Yudkowsky AI-Foom Debate
Robin Hanson and Eliezer Yudkowsky (2013)
In late 2008, economist Robin Hanson and AI theorist Eliezer Yudkowsky conducted an online debate about the future of artificial intelligence, and in particular about whether generally intelligent AIs will be able to improve their own capabilities very quickly (a.k.a. “foom”). James Miller and Carl Shulman also contributed guest posts to the debate.
The original debate took place in a long series of blog posts, which are collected here. This book also includes a transcript of a 2011 in-person debate between Hanson and Yudkowsky on this subject, a summary of the debate written by Kaj Sotala, and a 2013 technical report on AI takeoff dynamics (“intelligence explosion microeconomics”) written by Yudkowsky.
Other
MIRI’s Blog
- Analyses – MIRI publishes some of its most substantive research to its blog.
- Conversations – MIRI interviews a diverse array of researchers and intellectuals on topics related to its research.
Resources for Researchers
- A public BibLaTeX file.
- A public Mendeley group.
- A public GitHub repo.
