We’ve released a new working paper by Benja Fallenstein and Nate Soares, “Problems of self-reference in self-improving space-time embedded intelligence.”
Abstract:
By considering agents to be a part of their environment, Orseau and Ring’s space-time embedded intelligence is a better fi t for the real world than the traditional agent framework. However, a self-modifying AGI that sees future versions of itself as an ordinary part of the environment may run into problems of self-reference. We show that in one particular model based on formal logic, naive approaches either lead to incorrect reasoning that allows an agent to put off an important task forever (the procrastination paradox), or fail to allow the agent to justify even obviously safe rewrites (the Löbian obstacle). We argue that these problems have relevance beyond our particular formalism, and discuss partial solutions.
This working paper also cites a brief new technical report by Fallenstein, “Procrastination in probabilistic logic.”
Update 05/14/14: This paper has been accepted to AGI-14.
