New report: “Distributions allowing tiling of staged subjective EU maximizers”

 |   |  Papers

Distributions allowing reportMIRI has released a new technical report by Eliezer Yudkowsky, “Distributions allowing tiling of staged subjective EU maximizers,” which summarizes some work done at MIRI’s May 2014 workshop.


We consider expected utility maximizers making a staged series of sequential choices, and replacing themselves with successors on each time-step (to represent self-modification). We wanted to find conditions under which we could show that a staged expected utility maximizer would replace itself with another staged EU maximizer (representing stability of this decision criterion under self-modification). We analyzed one candidate condition and found that the “Optimizer’s Curse” implied that maximization at each stage was not actually optimal. To avoid this, we generated an extremely artificial function η that should allow expected utility maximizers to tile. We’re still looking for the exact necessary and sufficient condition.

  • János Kramár

    I think that, for better or for worse, if E means expectation according to your subjective Bayesian distribution and if the successor operates by doing Bayesian updating from your distribution, and if eta is your Bayesian expected utility, then the strong condition is met.