# Research Workshops

#### May 13–19, 2015 – Cambridge University

## Self-Prediction in Decision Theory and AI

- Arif AhmedArif Ahmed (Cambridge)
- Stuart ArmstrongStuart Armstrong (Oxford)
- Rachael BriggsRachael Briggs (ANU)
- Daniel DeweyDaniel Dewey (Oxford)
- Benja FallensteinBenja Fallenstein (Bristol U)
- Preston GreenePreston Greene (NTU)
- Alan HájekAlan Hájek (ANU)

- Joseph HalpernJoseph Halpern (Cornell)
- James JoyceJames Joyce (U Michigan)
- Huw PriceHuw Price (Cambridge)
- Wlodek RabinowiczWlodek Rabinowicz (Lund)
- Stuart RussellStuart Russell (Berkeley)
- Vladimir SlepnevVladimir Slepnev (Google)
- Eliezer YudkowskyEliezer Yudkowsky (MIRI)

This workshop will focus on problems of self-prediction in decision theory, from both philosophical and computer science perspectives. The workshop is co-organized by MIRI, CSER, and the philosophy department at Cambridge University.

#### May 3–11, 2014 – Berkeley, CA

## 7th Workshop on Logic, Probability, and Reflection

- Mihaly BaraszMihaly Barasz (Google)
- Paul ChristianoPaul Christiano (UC Berkeley)
- Benja FallensteinBenja Fallenstein (Bristol U)
- Marcello HerreshoffMarcello Herreshoff (Google)
- Patrick LaVictoirePatrick LaVictoire (Quixey)

- Nate SoaresNate Soares (Google)
- Nisan StiennonNisan Stiennon (Stanford)
- Qiaochu YuanQiaochu Yuan (UC Berkeley)
- Eliezer YudkowskyEliezer Yudkowsky (MIRI)

Participants at this workshop — all of them veterans of past workshops — worked on a variety of problems related to Friendly AI. The first tech report from this workshop is available here.

#### December 14–20, 2013 – Berkeley, CA

## 6th Workshop on Logic, Probability, and Reflection

- Nate AckermanNate Ackerman (Harvard)
- John BaezJohn Baez (UC Riverside)
- Paul ChristianoPaul Christiano (UC Berkeley)
- Benja FallensteinBenja Fallenstein (Bristol U)
- Cameron FreerCameron Freer (MIT)
- Jeremy HahnJeremy Hahn (Harvard)
- Wojtek MoczydlowskiWojtek Moczydlowski (Google)

- Michele ReillyMichele Reilly (independent)
- Will SawinWill Sawin (Princeton)
- Nate SoaresNate Soares (Google)
- Nisan StiennonNisan Stiennon (Stanford)
- Gregory WheelerGregory Wheeler (LMU Munich)
- Eliezer YudkowskyEliezer Yudkowsky (MIRI)

Participants at this workshop focused on the Löbian obstacle, probabilistic logic, and the intersection of logic and probability more generally. The results of this workshop are described here. See photos from the workshop here.

#### November 23-29, 2013 – Oxford, UK

## 5th Workshop on Logic, Probability, and Reflection

- Stuart ArmstrongStuart Armstrong (Oxford)
- Mihaly BaraszMihaly Barasz (Google)
- Catrin Campbell-MooreCatrin Campbell-Moore (LMU Munich)
- Daniel DeweyDaniel Dewey (Oxford)
- Benja FallensteinBenja Fallenstein (Bristol U)

- Jacob HiltonJacob Hilton (Oxford)
- Ramana KumarRamana Kumar (Cambridge)
- Jan LeikeJan Leike (U Freiburg)
- Bas SteunebrinkBas Steunebrink (IDSIA)
- Gregory WheelerGregory Wheeler (LMU Munich)
- Eliezer YudkowskyEliezer Yudkowsky (MIRI)

Participants at this workshop investigated problems related to reflective agents, probabilistic logic, and priors over logical statements / the logical omniscience problem. Some results from this workshop were developed further at the December 2013 workshop and described here.

#### September 7-13, 2013 – Berkeley, CA

## 4th Workshop on Logic, Probability, and Reflection

- Paul ChristianoPaul Christiano (UC Berkeley)
- Wei DaiWei Dai (independent)
- Gary DrescherGary Drescher (independent)
- Kenny EaswaranKenny Easwaran (USC)
- Cameron FreerCameron Freer (MIT)
- Patrick LaVictoirePatrick LaVictoire (Quixey)
- Ilya ShpitserIlya Shpitser (U Southampton)
- Vladimir SlepnevVladimir Slepnev (Google)
- Nisan StiennonNisan Stiennon (Stanford)
- Andreas StuhlmüllerAndreas Stuhlmüller (MIT & Stanford)
- Eliezer YudkowskyEliezer Yudkowsky (MIRI)

This workshop focused on a variety of open problems related to normative decision theory. Participants brainstormed “well-posed problems” in the area, built on LaVictoire et al.’s Löbian cooperation work, made some progress on formalizing updateless decision theory, and formulated additional toy problems such as the Ultimate Newcomb’s Problem.

These results are still being written up in various forms.

#### July 8-14, 2013 – Berkeley, CA

## 3rd Workshop on Logic, Probability, and Reflection

- Andrew CritchAndrew Critch (PhD, UC Berkeley)
- Abram DemskiAbram Demski (USC)
- Benja FallensteinBenja Fallenstein (Bristol U)
- Marcello HerreshoffMarcello Herreshoff (Google)
- Jonathan LeeJonathan Lee (Cambridge)
- Will SawinWill Sawin (Princeton)
- Qiaochu YuanQiaochu Yuan (UC Berkeley)
- Eliezer YudkowskyEliezer Yudkowsky (MIRI)

This workshop focused on a variety of issues related to the Löbian obstacle for self-modifying systems, and to Demski’s earlier work on logical prior probability. The primary result was a proof that attempting to create a probability distribution which performs scientific induction on Π_{1} statements, converging to probability 1 for the true versions of such statements, can create zero limiting probabilities assigned to true Π_{2} statements. This result is still being written up, but it has been discussed briefly in a blog post by Demski. Other bits of progress were developed at further workshops and described here.

#### April 3-24, 2013 – Berkeley, CA

## 2nd Workshop on Logic, Probability, and Reflection

- Stuart ArmstrongStuart Armstrong (Oxford)
- Mihaly BaraszMihaly Barasz (Google)
- Paul ChristianoPaul Christiano (UC Berkeley)
- Andrew CritchAndrew Critch (PhD, UC Berkeley)
- Daniel DeweyDaniel Dewey (Oxford)
- Benja FallensteinBenja Fallenstein (Bristol U)
- Marcello HerreshoffMarcello Herreshoff (Google)
- Patrick LaVictoirePatrick LaVictoire (U Wisconsin)
- Jacob SteinhardtJacob Steinhardt (Stanford)
- Jessica TaylorJessica Taylor (Stanford)
- Qiaochu YuanQiaochu Yuan (UC Berkeley)
- Eliezer YudkowskyEliezer Yudkowsky (MIRI)

This three-week workshop addressed multiple open research problems simultaneously. First, participants found an improved version of the reflection principle discovered in the previous workshop, though this progress is still being written up. Second, participants improved upon earlier work by LaVictoire, resulting in the paper “Robust Cooperation in the Prisoner’s Dilemma: Program Equilibrium via Provability Logic.” Third, participants improved upon Benja Fallenstein’s parametric polymorphism approach to tackling the Löbian obstacle for self-modifying systems.

#### November 11-18, 2012 – Berkeley, CA

## 1st Workshop on Logic, Probability, and Reflection

- Mihaly BaraszMihaly Barasz (Google)
- Paul ChristianoPaul Christiano (UC Berkeley)
- Marcello HerreshoffMarcello Herreshoff (Google)
- Eliezer YudkowskyEliezer Yudkowsky (MIRI)

This workshop pursued one line of attack on the Löbian obstacle for self-modifying systems. The primary result of this workshop was a non-constructive “loophole” in Tarski’s undefinability of truth (via a fixed point theorem), which was later written up in draft form as “Definability of Truth in Probabilistic Logic” (see discussions here, here, and here).