Shah and Yudkowsky on alignment failures

Interesting extracts from the public discussion of Ngo and Yudkowsky on AI capability gains:

Eliezer:

I think some of your confusion may be that you’re putting “probability theory” and “Newtonian gravity” into the same bucket.  You’ve been raised to believe that powerful theories ought to meet certain standards, like successful bold advance experimental predictions, such as Newtonian gravity made about the existence of Neptune (quite a while after the theory was first put forth, though).  “Probability theory” also sounds like a powerful theory, and the people around you believe it, so you think you ought to be able to produce a powerful advance prediction it made; but it is for some reason hard to come up with an example like the discovery of Neptune, so you cast about a bit and think of the central limit theorem.  That theorem is widely used and praised, so it’s “powerful”, and it wasn’t invented before probability theory, so it’s “advance”, right?  So we can go on putting probability theory in the same bucket as Newtonian gravity?

They’re actually just very different kinds of ideas, ontologically speaking, and the standards to which we hold them are properly different ones.  It seems like the sort of thing that would take a subsequence I don’t have time to write, expanding beyond the underlying obvious ontological difference between validities and empirical-truths, to cover the way in which “How do we trust this, when” differs between “I have the following new empirical theory about the underlying model of gravity” and “I think that the logical notion of ‘arithmetic’ is a good tool to use to organize our current understanding of this little-observed phenomenon, and it appears within making the following empirical predictions…”  But at least step one could be saying, “Wait, do these two kinds of ideas actually go into the same bucket at all?”

In particular it seems to me that you want properly to be asking “How do we know this empirical thing ends up looking like it’s close to the abstraction?” and not “Can you show me that this abstraction is a very powerful one?”  Like, imagine that instead of asking Newton about planetary movements and how we know that the particular bits of calculus he used were empirically true about the planets in particular, you instead started asking Newton for proof that calculus is a very powerful piece of mathematics worthy to predict the planets themselves – but in a way where you wanted to see some highly valuable material object that calculus had produced, like earlier praiseworthy achievements in alchemy.  I think this would reflect confusion and a wrongly directed inquiry; you would have lost sight of the particular reasoning steps that made ontological sense, in the course of trying to figure out whether calculus was praiseworthy under the standards of praiseworthiness that you’d been previously raised to believe in as universal standards about all ideas.

Richard:

I agree that “powerful” is probably not the best term here, so I’ll stop using it going forward (note, though, that I didn’t use it in my previous comment, which I endorse more than my claims in the original debate).

But before I ask “How do we know this empirical thing ends up looking like it’s close to the abstraction?”, I need to ask “Does the abstraction even make sense?” Because you have the abstraction in your head, and I don’t, and so whenever you tell me that X is a (non-advance) prediction of your theory of consequentialism, I end up in a pretty similar epistemic state as if George Soros tells me that X is a prediction of the theory of reflexivity, or if a complexity theorist tells me that X is a prediction of the theory of self-organisation. The problem in those two cases is less that the abstraction is a bad fit for this specific domain, and more that the abstraction is not sufficiently well-defined (outside very special cases) to even be the type of thing that can robustly make predictions.

Perhaps another way of saying it is that they’re not crisp/robust/coherent concepts (although I’m open to other terms, I don’t think these ones are particularly good). And it would be useful for me to have evidence that the abstraction of consequentialism you’re using is a crisper concept than Soros’ theory of reflexivity or the theory of self-organisation. If you could explain the full abstraction to me, that’d be the most reliable way – but given the difficulties of doing so, my backup plan was to ask for impressive advance predictions, which are the type of evidence that I don’t think Soros could come up with.

I also think that, when you talk about me being raised to hold certain standards of praiseworthiness, you’re still ascribing too much modesty epistemology to me. I mainly care about novel predictions or applications insofar as they help me distinguish crisp abstractions from evocative metaphors. To me it’s the same type of rationality technique as asking people to make bets, to help distinguish post-hoc confabulations from actual predictions.

Of course there’s a social component to both, but that’s not what I’m primarily interested in. And of course there’s a strand of naive science-worship which thinks you have to follow the Rules in order to get anywhere, but I’d thank you to assume I’m at least making a more interesting error than that.

Lastly, on probability theory and Newtonian mechanics: I agree that you shouldn’t question how much sense it makes to use calculus in the way that you described, but that’s because the application of calculus to mechanics is so clearly-defined that it’d be very hard for the type of confusion I talked about above to sneak in. I’d put evolutionary theory halfway between them: it’s partly a novel abstraction, and partly a novel empirical truth. And in this case I do think you have to be very careful in applying the core abstraction of evolution to things like cultural evolution, because it’s easy to do so in a confused way.

Newsletter summaries attached, would appreciate it if Eliezer and Richard checked that I wasn’t misrepresenting them. (Conversation is a lot harder to accurately summarize than blog posts or papers.)

Best,

Rohin

Planned summary for the Alignment Newsletter:

Eliezer is known for being pessimistic about our chances of averting AI catastrophe. His main argument is roughly as follows:

I request that people stop describing things as my “main argument” unless I’ve described them that way myself.

Fair enough. It still does seem pretty relevant to know the purpose of the argument, and I would like to state something along those lines in the summary. For example, perhaps it is:

1. One of several relatively-independent lines of argument that suggest we’re doomed; cutting this argument would make almost no difference to the overall take
2. Your main argument, but with weird Richard-specific emphases that you wouldn’t have necessarily included if making this argument more generally; if someone refuted the core of the argument to your satisfaction it would make a big difference to your overall take
3. Not actually an argument you think much about at all, but somehow became the topic of discussion
4. Something in between these options
5. Something else entirely

If you can’t really say, then I guess I’ll just say “His argument in this particular dialogue”.

I’d also like to know what the main argument is (if there is a main argument rather than lots of independent lines of evidence or something else entirely); it helps me orient to the discussion, and I suspect would be useful for newsletter readers as well.

1. We are very likely going to keep improving AI capabilities until we reach AGI, at which point either the world is destroyed, or we use the AI system to take some pivotal act before some careless actor destroys the world.

2. In either case, the AI system must be producing high-impact, world-rewriting plans; such plans are “consequentialist” in that the simplest way to get them (and thus, the one we will first build) is if you are forecasting what might happen, thinking about the expected consequences, considering possible obstacles, searching for routes around the obstacles, etc. If you don’t do this sort of reasoning, your plan goes off the rails very quickly; it is highly unlikely to lead to high impact. In particular, long lists of shallow heuristics (as with current deep learning systems) are unlikely to be enough to produce high-impact plans.

3. We’re producing AI systems by selecting for systems that can do impressive stuff, which will eventually produce AI systems that can accomplish high-impact plans using a general underlying “consequentialist”-style reasoning process (because that’s the only way to keep doing more impressive stuff). However, this selection process does not constrain the goals towards which those plans are aimed. In addition, most goals seem to have convergent instrumental subgoals like survival and power-seeking that would lead to extinction. This suggests that, unless we find a way to constrain the goals towards which plans are aimed, we should expect an existential catastrophe.

4. None of the methods people have suggested for avoiding this outcome seem like they actually avert this story.

I would not say we face catastrophe “unless we find a way to constrain the goals towards which plans are aimed”.

Should I also change “However, this selection process does not constrain the goals towards which those plans are aimed”, and if so what to? (Something along these lines seems crucial to the argument, but if this isn’t your native ontology, then presumably you have some other thing you’d say here.)

Richard responds to this with a few distinct points:

1. It might be possible to build narrow AI systems that humans use to save the world, for example, by making AI systems that do better alignment research. Such AI systems do not seem to require the property of making long-term plans in the real world in point (3) above, and so could plausibly be safe. We might say that narrow AI systems could save the world but can’t destroy it, because humans will put plans into action for the former but not the latter.

2. It might be possible to build general AI systems that only state plans for achieving a goal of interest that we specify, without executing that plan.

3. It seems possible to create consequentialist systems with constraints upon their reasoning that lead to reduced risk.

4. It also seems possible to create systems that make effective plans, but towards ends that are not about outcomes in the real world, but instead are about properties of the plans — think for example of corrigibility (AN #35) or deference to a human user.

5. (Richard is also more bullish on coordinating not to use powerful and/or risky AI systems, though the debate did not discuss this much.)

Eliezer’s responses:

1. This is plausible, but seems unlikely; narrow not-very-consequentialist AI (aka “long lists of shallow heuristics”) will probably not scale to the point of doing alignment research better than humans.

No, your summarized-Richard-1 is just not plausible. “AI systems that do better alignment research” are dangerous in virtue of the lethally powerful work they are doing, not because of some particular narrow way of doing that work.

How about “AI systems that help with alignment research to a sufficient degree that it actually makes a difference are almost certainly already dangerous.”?

(Fwiw, I used the word “plausible” because of this sentence from the doc: “Definitely, <description of summarized-Richard-1> is among the more plausible advance-specified miracles we could get.“, though I guess the point was that it is still a miracle, it just also is more likely than other miracles.)

Thanks Rohin! Your efforts are much appreciated.

Eliezer: when you say “No, your summarized-Richard-1 is just not plausible”, do you mean the argument is implausible, or it’s not a good summary of my position (which you also think is implausible)?

For my part the main thing I’d like to modify is the term “narrow AI”. In general I’m talking about all systems that are not of literally world-destroying intelligence+agency. E.g. including oracle AGIs which I wouldn’t call “narrow”.

More generally, I don’t think all AGIs are capable of destroying the world. E.g. humans are GIs. So it might be better to characterise Eliezer as talking about some level of general intelligence which leads to destruction, and me as talking about the things that can be done with systems that are less general or less agentic than that.

We might say that narrow AI systems could save the world but can’t destroy it, because humans will put plans into action for the former but not the latter.

I don’t endorse this, I think plenty of humans would be willing to use narrow AI systems to do things that could destroy the world.

systems that make effective plans, but towards ends that are not about outcomes in the real world, but instead are about properties of the plans

I’d change this to say “systems with the primary aim of producing plans with certain properties (that aren’t just about outcomes in the world)” 

2. This might be an improvement, but not a big one. It is the plan itself that is risky; if the AI system made a plan for a goal that wasn’t the one we actually meant, and we don’t understand that plan, that plan can still cause extinction. It is the misaligned optimization that produced the plan that is dangerous, even if there was no “agent” that specifically wanted the goal that the plan was optimized for.

3 and 4. It is certainly possible to do such things; the space of minds that could be designed is very large. However, it is difficult to do such things, as they tend to make consequentialist reasoning weaker, and on our current trajectory the first AGI that we build will probably not look like that.

Planned opinion:

I first want to note my violent agreement with the notion that a major scary thing is “consequentialist reasoning”, and that high-impact plans require such reasoning, and that we will end up building AI systems that produce high-impact plans. Nonetheless, I am still optimistic about AI safety relative to Eliezer, which I suspect comes down to three main disagreements:

1. There are many approaches that don’t solve the problem, but do increase the level of intelligence required before the problem leads to extinction. Examples include Richard’s points 1-4 above. For example, if we build a system that states plans without executing them, then for the plans to cause extinction they need to be complicated enough that the humans executing those plans don’t realize that they are leading to an outcome that was not what they wanted. It seems non-trivially probable to me that such approaches are sufficient to prevent extinction up to the level of AI intelligence needed before we can execute a pivotal act.

2. The consequentialist reasoning is only scary to the extent that it is “aimed” at a bad goal. It seems non-trivially probable to me that it will be “aimed” at a goal sufficiently good to not lead to existential catastrophe, without putting in much alignment effort.
3. I do expect some coordination to not do the most risky things.

I wish the debate had focused more on the claim that narrow AI can’t e.g. do better alignment research, as it seems like a major crux. (For example, I think that sort of intuition drives my disagreement #1.) I expect AI progress looks a lot like “the heuristics get less and less shallow in a gradual / smooth / continuous manner” which eventually leads to the sorts of plans Eliezer calls “consequentialist”, whereas I think Eliezer expects a sharper qualitative change between “lots of heuristics” and that-which-implements-consequentialist-planning.

It’s reasonable to suspect failures to be concrete, but I don’t buy that hypothesis as applied to me; I think I have sufficient personal evidence against it, despite the fact that I usually speak abstractly. I don’t expect to convince you of this, nor do I particularly want to get into that sort of debate.

I’ll note that I have the exact same experience of not seeing much concreteness, both of other people and myself, about stories that lead to doom. To be clear, in what I take to be the Eliezer-story, the part where the misaligned AI designs a pathogen that wipes out all humans or solves nanotech and gains tons of power or some other pivotal act seems fine. The part that seems to lack concreteness is how we built the superintelligence and why the superintelligence was misaligned enough to lead to extinction. (Well, perhaps. I also wouldn’t be surprised if you gave a concrete example and I disagreed that it would lead to extinction.)

From my perspective, the simple concrete stories about the future are wrong and the complicated concrete stories about the future don’t sound plausible, whether about safety or about doom.

Nonetheless, here’s an attempt at some concrete stories. It is not the case that I think these would be convincing to you. I do expect you to say that it won’t be useful enough to flip the gameboard (or perhaps that if it could possibly flip the gameboard then it couldn’t be safe), but that seems to be because you think alignment will be way more difficult than I do (in expectation), and perhaps we should get into that instead.

• Instead of having to handwrite code that does feature visualization or other methods of “naming neurons”, an AI assistant can automatically inspect a neural net’s weights, perform some experiments with them, and give them human-understandable “names”. What a “name” is depends on the system being analyzed, but you could imagine that sometimes it’s short memorable phrases (e.g. for the later layers of a language model), or pictures of central concepts (e.g. for image classifiers), or paragraphs describing the concept (e.g. for novel concepts discovered by a scientist AI). Given these names, it is much easier for humans to read off “circuits” from the neural net to understand how it works.
• Like the above, except the AI assistant also reads out the circuits, and efficiently reimplements the neural network in, say, readable Python, that humans can then more easily mechanistically understand. (These two tasks could also be done by two different AI systems, instead of the same one; perhaps that would be easier / safer.)
• We have AI assistants search for inputs on which the AI system being inspected would do something that humans would rate as bad. (We can choose any not-horribly-unnatural rating scheme we want that humans can understand, e.g. “don’t say something the user said not to talk about, even if it’s in their best interest” can be a tenet for finetuned GPT-N if we want.) We can either train on those inputs, or use them as a test for how well our other alignment schemes have worked.

(These are all basically leveraging the fact that we could have AI systems that are really knowledgeable in the realm of “connecting neural net activations to human concepts”, which seems plausible to do without being super general or consequentialist.)

There’s also lots of meta stuff, like helping us with literature reviews, speeding up paper- and blog-post-writing, etc, but I doubt this is getting at what you care about

I mean, doesn’t seem crazy to just spend $50M on effective PAs, but in any case I agree with you that this is not the main thing to be thinking about

By this concreteness, it is revealed that we are not speaking of von-Neumann-plus-level AGIs who come over and firmly but gently set aside our paradigm of giant inscrutable matrices, and do something more alignable and transparent; rather, we are trying more tricks with loss functions to get human-language translations of the giant inscrutable matrices.

Agreed, but I don’t see the point here

(Beyond “Rohin and Eliezer disagree on how impossible it is to align giant inscrutable matrices”)

(I might dispute “tricks with loss functions”, but that’s nitpicky, I think)

The adversarial training case is one example where you are trying to change the system, and if you’d like I can generate more along these lines, but they aren’t going to be that different and are still going to come down to what I expect you will call “playing tricks with loss functions”

I’m definitely thinking of the last one, but I take your point that disambiguating between these is good

And I also think it’s revealing that this is not in fact the crux of disagreement

I guess the main thing is to have an example of a story which includes a method for building a superintelligence (yes, I realize this is info-hazard-y, sorry, an abstract version might work) + how it becomes misaligned and what its plans become optimized for. Though as I type this out I realize that I’m likely going to disagree on the feasibility of the method for building a superintelligence?

Yup, makes sense

What sort of domain and what training data?

Like, do we release a robot in the real world, have it collect data, build a world model, and run MuZero with a reward for making a number in a bank account go up?

Okay, so far I’m with you. Is there a separate deployment step, and if so, how did they finetune the agent for the deployment task? Or did it just take over the world halfway through training?

(I’m okay with that, I think)

Actually I think I want concreteness earlier

it started visibly talking at people and trying to manipulate them

What caused this?

Was it manipulating people in order to make e.g. sensory stuff easier to predict?

Checking understanding: At this point it is able to make novel plans that involve applying knowledge about humans and their role in the data-generating process in order to create a plan that leads to more reward for the real-world problems?

(Which we call “manipulating humans”)

Yup, makes sense. Not sure I totally agree that this sort of thing is likely to happen as quickly as it sounds like you believe but I’m happy to roll with it; I do think it will happen eventually

So doesn’t seem particularly cruxy

I can see how this leads to existential catastrophe, if you don’t expect the programmers to be worried at this early manipulation warning sign. (This is potentially cruxy for p(doom), but doesn’t feel like the main action.)

I assume this is primarily about social dynamics + the ability to patch things such that things look fixed?

Yeah, makes sense

I assume the “real underlying problem” is somehow not the fact that the task you were training your AI system to do was not what you actually wanted it to do?

Yeah, I think if I adopted (my understanding of) your beliefs about alignment difficulty, and there wasn’t already a non-racing scheme set in place, seems like we’re in trouble

I don’t actually know the response you’d have to “why not just do value alignment?” I can name several guesses

• Fragility of value
• Not sufficiently concrete
• Can’t give correct labels for human values

even if that wasn’t true, you still couldn’t do it.

Why not? Nearest unblocked strategy?

Oh, I thought you were asking me to imagine away that effect with your second sentence

In fact, I still don’t understand what it was supposed to mean

(Specifically this one:

In part, you can’t do that because one billion samples will get you one billion lethal systems, but even if that wasn’t true, you still couldn’t do it.

)

Okay, makes sense

Let me think a bit

(Agreed that is another major fundamental problem, in the sense of something that could go wrong, as opposed to something that almost certainly goes wrong)

I am still curious about the “why not value alignment” question, where to expand, it’s something like “let’s get a wide range of situations and train the agent with gradient descent to do what a human would say is the right thing to do”. (We might also call this “imitation”; maybe “value alignment” isn’t the right term, I was thinking of it as trying to align the planning with “human values”.)

My own answer is that we shouldn’t expect this to generalize to nanosystems, but that’s again much more of a “there’s not great reason to expect this to go right, but also not great reason to go wrong either”.

(This is a place where I would be particularly interested in concreteness, i.e. what does the AI system do in these cases, and how does that almost-necessarily follow from the way it was trained?)

Uh, let’s say that the AI system is being trained to act well on the Internet, and it’s shown some tweet / email / message that a user might have seen, and asked to reply to the tweet / email / message. User says whether the replies are good or not (perhaps via comparisons, a la Deep RL from Human Preferences)

If I were not making it up on the spot, it would be more varied than that, but would not include “building nanosystems”

Yeah, let’s assume that for the moment

“want to do” is putting it a bit strongly. This is more like a scenario I can’t prove is unsafe, but do not strongly believe is safe

Definitely, but my reaction to this is “okay, no particular reason for it to be safe” — but also not huge reason for it to be unsafe. Like, it would not hugely shock me if what-we-want is sufficiently “natural” that the AI system picks up on the right thing form the ‘weak-safe’ domains alone

I mean, you correctly would not accept this argument if I said that by training blue-car-driving robots solely on blue cars I am ensuring they would be bad on red-car-driving

^Right, that

I am not clear on why you expect this so strongly

Maybe you think generalization is extremely rare and optimization is a special case because of how it is so useful for basically everything?

Ah, right, I should have predicted that. (Yes, I did read it.)

I think I disagreed with that part more directly, in that it seemed like in those sections the corrigibility was assumed to be imposed “from the outside” on top of a system with a goal, rather than having a goal that was corrigible. (I also had a similar reaction to the 2015 Corrigibility paper.)

So, for example, it seems to me like CIRL is an example of an objective that can be maximized in which the agent is corrigible-in-a-certain-sense. I agree that due to updated deference it will eventually stop seeking information from the human / be subject to corrections by the human. I don’t see why, at that point, it wouldn’t have just learned to do what the humans actually want it to do.

(There are objections like misspecification of the reward prior, or misspecification of the P(behavior | reward), but those feel like different concerns to the ones you’re describing.)

I do agree that a CIRL agent would not let you shut it down

And this is something that should maybe give you pause, and be a lot more careful about potential misspecification problems

I’m not trying to claim this is a solution. I’m more trying to point at a reason why I am not convinced that corrigibility is anti-natural.

In the world where it doesn’t immediately start out as a superintelligence, it spends a lot of time trying to figure out what you want, asking you what you prefer it does, making sure to focus on the highest-EV questions, being very careful around any irreversible actions, etc

Yup, that too

You’d do that stuff because you’d want to make sure you don’t accidentally get killed by the aliens; a CIRL agent does it because it “wants to help the human”

I agree this is also an accurate description of CIRL

A more accurate description, even

Wait why is it vanishingly small relative utility? Is the assumption that the True Utility Function doesn’t care much about humans? Or was there something going on with short vs. long time horizons that I didn’t catch

Okay, cool, short vs. long time horizons

Makes sense

(Attempted rephrasing of a point I read Eliezer as making upstream, in hopes that a rephrasing makes it click for Rohin:) 

Corrigibility isn’t for bug-free CIRL agents with a prior that actually dials in on goodness given enough observation; if you have one of those you can just run it and call it a day. Rather, corrigibility is for surviving your civilization’s inability to do the job right on the first try.

CIRL doesn’t have this property; it instead amounts to the assertion “if you are optimizing with respect to a distribution on utility functions that dials in on goodness given enough observation then that gets you just about as much good as optimizing goodness”; this is somewhat tangential to corrigibility.

I guess I am not sure that I agree that this is the purpose of corrigibility-as-I-see-it. The point of corrigibility-as-I-see-it is that you don’t have to specify the object-level outcomes that your AI system must produce, and instead you can specify the meta-level processes by which your AI system should come to know what the object-level outcomes to optimize for are

(At CHAI we had taken to talking about corrigibility_MIRI and corrigibility_Paul as completely separate concepts and I have clearly fallen out of that good habit)

Cool. Sorry for the miscommunication, I think we should probably backtrack to here

so the actual problem is that I expect the optimization to generalize and the corrigibility to fail

and restart.

Though possibly I should go to bed, it is quite late here and there was definitely a time at which I would not have confused corrigibility_MIRI with corrigibility_Paul, and I am a bit worried at my completely having missed that this time

Not exactly “going meta”, no (and I don’t think I exactly mean that either). But I definitely infer a different concept from https://www.alignmentforum.org/posts/fkLYhTQteAu5SinAc/corrigibility than the one you’re describing here. It is definitely possible that this comes from me misunderstanding Paul; I have done so many times

Yup, sounds good. Talk to you tomorrow!

Quick summary of discussion so far (in which I ascribe views to Eliezer, for the sake of checking understanding, omitting for brevity the parts about how these are facts about my beliefs about Eliezer’s beliefs and not Eliezer’s beliefs themselves):

• Some discussion of “how to use non-world-optimizing AIs to help with AI alignment”, which are mostly in the category “clever tricks with gradient descent and loss functions and labeled datasets” rather than “textbook from the future”. Rohin thinks these help significantly (and that “significant help” = “reduced x-risk”). Eliezer thinks that whatever help they provide is not sufficient to cross the line from “we need a miracle” to “we have a plan that has non-trivial probability of success without miracles”. The crux here seems to be alignment difficulty.
• Some discussion of how doom plays out. I agree with Eliezer that if the AI is catastrophic by default, and we don’t have a technique that stops the AI from being catastrophic by default, and we don’t already have some global coordination scheme in place, then bad things happen. Cruxes seem to be alignment difficulty and the plausibility of a global coordination scheme, of which alignment difficulty seems like the bigger one.
• On alignment difficulty, an example scenario is “train on human judgments about what the right thing to do is on a variety of weak-safe domains, and hope for generalization to potentially-lethal domains”. Rohin views this as neither confidently safe nor confidently unsafe. Eliezer views this as confidently unsafe, because he strongly expects the optimization to generalize while the corrigibility doesn’t, because corrigibility is anti-natural.

(Incidentally, “optimization generalizes but corrigibility doesn’t” is an example of the sort of thing I wish were more concrete, if you happen to be able to do that)

My current take on “corrigibility”:

• Prior to this discussion, in my head there was corrigibility_A and corrigibility_B. Corrigibility_A, which I associated with MIRI, was about imposing a constraint “from the outside”. Given an AI system, it is a method of modifying that AI system to (say) allow you to shut it down, by performing some sort of operation on its goal. Corrigibility_B, which I associated with Paul, was about building an AI system which would have particular nice behaviors like learning about the user’s preferences, accepting corrections about what it should do, etc.
• After this discussion, I think everyone meant corrigibility_B all along. The point of the 2015 MIRI paper was to check whether it is possible to build a version of corrigibility_B that was compatible with expected utility maximization with a not-terribly-complicated utility function; the point of this was to see whether corrigibility could be made compatible with “plans that lase”.
• While I think people agree on the behaviors of corrigibility, I am not sure they agree on why we want it. Eliezer wants it for surviving failures, but maybe others want it for “dialing in on goodness”. When I think about a “broad basin of corrigibility”, that intuitively seems more compatible with the “dialing in on goodness” framing (but this is an aesthetic judgment that could easily be wrong).
• I don’t think I meant “going meta”, e.g. I wouldn’t have called indirect normativity an example of corrigibility. I think I was pointing at “dialing in on goodness” vs. “specifying goodness”.
• I agree CIRL doesn’t help survive failures. But if you instead talk about “dialing in on goodness”, CIRL does in fact do this, at least conceptually (and other alternatives don’t).
• I am somewhat surprised that “how to conceptually dial in on goodness” is not something that seems useful to you. Maybe you think it is useful, but you’re objecting to me calling it corrigibility, or saying we knew how to do it before CIRL?

(A lot of the above on corrigibility is new, because the distinction between surviving-failures and dialing-in-on-goodness as different use cases for very similar kinds of behaviors is new to me. Thanks for discussion that led me to making such a distinction.)

Possible avenues for future discussion, in the order of my-guess-at-usefulness:

1. Discussing anti-naturality of corrigibility. As a starting point: you say that an agent that makes plans but doesn’t execute them is also dangerous, because it is the plan itself that lases, and corrigibility is antithetical to lasing. Does this mean you predict that you, or I, with suitably enhanced intelligence and/or reflectivity, would not be capable of producing a plan to help an alien civilization optimize their world, with that plan being corrigible w.r.t the aliens? (This seems like a strange and unlikely position to me, but I don’t see how to not make this prediction under what I believe to be your beliefs. Maybe you just bite this bullet.)
2. Discussing why it is very unlikely for the AI system to generalize correctly both on optimization and values-or-goals-that-guide-the-optimization (which seems to be distinct from corrigibility). Or to put it another way, why is “alignment by default according to John Wentworth” doomed to fail? https://www.lesswrong.com/posts/Nwgdq6kHke5LY692J/alignment-by-default
3. More checking of where I am failing to pass your ITT
4. Why is “dialing in on goodness” not a reasonable part of the solution space (to the extent you believe that)?
5. More concreteness on how optimization generalizes but corrigibility doesn’t, in the case where the AI was trained by human judgment on weak-safe domains Just to continue to state it so people don’t misinterpret me: in most of the cases that we’re discussing, my position is not that they are safe, but rather that they are not overwhelmingly likely to be unsafe.

I don’t understand what you mean by dialling in on goodness. Could you explain how CIRL does this better than, say, reward modelling?

Reward modeling does not by default (a) choose relevant questions to ask the user in order to get more information about goodness, (b) act conservatively, especially in the face of irreversible actions, while it is still uncertain about what goodness is, or (c) take actions that are known to be robustly good, while still waiting for future information that clarifies the nuances of goodness

You could certainly do something like Deep RL from Human Preferences, where the preferences are things like “I prefer you ask me relevant questions to get more information about goodness”, in order to get similar behavior. In this case you are transferring desired behaviors from a human to the AI system, whereas in CIRL the behaviors “fall out of” optimization for a specific objective

In Eliezer/Nate terms, the CIRL story shows that dialing on goodness is compatible with “plans that lase”, whereas reward modeling does not show this

The meta-level objective that CIRL is pointing to, what makes that thing deserve the name “goodness”? Like, if I just gave an alien CIRL, and I said “this algorithm dials an AI towards a given thing”, and they looked at it without any preconceptions of what the designers wanted to do, why wouldn’t they say “huh, it looks like an algorithm for dialling in on some extrapolation of the unintended consequences of people’s behaviour” or something like that?

See also this part of my second discussion with Eliezer, where he brings up CIRL: [https://www.lesswrong.com/posts/7im8at9PmhbT4JHsW/ngo-and-yudkowsky-on-alignment-difficulty#3_2__Brain_functions_and_outcome_pumps] He was emphasising that CIRL, and most other proposals for alignment algorithms, just shuffle the problematic consequentialism from the original place to a less visible place. I didn’t engage much with this argument because I mostly agree with it.

I think you are misunderstanding my point. I am not claiming that we know how to implement CIRL such that it produces good outcomes; I agree this depends a ton on having a sufficiently good P(obs | reward). Similarly, if you gave CIRL to aliens, whether or not they say it is about getting some extrapolation of unintended consequences depends on exactly what P(obs | reward) you ended up using. There is some not-too-complicated P(obs | reward) such that you do end up getting to “goodness”, or something sufficiently close that it is not an existential catastrophe; I do not claim we know what it is.

I am claiming that behaviors like (a), (b) and (c) above are compatible with expected utility theory, and thus compatible with “plans that lase”. This is demonstrated by CIRL. It is not demonstrated by reward modeling, see e.g. these three papers for problems that arise (which make it so that it is working at cross purposes with itself and seems incompatible with “plans that lase”). (I’m most confident in the first supporting my point, it’s been a long time since I read them so I might be wrong about the others.) To my knowledge, similar problems don’t arise with CIRL (and they shouldn’t, because it is a nice integrated Bayesian agent doing expected utility theory).

I could imagine an objection that P(obs | reward), while not as complicated as “the utility function that rationalizes a twitching robot”, is still too complicated to really show compatibility with plans-that-lase, but pointing out that P(obs | reward) could be misspecified doesn’t seem particularly relevant to whether behaviors (a), (b) and (c) are compatible with plans-that-lase.

Re: shuffling around the problematic consequentialism: it is not my main plan to avoid consequentialism in the sense of plans-that-lase. I broadly agree with Eliezer that you need consequentialism to do high-impact stuff. My plan is for the consequentialism to be aimed at good ends. So I agree that there is still consequentialism in CIRL, and I don’t see this as a damning point; when I talk about “dialing in to goodness”, I am thinking of aiming the consequentialism at goodness, not getting rid of consequentialism.

(You can still do things like try to be domain-specific rather than domain-general; I don’t mean to completely exclude such approaches. They do seem to give additional safety. But the mainline story is that the consequentialism / optimization is directed at what we want rather than something else.)

If you don’t know how to implement CIRL in such a way that it actually aims at goodness, then you don’t have an algorithm with properties a, b and c above.

Or, to put it another way: suppose I replace the word “goodness” with “winningness”. Now I can describe AlphaStar as follows:

• it choose relevant questions to ask (read: scouts to send) in order to get more information about winningness
• it acts conservatively while it is still uncertain about what winningness is
• it take actions that are known to be robustly good winningish, while still waiting for future information that clarifies the nuances of winningness

Now, you might say that the difference is that CIRL implements uncertainty over possible utility functions, not possible empirical beliefs. But this is just a semantic difference which shuffles the problem around without changing anything substantial. E.g. it’s exactly equivalent if we think of CIRL as an agent with a fixed (known) utility function, which just has uncertainty about some empirical parameter related to the humans it interacts with.

[…] it take actions that are known to be robustly good, while still waiting for future information that clarifies the nuances of winningness

(typo: “known to be robustly good” -> “known to be robustly winningish” :-p)

Some quick reactions, some from me and some from my model of Eliezer:

Eliezer thinks that whatever help they provide is not sufficient […] The crux here seems to be alignment difficulty.

I’d be more hesitant to declare the crux “alignment difficulty”. My understanding of Eliezer’s position on your “use AI to help with alignment” proposals (which focus on things like using AI to make paradigmatic AI systems more transparent) is “that was always the plan, and it doesn’t address the sort of problems I’m worried about”. Maybe you understand the problems Eliezer’s worried about, and believe them not to be very difficult to overcome, thus putting the crux somewhere like “alignment difficulty”, but I’m not convinced. 

I’d update towards your crux-hypothesis if you provided a good-according-to-Eliezer summary of what other problems Eliezer sees and the reasons-according-to-Eliezer that “AI make our tensors more transparent” doesn’t much address them.

Corrigibility_A […] Corrigibility_B […]

Of the two Corrigibility_B does sound a little closer to my concept, though neither of your descriptions cause me to be confident that communication has occurred. Throwing some checksums out there:

• There are three reasons a young weak AI system might accept your corrections. It could be corrigible, or it could be incorrigibly pursuing goodness, or it could be incorrigibly pursuing some other goal while calculating that accepting this correction is better according to its current goals than risking a shutdown.
• One way you can tell that CIRL is not corrigible is that it does not accept corrections when old and strong.
• There’s an intuitive notion of “you’re here to help us implement a messy and fragile concept not yet clearly known to us; work with us here?” that makes sense to humans, that includes as a side effect things like “don’t scan my brain and then disregard my objections; there could be flaws in how you’re inferring my preferences from my objections; it’s actually quite important that you be cautious and accept brain surgery even in cases where your updated model says we’re about to make a big mistake according to our own preferences”.

The point of the 2015 MIRI paper was to check whether it is possible to build a version of corrigibility_B that was compatible with expected utility maximization with a not-terribly-complicated utility function; the point of this was to see whether corrigibility could be made compatible with “plans that lase”.

More like:

• Corrigibility seems, at least on the surface, to be in tension with the simple and useful patterns of optimization that tend to be spotlit by demands for cross-domain success, similar to how acting like two oranges are worth one apple and one apple is worth one orange is in tension with those patterns.
• In practice, this tension seems to run more than surface-deep. In particular, various attempts to reconcile the tension fail, and cause the AI to have undesirable preferences (eg, incentives to convince you to shut it down whenever its utility is suboptimal), exploitably bad beliefs (eg, willingness to bet at unreasonable odds that it won’t be shut down), and/or to not be corrigible in the first place (eg, a preference for destructively uploading your mind against your protests, at which point further protests from your coworkers are screened off by its access to that upload).

(There’s an argument I occasionally see floating around these parts that goes “ok, well what if the AI is fractally corrigible, in the sense that instead of its cognition being oriented around pursuit of some goal, its cognition is oriented around doing what it predicts a human would do (or what a human would want it to do) in a corrigible way, at every level and step of its cognition”. This is perhaps where you perceive a gap between your A-type and B-type notions, where MIRI folk tend to be more interested in reconciling the tension between corrigibility and coherence, and Paulian folk tend to place more of their chips on some such fractal notion? 

I admit I don’t find much hope in the “fractally corrigible” view myself, and I’m not sure whether I could pass a proponent’s ITT, but fwiw my model of the Yudkowskian rejoinder is “mindspace is deep and wide; that could plausibly be done if you had sufficient mastery of minds; you’re not going to get anywhere near close to that in practice, because of the way that basic normal everyday cross-domain training will highlight patterns that you’d call orienting-cognition-around-a-goal”.)

And my super-quick takes on your avenues for future discussion:

1. Discussing anti-naturality of corrigibility.

Hopefully the above helps.

2. Discussing why it is very unlikely for the AI system to generalize correctly both on optimization and values-or-goals-that-guide-the-optimization

The concept “patterns of thought that are useful for cross-domain success” is latent in the problems the AI faces, and known to have various simple mathematical shadows, and our training is more-or-less banging the AI over the head with it day in and day out. By contrast, the specific values we wish to be pursued are not latent in the problems, are known to lack a simple boundary, and our training is much further removed from it.

3. More checking of where I am failing to pass your ITT

+1

4. Why is “dialing in on goodness” not a reasonable part of the solution space?

It has long been the plan to say something less like “the following list comprises goodness: …” and more like “yo we’re tryin to optimize some difficult-to-name concept; help us out?”. “Find a prior that, with observation of the human operators, dials in on goodness” is a fine guess at how to formalize the latter. 

If we had been planning to take the former tack, and you had come in suggesting CIRL, that might have helped us switch to the latter tack, which would have been cool. In that sense, it’s a fine part of the solution. 

It also provides some additional formality, which is another iota of potential solution-ness, for that part of the problem. 

It doesn’t much address the rest of the problem, which is centered much more around “how do you point powerful cognition in any direction at all” (such as towards your chosen utility function or prior thereover).

5. More concreteness on how optimization generalizes but corrigibility doesn’t, in the case where the AI was trained by human judgment on weak-safe domains

+1

If you don’t know how to implement CIRL in such a way that it actually aims at goodness, then you don’t have an algorithm with properties a, b and c above.

I want clarity on the premise here:

• Is the premise “Rohin cannot write code that when run exhibits properties a, b, and c”? If so, I totally agree, but I’m not sure what the point is. All alignment work ever until the very last step will not lead you to writing code that when run exhibits an aligned superintelligence, but this does not mean that the prior alignment work was useless.
• Is the premise “there does not exist code that (1) we would call an implementation of CIRL and (2) when run has properties a, b, and c”? If so, I think your premise is false, for the reasons given previously (I can repeat them if needed)

I imagine it is neither of the above, and you are trying to make a claim that some conclusion that I am drawing from or about CIRL is invalid, because in order for me to draw that conclusion, I need to exhibit the correct P(obs | reward). If so, I want to know which conclusion is invalid and why I have to exhibit the correct P(obs | reward) before I can reach that conclusion.

I agree that the fact that you can get properties (a), (b) and (c) are simple straightforward consequences of being Bayesian about a quantity you are uncertain about and care about, as with AlphaStar and “winningness”. I don’t know what you intend to imply by this — because it also applies to other Bayesian things, it can’t imply anything about alignment? I also agree the uncertainty over reward is equivalent to uncertainty over some parameter of the human (and have proved this theorem myself in the paper I wrote on the topic). I do not claim that anything in here is particularly non-obvious or clever, in case anyone thought I was making that claim.

To state it again, my claim is that behaviors like (a), (b) and (c) are consistent with “plans-that-lase”, and as evidence for this claim I cite the existence of an expected-utility-maximizing algorithm that displays them, specifically CIRL with the correct p(obs | reward). I do not claim that I can write down the code, I am just claiming that it exists. If you agree with the claim but not the evidence then let’s just drop the point. If you disagree with the claim then tell me why it’s false. If you are unsure about the claim then point to the step in the argument you think doesn’t work.

The reason I care about this claim is that it seems to me like even if you think that superintelligences only involve plans-that-lase, it seems to me like this does not rule out what we might call “dialing in to goodness” or “assisting the user”, and thus it seems like this is a valid target for you to try to get your superintelligence to do.

I suspect that I do not agree with Eliezer about what plans-that-lase can do, but it seems like the two of us should at least agree that behaviors like (a), (b) and (c) can be exhibited in plans-that-lase, and if we don’t agree on that some sort of miscommunication has happened.

Throwing some checksums out there

The checksums definitely make sense. (Technically I could name more reasons why a young AI might accept correction, such as “it’s still sphexish in some areas, accepting corrections is one of those reasons”, and for the third reason the AI could be calculating negative consequences for things other than shutdown, but that seems nitpicky and I don’t think it means I have misunderstood you.) 

I think the third one feels somewhat slippery and vague, in that I don’t know exactly what it’s claiming, but it clearly seems to be the same sort of thing as corrigibility. Mostly it’s more like I wouldn’t be surprised if the Textbook from the Future tells us that we mostly had the right concept of corrigibility, but that third checksum is not quite how they would describe it any more. I would be a lot more surprised if the Textbook says we mostly had the right concept but then says checksums 1 and 2 were misguided.

“The point of the 2015 MIRI paper was to check whether it is possible to build a version of corrigibility_B that was compatible with expected utility maximization with a not-terribly-complicated utility function; the point of this was to see whether corrigibility could be made compatible with ‘plans that lase’.”

More like:

• Corrigibility seems, at least on the surface, to be in tension with the simple and useful patterns of optimization that tend to be spotlit by demands for cross-domain success, similar to how as acting like an two oranges are worth one apple and one apple is worth one orange is in tension with those patterns.
• In practice, this tension seems to run more than surface-deep. In particular, various attempts to reconcile the tension fail, and cause the AI to have undesirable preferences (eg, incentives to convince you to shut it down whenever its utility is suboptimal), exploitably bad beliefs (eg, willingness to bet at unreasonable odds that it won’t be shut down), and/or to not be corrigible in the first place (eg, a preference for destructively uploading your mind against your protests, at which point further protests from your coworkers are screened off by its access to that upload).

On the 2015 Corrigibility paper, is this an accurate summary: “it wasn’t that we were checking whether corrigibility could be compatible with useful patterns of optimization; it was already obvious at least at a surface level that corrigibility was in tension with these patterns, and we wanted to check and/or show that this tension persisted more deeply and couldn’t be easily fixed”.

(My other main hypothesis is that there’s an important distinction between “simple and useful patterns of optimization” (term in your message) and “plans that lase” (term in my message) but if so I don’t know what it is.)

What we wanted to do was show that the apparent tension was merely superficial. We failed.

(Also, IIRC — and it’s been a long time since I checked — the 2015 paper contains only one exploration, relating to an idea of Stuart Armstrong’s. There were another host of ideas raised and shot down in that era, that didn’t make it into that paper, pro’lly b/c they came afterwards.)

What we wanted to do was show that the apparent tension was merely superficial. We failed.

(That sounds like what I originally said? I’m a bit confused why you didn’t just agree with my original phrasing:

The point of the 2015 MIRI paper was to check whether it is possible to build a version of corrigibility_B that was compatible with expected utility maximization with a not-terribly-complicated utility function; the point of this was to see whether corrigibility could be made compatible with “plans that lase”.

)

(I’m kinda worried that there’s some big distinction between “EU maximization”, “plans that lase”, and “simple and useful patterns of optimization”, that I’m not getting; I’m treating them as roughly equivalent at the moment when putting on my MIRI-ontology-hat.)

(There are a bunch of aspects of your phrasing that indicated to me a different framing, and one I find quite foreign. For instance, this talk of “building a version of corrigibility_B” strikes me as foreign, and the talk of “making it compatible with ‘plans that lase'” strikes me as foreign. It’s plausible to me that you, who understand your original framing, can tell that my rephrasing matches your original intent. I do not yet feel like I could emit the description you emitted without contorting my thoughts about corrigibility in foreign ways, and I’m not sure whether that’s an indication that there are distinctions, important to me, that I haven’t communicated.)

(I’m kinda worried that there’s some big distinction between “EU maximization”, “plans that lase”, and “simple and useful patterns of optimization”, that I’m not getting; I’m treating them as roughly equivalent at the moment when putting on my MIRI-ontology-hat.)

I, too, believe them to be basically equivalent (with the caveat that the reason for using expanded phrasings is because people have a history of misunderstanding “utility maximization” and “coherence”, and so insofar as you round them all to “coherence” and then argue against some very narrow interpretation of coherence, I’m gonna protest that you’re bailey-and-motting).

Hopefully the above helps.

I’m still interested in the question “Does this mean you predict that you, or I, with suitably enhanced intelligence and/or reflectivity, would not be capable of producing a plan to help an alien civilization optimize their world, with that plan being corrigible w.r.t the aliens?” I don’t currently understand how you avoid making this prediction given other stated beliefs. (Maybe you just bite the bullet and do predict this?)

By contrast, the specific values we wish to be pursued are not latent in the problems, are known to lack a simple boundary, and our training is much further removed from it.

I’m not totally sure what is meant by “simple boundary”, but it seems like a lot of human values are latent in text prediction on the Internet, and when training from human feedback the training is not very removed from values.

It has long been the plan to say something less like “the following list comprises goodness: …” and more like “yo we’re tryin to optimize some difficult-to-name concept; help us out?”. […]

I take this to mean that “dialing in on goodness” is a reasonable part of the solution space? If so, I retract that question. I thought from previous comments that Eliezer thought this part of solution space was more doomed than corrigibility.

(I get the sense that people think that I am butthurt about CIRL not getting enough recognition or something. I do in fact think this, but it’s not part of my agenda here. I originally brought it up to make the argument that corrigibility is not in tension with EU maximization, then realized that I was mistaken about what “corrigibility” meant, but still care about the argument that “dialing in on goodness” is not in tension with EU maximization. But if we agree on that claim then I’m happy to stop talking about CIRL.)

I’d be capable of helping aliens optimize their world, sure. I wouldn’t be motivated to, but I’d be capable.

(There are a bunch of aspects of your phrasing that indicated to me a different framing, and one I find quite foreign. For instance, this talk of “building a version of corrigibility_B” strikes me as foreign, and the talk of “making it compatible with ‘plans that lase'” strikes me as foreign. It’s plausible to me that you, who understand your original framing, can tell that my rephrasing matches your original intent. I do not yet feel like I could emit the description you emitted without contorting my thoughts about corrigibility in foreign ways, and I’m not sure whether that’s an indication that there are distinctions, important to me, that I haven’t communicated.)

This makes sense. I guess you might think of these concepts as quite pinned down? Like, in your head, EU maximization is just a kind of behavior (= set of behaviors), corrigibility is just another kind of behavior (= set of behaviors), and there’s a straightforward yes-or-no question about whether the intersection is empty which you set out to answer, you can’t “make” it come out one way or the other, nor can you “build” a new kind of corrigibility

Re: CIRL, my current working hypothesis is that by “use CIRL” you mean something analogous to what I say when I say “do CEV” — namely, direct the AI to figure out what we “really” want in some correct sense, rather than attempting to specify what we want concretely. And to be clear, on my model, this is part of the solution to the overall alignment problem, and it’s more-or-less why we wouldn’t die immediately on the “value is fragile / we can’t name exactly what we want” step if we solved the other problems.

My guess as to the disagreement about how much credit CIRL should get, is that there is in fact a disagreement, but it’s not coming from MIRI folk saying “no we should be specifying the actual utility function by hand”, it’s coming from MIRI folk saying “this is just the advice ‘do CEV’ dressed up in different clothing and presented as a reason to stop worrying about corrigibility, which is irritating, given that it’s orthogonal to corrigibility”.

If you wanna fight that fight, I’d start by asking: Do you think CIRL is doing anything above and beyond what “use CEV” is doing? If so, what?

Regardless, I think it might be a good idea for you to try to pass my (or Eliezer’s) ITT about what parts of the problem remain beyond the thing I’d call “do CEV” and why they’re hard. (Not least b/c if my working hypothesis is wrong, demonstrating your mastery of that subject might prevent a bunch of toil covering ground you already know.)

I’d be capable of helping aliens optimize their world, sure. I wouldn’t be motivated to, but I’d be capable.

Okay, so it seems like the danger requires the thing-producing-the-plan to be badly-motivated. But then I’m not sure why it seems so impossible to have a (not-badly-motivated) thing that, when given a goal, produces a plan to corrigibly get that goal. (This is a scenario Richard mentioned earlier.)

This makes sense. I guess you might think of these concepts as quite pinned down? Like, in your head, EU maximization is just a kind of behavior (= set of behaviors), corrigibility is just another kind of behavior (= set of behaviors), and there’s a straightforward yes-or-no question about whether the intersection is empty which you set out to answer, you can’t “make” it come out one way or the other, nor can you “build” a new kind of corrigibility

That sounds like one of the big directions in which your framing felt off to me, yeah :-). (I don’t fully endorse that rephrasing, but it seems directionally correct to me.)

Okay, so it seems like the danger requires the thing-producing-the-plan to be badly-motivated. But then I’m not sure why it seems so impossible to have a (not-badly-motivated) thing that, when given a goal, produces a plan to corrigibly get that goal. (This is a scenario Richard mentioned earlier.)

On my model, aiming the powerful optimizer is the hard bit.

Like, once I grant “there’s a powerful optimizer, and all it does is produce plans to corrigibly attain a given goal”, I agree that the problem is mostly solved.

There’s maybe some cleanup, but the bulk of the alignment challenge preceded that point.

(This is hard for all the usual reasons, that I suppose I could retread.)

[…] Regardless, I think it might be a good idea for you to try to pass my (or Eliezer’s) ITT about what parts of the problem remain beyond the thing I’d call “do CEV” and why they’re hard. (Not least b/c if my working hypothesis is wrong, demonstrating your mastery of that subject might prevent a bunch of toil covering ground you already know.)

(Working on ITT)

(To clarify some points of mine, in case this gets published later to other readers: (1) I might call it more centrally something like “build a DWIM system” rather than “use CEV”; and (2) this is not advice about what your civilization should do with early AGI systems, I strongly recommend against trying to pull off CEV under that kind of pressure.)

I don’t particularly want to have fights about credit. I just didn’t want to falsely state that I do not care about how much credit CIRL gets, when attempting to head off further comments that seemed designed to appease my sense of not-enough-credit. (I’m also not particularly annoyed at MIRI, here.)

On passing ITT, about what’s left beyond “use CEV” (stated in my ontology because it’s faster to type; I think you’ll understand, but I can also translate if you think that’s important):

• The main thing is simply how to actually get the AI system to care about pursuing CEV. I think MIRI ontology would call this the target loading problem.
• This is hard because (a) you can’t just train on CEV, because you can’t just implement CEV and provide that as training and (b) even if you magically could train on CEV, that does not establish that the resulting AI system then wants to optimize CEV. It could just as well optimize some other objective that correlated with CEV in the situations you trained, but no longer correlates in some new situation (like when you are building a nanosystem). (Point (b) is how I would talk about inner alignment.)
• This is made harder for a variety of reasons, including (a) you’re working with inscrutable matrices that you can’t look at the details of, (b) there are clear racing incentives when the prize is to take over the world (or even just lots of economic profit), (c) people are unlikely to understand the issues at stake (unclear to me of the exact reasons, I’d guess it would be that the issues are too subtle / conceptual, + pressure to rationalize it away), (d) there’s very little time in which we have a good understanding of the situation we face, because of fast / discontinuous takeoff

Passable ^_^ (Not exhaustive, obviously; “it will have a tendency to kill you on the first real try if you get it wrong” being an example missing piece, but I doubt you were trying to be exhaustive.) Thanks.

Okay, so it seems like the danger requires the thing-producing-the-plan to be badly-motivated. But then I’m not sure why it seems so impossible to have a (not-badly-motivated) thing that, when given a goal, produces a plan to corrigibly get that goal. (This is a scenario Richard mentioned earlier.)

I’m uncertain where the disconnect is here. Like, I could repeat some things from past discussions about how “it only outputs plans, it doesn’t execute them” does very little (not nothing, but very little) from my perspective? Or you could try to point at past things you’d expect me to repeat and name why they don’t seem to apply to you?

(Flagging that I should go to bed soon, though it doesn’t have to be right away)

I really honestly am not trying to say that MIRI didn’t think of CIRL-like things, nor am I trying to get credit for CIRL. I really just wanted to establish that “learn what is good to do” seems not-ruled-out by EU maximization. That’s all. It sounds like we agree on this point and if so I’d prefer to drop it.

I think we all agree on this point.

One thing I’d note is that during my debate with Eliezer, I’d keep saying “oh so you think X is impossible” and he’d say “no, all these things are possible, they’re just really really hard”.

Maybe it’s fine; perhaps the point is just that target loading is hard, and the question is why target loading is so hard.

From my perspective, the main confusing thing about the Eliezer/Nate view is how confident it is. With each individual piece, I (usually) find myself nodding along and saying “yes, it seems like if we wanted to guarantee safety, we would need to solve this”. What I don’t do is say “yes, it seems like without a solution to this, we’re near-certainly dead”. The uncharitable view (which I share mainly to emphasize where the disconnect is, not because I think it is true) would be something like “Eliezer/Nate are falling to a Murphy bias, where they assume that unless they have an ironclad positive argument for safety, the worst possible thing will happen and we all die”. I try to generate things that seem more like ironclad (or at least “leatherclad”) positive arguments for doom, and mostly don’t succeed; when I say “human values are very complicated” there’s the rejoinder that “a superintelligence will certainly know about human values; pointing at them shouldn’t take that many more bits”; when I say “this is ultimately just praying for generalization”, there’s the rejoinder “but it may in fact actually generalize”; add to all of this the fact that a bunch of people will be trying to prevent the problem and it seems weird to be so confident in doom.

A lot of my questions are going to be of the form “it seems like this is a way that we could survive; it definitely involves luck and does not say good things about our civilization, but it does not seem as improbable as the word ‘miracle’ would imply”

(I have read your Rocket Alignment and security mindset posts. Not claiming this absolves me of bias, just saying that I am familiar with them)

Do you want to bet on that? That seems like a testable prediction about beliefs of real people in the not-too-distant future

Never mind, I wanted to bet on “people becoming more pessimistic as they try ideas and see them fail”, but if your idea of “see them fail” is “superintelligence kills everyone” then obviously we can’t bet on that

(people here being alignment researchers, obviously ones who are not me)

(I think what they actually do is say “it’s not obvious to me that this is one of those scenarios where we become grizzled veterans, as opposed to things just actually working out easily”)

(“many things work out easily all the time; obviously society spends a bunch more focus on things that don’t work out easily b/c the things that work easily tend to get resolved fairly quickly and then you don’t notice them”, or something)

(more generally, I kinda suspect that bickering closer to the object level is likely more productive)

(and i suspect this convo might be aided by Rohin naming a concrete scenario where things go well, so that Eliezer can lament the lack of genre saviness in various specific points)

(and possibly also Eliezer naming a concrete story where things go poorly, so that Rohin may lament the seemingly blind pessimism & premature grizzledness)

Unsurprisingly I have disagreements with the meta-level story, but it seems really thorny to make progress on and I’m kinda inclined to not discuss it. I also should go to sleep now.

One thing it did make me think of — it’s possible that the “do it correctly on your first try when a failed attempt kills you” could be the crux here. There’s a clearly-true sense which is “the first time you build a superintelligence that you cannot control, if you have failed in your alignment, then you die”. There’s a different sense which is “and also, anything you try to do with non-superintelligences that you can control, will tell you approximately nothing about the situation you face when you build a superintelligence”. I mostly don’t agree with the second sense, but if Eliezer / Nate do agree with it, that would go a long way to explaining the confidence in doom.

Two arguments I can see for the second sense: (1) the non-superintelligences only seem to respond well to alignment schemes because they don’t yet have the core of general intelligence, and (2) the non-superintelligences only seem to respond well to alignment schemes because despite being misaligned they are doing what we want in order to survive and later execute a treacherous turn. EDIT: And (3) fast takeoff = not much time to look at the closest non-dangerous examples

(I still should sleep, but would be interested in seeing thoughts tomorrow, and if enough people think it’s actually worthwhile to engage on the meta level I can do that. I’m cheerful about engaging on specific object-level ideas.)

Linking to a chat log between Eliezer and some anonymous people (and Steve Omohundro) from early September: [https://www.lesswrong.com/posts/CpvyhFy9WvCNsifkY/discussion-with-eliezer-yudkowsky-on-agi-interventions]

Eliezer tells me he thinks it pokes at some of Rohin’s questions

[…] Absent the superuniversal entity bribing me, I wouldn’t want to behave corrigibly towards the Babyeaters. […]

Got it. Yeah I think I just misunderstood a point you were saying previously. When Richard asked about systems that simply produce plans rather than execute them, you said something like “the plan itself is dangerous”, which I now realize meant “you don’t get additional safety from getting to read the plan, the superintelligence would have just chosen a plan that was convincing to you but nonetheless killed everyone / otherwise worked in favor of the superintelligence’s goals”, but at the time I interpreted it as “any reasonable plan that can actually build nanosystems is going to be dangerous, regardless of the source”, which seemed obviously false in the case of a well-motivated system.

[…] This is a weird thing to say in my own ontology. […]

When I say “dialing in on goodness”, I mean a specific class of strategies for getting a superintelligence to do a useful pivotal thing, in which you build it so that the superintelligence is applying its force towards figuring out what it is that you actually want it to do and pursuing that, which among other things would involve taking a pivotal act to reduce x-risk to ~zero.

I previously had the mistaken impression that you thought this class of strategies was probably doomed because it was incompatible with expected utility theory, which seemed wrong to me. (I don’t remember why I had this belief; possibly it was while I was still misunderstanding what you meant by “corrigibility” + the claim that corrigibility is anti-natural.)

I now think that you think it is probably doomed for the same reason that most other technical strategies are probably doomed, which is that there still doesn’t seem to be any plausible way of loading in the right target to the superintelligence, even when that target is a process for learning-what-to-optimize, rather than just what-to-optimize.

Linking to a chat log between Eliezer and some anonymous people (and Steve Omohundro) from early September: [https://www.lesswrong.com/posts/CpvyhFy9WvCNsifkY/discussion-with-eliezer-yudkowsky-on-agi-interventions]

Eliezer tells me he thinks it pokes at some of Rohin’s questions

I’m surprised that you think this addresses (or even pokes at) my questions. As far as I can tell, most of the questions there are either about social dynamics, which I’ve been explicitly avoiding, and the “technical” questions seem to treat “AGI” or “superintelligence” as a symbol; there don’t seem to be any internal gears underlying that symbol. The closest anyone got to internal gears was mentioning iterated amplification as a way of bootstrapping known-safe things to solving hard problems, and that was very brief.

I am much more into the question “how difficult is technical alignment”. It seems like answers to this question need to be in one of two categories: (1) claims about the space of minds that lead to intelligent behavior (probably weighted by simplicity, to account for the fact that we’ll get the simple ones first), (2) claims about specific methods of building superintelligences. As far as I can tell the only thing in that doc which is close to an argument of this form is “superintelligent consequentialists would find ways to manipulate humans”, which seems straightforwardly true (when they are misaligned). I suppose one might also count the assertion that “the speedup step of iterated amplification will introduce errors” as an argument of this form.

It could be that you are trying to convince me of some other beliefs that I wasn’t asking about, perhaps in the hopes of conveying some missing mood, but I suspect that it is just that you aren’t particularly clear on what my beliefs are / what I’m interested in. (Not unreasonable, given that I’ve been poking at your models, rather than the other way around.) I could try saying more about that, if you’d like.

FWIW, a voice from the audience: +1 to going back to sketching concrete scenarios. even though i learned a few things from the abstract discussion of goodness/corrigibility/etc myself (eg, that “corrigible” was meant to be defined at the limit of self-improvement till maturity, not just as a label for code that does not resist iterated development), the progress felt more tangible during the “scaled up muzero” discussion above.

Not sure I count, but one I’d enjoy a concrete response to: “The leading AI lab vaguely thinks it’s important that their systems are ‘mere predictors’, and wind up creating an AGI that is dangerous; how concretely does it wind up being a scary planning optimizer or whatever, that doesn’t run through a scary abstract “waking up” step”.

(asking for a friend; @Joe Carlsmith or whoever else finds this scenario unintuitive plz clarify with more detailed requests if interested)

I’m still interested in:

5. More concreteness on how optimization generalizes but corrigibility doesn’t, in the case where the AI was trained by human judgment on weak-safe domains

Specifically, we can go back to the scaled-up MuZero example. Some (lightly edited) details we had established there:

Pretraining: playing all the videogames, predicting all the text and images, solving randomly generated computer puzzles, accomplishing sets of easily-labelable sensorymotor tasks using robots and webcams

Finetuning: The AI system is being trained to act well on the Internet, and it’s shown some tweet / email / message that a user might have seen, and asked to reply to the tweet / email / message. User says whether the replies are good or not (perhaps via comparisons, a la Deep RL from Human Preferences). It would be more varied than that, but would not include “building nanosystems”.

The AI system is not smart enough that exposing humans to text it generates is already a world-wrecking threat if the AI is hostile.

At that point we moved from concrete to abstract:

Abstract description: train on ‘weak-safe’ domains where the AI isn’t smart enough to do damage, and the humans can label the data pretty well because the AI isn’t smart enough to fool them

Abstract problem: Optimization generalizes and corrigibility fails

I would be interested in a more concrete description here. I’m not sure exactly what details I’m looking for — on my ontology the question is something like “what algorithm is the AI system forced to learn; how does that lead to generalized optimization and failed corrigibility; why weren’t there simple safer algorithms that were compatible with the training, or if there were such algorithms why didn’t the AI system learn them”. I don’t really see how to answer all of that without abstraction, but perhaps you’ll have an answer anyway

(I am hoping to get some concrete detail on “how did it go from non-hostile to hostile”, though I suppose you might confidently predict that it was already hostile after pretraining, conditional on it being an AGI at all. I can try devising a different concrete scenario if that’s a blocker.)

Agree assuming that it is pursuing a misaligned goal, but I am also asking what misaligned goal it is pursuing (and depending on the answer, maybe also how it came to be pursuing that misaligned goal given the specified training setup).

In fact I think “what misaligned goal is it pursuing” is probably the more central question for me

Yeah I agree with that in the abstract, but have had trouble giving compelling-to-me concrete examples

Yeah I also agree with that

To be clear I am fine with “this is a case where we predictably can’t have good concrete stories and this does not mean we are safe” (and indeed argued the same thing in a doc I linked here many messages ago)

But weird false specifics could still be interesting

Although let me think if it is actually valuable

Probably it is not going to change my mind very much on alignment difficulty, if it is “weird false specifics”, so maybe this isn’t the most productive line of discussion. I’d be “selfishly” interested in that “weird false specifics” seems good for me to generate novel thoughts about these sorts of scenarios, but that seems like a bad use of this Discord

I think given the premises that (1) superintelligence is coming soon, (2) it pursues a misaligned goal by default, and (3) we currently have no technical way of preventing this and no realistic-seeming avenues for generating such methods, I am very pessimistic. I think (2) and (3) are the parts that I don’t believe and am interested in digging into, but perhaps “concrete stories” doesn’t really work for this.

In this framing I do buy that you don’t learn exactly the function that generates the feedback — I have ~5 contrived specific examples where this is the case (i.e. you learn something that wasn’t what the feedback function would have rewarded in a different distribution)

(I’m now thinking about what I actually want to say about this framing)

Actually, maybe I do think you might end up learning the function that generates the feedback. Not literally exactly, if for no other reason than rounding errors, but well enough that the inaccuracies don’t matter much. The AGI presumably already knows and understands the concepts we use based on its pretraining, is it really so shocking if gradient descent hooks up those concepts in the right way? (GPT-3 on the other hand doesn’t already know and understand the relevant concepts, so I wouldn’t predict this of GPT-3.) I do feel though like this isn’t really getting at my reason for (relative) optimism, and that reason is much more like “I don’t really buy that AGI must be very coherent in a way that would prevent corrigibility from working” (which we could discuss if desired)

On the comment that learning the exact feedback function is still fatal — I am unclear on why you are so pessimistic on having “human + AI” supervise “AI”, in order to have the supervisor be smarter than the thing being supervised. (I think) I understand the pessimism that the learned function won’t generalize correctly, but if you imagine that magically working, I’m not clear what additional reason prevents the “human + AI” supervising “AI” setup.

• I can see how you die if the AI ever becomes misaligned, i.e. there isn’t a way to fix mistakes, but I don’t see how you get the misaligned AI in the first place.
• I could also see things like “Just like a student can get away with plagiarism even when the teacher is smarter than the student, the AI knows more about its cognition than the human + AI system, and so will likely be incentivized to do bad things that it knows are bad but the human + AI system doesn’t know is bad”. But that sort of thing seems solvable with future research, e.g. debate, interpretability, red teaming all seem like feasible approaches.

One concrete training algorithm would be debate, ideally with mechanisms that allow the AI systems to “look into each other’s thoughts” and make credible statements about them, but we can skip that for now as it isn’t very concrete

Would you like a training domain and data as well?

I don’t like the fact that a smart AI system in this position could notice that it is playing against itself and decide not to participate in a zero-sum game, but I am not sure if that worry actually makes sense or not

(Debate can be thought of as simultaneously “human + first AI evaluate second AI” and “human + second AI evaluate first AI”)

You start with “easy” debates like mathematical theorem proving or fact-based questions, and ramp up until eventually the questions are roughly “what is the next thing to do in order to execute a pivotal act”

Intermediate questions might be things like “is it a good idea to have a minimum wage”

Yup, that could be it

Humans are judging debates based on reasoning though, not just outcomes-after-executing-the-plan

So assuming perfect generalization there should be some large implicit debate tree that justifies the plan in human-understandable form

Okay, so the worry is that this just won’t scale, not that (assuming perfect generalization) it is unsafe? Or perhaps you also think it is unsafe but it’s hard to engage with because you don’t believe it will scale?

And the issue is that relying on reasoning confines you to a space of possible thoughts that doesn’t include the kinds of thoughts required to develop new stuff (e.g. intuition)?

What additional concrete detail do you want?

It feels like I specified something that we could code up a stupidly inefficient version of now

In terms of what actually happens: Each episode, there is an initial question specified by the human. Agent A and agent B, which are copies of the same neural net, simultaneously produce statements (“answers”). They then have a conversation. At the end the human judge decides which answer is better, and rewards the appropriate agent. The agents are updated using some RL algorithm.

I can say stuff about why we might hope this works, or about tricks you have to play in order to get learning to happen at all, or other things

For the stupidly inefficient version they could have spent their whole lives inside text

In the stupidly inefficient version human feedback has to teach the agents facts about the real world

Oh I definitely agree this is not competitive

So when you say this is too abstract, you mean that there isn’t a story for how they incorporate e.g. physical real-world knowledge?

That does seem like a disagreement, in that I think this process does eventually reach the “deep patterns”, but I do agree it is unlikely to be competitive

It would be quite surprising to me if it took that much. It would be especially surprising to me if we couldn’t figure out some alternative reasonably-simple training scheme like “imitate a human doing good reasoning” that still remained entirely in text that could reach the “deep patterns”. (This is now no longer a discussion about whether the training scheme is aligned, not sure if we should continue it.)

I realize that this might be hard to do, but if you imagine that GPT-5 + human feedback finetuning does run across the deep patterns and could in theory do the right stuff, and also generalization magically works, what’s the next failure?

(Tbc by “human feedback finetuning” I mean debate, and I suspect that “generalization magically works” will be meant to rule out the thing that you say next, but seems worth checking so let me write an answer)

the deep thing is the most predictive thing it found for human text about science?

Wait, the most predictive thing? I was imagining it as just a thing that is present in addition to all the other things. Like, I don’t think I’ve learned a “deep thing” that is most useful for riding a bike. Probably I’m just misunderstanding what you mean here.

I don’t think I can give a good answer here, but to give some answer, it has a belief that there is a universe “out there”, that lots but not all of the text it reads is making claims about (some aspect of) the universe, those claims can be true or false, there are some claims that are known to be true, there are some ways to take assumed-true claims and generate new assumed-true claims, which includes claims about optimal actions for goals, as well as claims about how to build stuff, or what the effect of a specified machine is

In fact I am skeptical of some strong versions of the lottery ticket hypothesis, though it’s been a while since I read the paper and I don’t remember exactly what the original hypothesis was

Giant inscrutable vectors encoding meaningful propositions

How does strong lottery ticket hypothesis explain GPT-3? Seems like that should already be enough to determine that there’s an incremental trail of successively more complicated gears

I expect that probably we will figure out some way to have adaptive computation time be a thing (it’s been investigated for years now, but afaik hasn’t worked very well), which will allow for this sort of thing to happen

In the stupidly inefficient version, you have a really really giant and deep neural net that does all of that in successive layers of the neural net. (And when it doesn’t need to do that, those layers are noops.)

Just to check my understanding: if we’re talking about, say, how humans might go about understanding neural nets, there’s a goal of “have a theory that can retrodict existing observations and make new predictions”, backchaining might say “come up with hypotheses that would explain double descent”, forward chaining might say “look into bias and variance measurements”?

If so, yes, I think the AGI / GPT-5-that-is-an-AGI is doing something similar

Fair point re using a different domain

To be clear I do not in fact think that GPT-N is safe because it is trained with supervised learning and I am confused at the combination of views that GPT-N will be AGI and GPT-N will be safe because it’s just doing predictions

Maybe there is marginal additional safety but you clearly can’t say it is “definitely safe” without some additional knowledge that I have not seen so far

Going back to the original question, of what the next failure mode of debate would be assuming magical generalization, I think it’s just not one that makes sense to ask on your worldview / ontology; “magical generalization” is the equivalent of “assume that the goal-oriented mind somehow doesn’t do dangerous optimization towards its goal, yet nonetheless produces things that can only be produced by dangerous optimization towards a goal”, and so it is assuming the entire problem away

Potentially dumb question: when humans do science, why don’t they then try to take over the world to do the best possible science? (If humans are doing dangerous goal-directed optimization when doing science, why doesn’t that lead to catastrophe?)

You could of course say that they just aren’t smart enough to do so, but it sure feels like (most) humans wouldn’t want to do the best possible science even if they were smarter

I think this is similar to a question I asked before about plans being dangerous independent of their source, and the answer was that the source was misaligned

But in the description above you didn’t say anything about the thing-doing-science being misaligned, so I am once again confused

But like, in Eliezer’s case, there is a different goal that is motivating him to use Science and Engineering for this purpose

It is not the prediction-goal that he instantiated in his mind as part of the method of doing Science

Well, that might also be interesting to hear the answer to — I don’t know how I’d answer that through an Eliezer-lens — though it wasn’t exactly what I was asking

(Going back to what I meant to ask) It seems to me like humans demonstrate that you can have a prediction goal without that being your final/terminal goal. So it seems like with AI you similarly need to talk about the final/terminal goal. But then we talked about GPT and debate and so on for a while, and then you explained how GPTs would have deep patterns that do dangerous optimization, where the deep patterns involved instantiating a prediction goal. Notably, you didn’t say anything about a final/terminal goal. Do you see why I am confused?

OH you mean that the outer objective is prediction

I have for quite a while thought that you meant that Science involves internally setting a subgoal of “predict a confusing part of reality”

Okay wait. There are two things. One is that GPT-3 is trained with a loss function that one might call a prediction objective for human text. Two is that Science involves looking at a part of reality and figuring out how to predict it. These two things are totally different. I am now unsure which one(s) you were talking about in the conversation above

Yeah, so you meant the first thing, and I misinterpreted as the second thing

(I will head to bed in this case — I was meaning to do that soon anyway — but I’ll first summarize.)

Okay let me look back at the conversation

is there a little goal-oriented mind inside there that solves science problems the same way humans solve them, by engineering mental constructs that serve a goal of prediction, including backchaining for prediction goals and forward chaining from alternative hypotheses / internal tweaked states of the mental construct?

Here, is the word “prediction” meant to refer to the outer objective and/or predicting what English sentences about Science one might say, or is it referring to a subpart of the Process Of Science in which one aims to predict some aspect of reality (which is typically not in the form of English sentences)?

Okay I think my original understanding was correct in that case

from my perspective the whole field of mental endeavor as practiced by alignment optimists consists of ancient alchemists wondering if they can get collections of surface properties, like a metal as shiny as gold, as hard as steel, and as self-healing as flesh, where optimism about such wonderfully combined properties can be infinite as long as you stay ignorant of underlying structures that produce some properties but not others

I actually think something like this might be a crux for me, though obviously I wouldn’t put it the way you’re putting it. More like “are arguments about internal mechanisms more or less trustworthy than arguments about what you’re selecting for” (limiting to arguments we actually have access to, of course in the limit of perfect knowledge internal mechanisms beats selection). But that is I think a discussion for another day.

(For the reader, I don’t think that “arguments about what you’re selecting for” is the same thing as “freely combining surface desiderata”, though I do expect they look approximately the same to Eliezer)

Yeah, I think I do not in fact understand why that is true for any consistent reasonable story.

From my perspective, when I posit a hypothetical, you demonstrate that there is an underlying mechanism that produces strong capabilities that generalize combined with real world knowledge. I agree that a powerful AI system that we build capable of executing a pivotal act will have strong capabilities that generalize and real world knowledge. I am happy to assume for the purposes of this discussion that it involves backchaining from a target and forward chaining from things that you currently know or have. I agree that such capabilities could be used to cause an existential catastrophe (at least in a unipolar world, multipolar case is more complicated, but we can stick with unipolar for now). None of my arguments so far are meant to factor through the route of “make it so that the AGI can’t cause an existential catastrophe even if it wants to”.

The main question according to me is why those capabilities are aimed towards achievement of a misaligned goal.

It feels like when I try to ask why we have misaligned goals, I often get answers that are of the form “look at the deep patterns underlying the strong capabilities that generalize, obviously given a misaligned goal they would generate the plan of killing the humans who are an obstacle towards achieving that goal”. This of course doesn’t work since it’s a circular argument.

I can generate lots of arguments for why it would be aimed towards achievement of a misaligned goal, such as (1) only a tiny fraction of goals are aligned; the rest are misaligned, (2) the feedback we provide is unlikely to be the right goal and even small errors are fatal, (3) lots of misaligned goals are compatible with the feedback we provide even if the feedback is good, since the AGI might behave well until it can execute a treacherous turn, (4) the one example of strategically aware intelligence (i.e. humans) is misaligned relative to its creator. (I’m not saying I agree with these arguments, but I do understand them.)

Are these the arguments that make you think that you get misaligned goals by default? Or is it something about “deep patterns” that isn’t captured by “strong capabilities that generalize, real-world knowledge, ability to cause an existential catastrophe if it wants to”?

This is the last log in the Late 2021 MIRI Conversations. We’ll be concluding the sequence with a public Ask Me Anything (AMA) this Wednesday; you can start posting questions there now.

MIRI has found the Discord format useful, and we plan to continue using it going into 2022. This includes follow-up conversations between Eliezer and Rohin, and a forthcoming conversation between Eliezer and Scott Alexander of Astral Codex Ten.

Some concluding thoughts from Richard Ngo:

Many thanks to Eliezer and Nate for their courteous and constructive discussion and moderation, and to Rob for putting the transcripts together.

This debate updated me about 15% of the way towards Eliezer’s position, with Eliezer’s arguments about the difficulties of coordinating to ensure alignment responsible for most of that shift. While I don’t find Eliezer’s core intuitions about intelligence too implausible, they don’t seem compelling enough to do as much work as Eliezer argues they do. As in the Foom debate, I think that our object-level discussions were constrained by our different underlying attitudes towards high-level abstractions, which are hard to pin down (let alone resolve).

Given this, I think that the most productive mode of intellectual engagement with Eliezer’s worldview going forward is probably not to continue debating it (since that would likely hit those same underlying disagreements), but rather to try to inhabit it deeply enough to rederive his conclusions and find new explanations of them which then lead to clearer object-level cruxes. I hope that these transcripts shed sufficient light for some readers to be able to do so.