it’s bound to improve in catastrophic manner

I think this is true with quite high probability about an AI that gets high enough, if not otherwise corrigibilized, boosting up to strong superintelligence – this is what it means metaphorically to get “past the atmospheric turbulence layer”.

“High enough” should not be very far above the human level and may be below it; John von Neumann with the ability to run some chains of thought at high serial speed, access to his own source code, and the ability to try branches of himself, seems like he could very likely do this, possibly modulo his concerns about stomping his own utility function making him more cautious.

People noticeably less smart than von Neumann might be able to do it too.

An AI whose components are more modular than a human’s and more locally testable might have an easier time of the whole thing; we can imagine the FOOM getting rolling from something that was in some sense dumber than human.

But the strong prediction is that when you get well above the von Neumann level, why, that is clearly enough, and things take over and go Foom.  The lower you go from that threshold, the less sure I am that it counts as “out of the atmosphere”.  This epistemic humility on my part should not be confused for knowledge of a constraint on the territory that requires AI to go far above humans to Foom.  Just as DL-based AI over the 2010s scaled and generalized much faster and earlier than the picture I argued to Hanson in the Foom debate, reality is allowed to be much more ‘extreme’ than the sure-thing part of this proposition that I defend.

I think I’d expect something like… chess engine slope jumps a bit for Deep Blue, then levels off with increasing excitement, then jumps for the Alpha series? Albeit it’s worth noting that Deepmind’s efforts there were going towards generality rather than raw power; chess was solved to the point of being uninteresting, so they tried to solve chess with simpler code that did more things. I don’t think I do have strong opinions about what the chess trend should look like, vs. the Go trend; I have no memories of people saying the chess trend was breaking upwards or that there was a surprise there.

Incidentally, the highly well-traded financial markets are currently experiencing sharp dips surrounding the Chinese firm of Evergrande, which I was reading about several weeks before this.

I don’t see the basic difference in the kind of reasoning that says “Surely foresightful firms must produce investments well in advance into earlier weaker applications of AGI that will double the economy”, and the reasoning that says “Surely world economic markets and particular Chinese stocks should experience smooth declines as news about Evergrande becomes better-known and foresightful financial firms start to remove that stock from their portfolio or short-sell it”, except that in the latter case there are many more actors with lower barriers to entry than presently exist in the auto industry or semiconductor industry never mind AI.

or if not smooth because of bandwagoning and rational fast actors, then at least the markets should (arguendo) be reacting earlier than they’re reacting now, given that I heard about Evergrande earlier; and they should have options-priced Covid earlier; and they should have reacted to the mortgage market earlier. If even markets there can exhibit seemingly late wild swings, how is the economic impact of AI – which isn’t even an asset market! – forced to be earlier and smoother than that, as a result of wise investing?

There’s just such a vast gap between hopeful reasoning about how various agents and actors should all do the things the speaker finds very reasonable, thereby yielding smooth behavior of the Earth, versus reality.

maybe I’ll try liveblogging some https://sideways-view.com/2018/02/24/takeoff-speeds/ here in the meanwhile

The main disagreement is not about what will happen once we have a superintelligent AI, it’s about what will happen before we have a superintelligent AI. So slow takeoff seems to mean that AI has a larger impact on the world, sooner.

It seems to me to be disingenuous to phrase it this way, given that slow-takeoff views usually imply that AI has a large impact later relative to right now (2021), even if they imply that AI impacts the world “earlier” relative to “when superintelligence becomes reachable”.

“When superintelligence becomes reachable” is not a fixed point in time that doesn’t depend on what you believe about cognitive scaling. The correct graph is, in fact, the one where the “slow” line starts a bit before “fast” peaks and ramps up slowly, reaching a high point later than “fast”. It’s a nice try at reconciliation with the imagined Other, but it fails and falls flat.

This may seem like a minor point, but points like this do add up.

In the fast takeoff scenario, weaker AI systems may have significant impacts but they are nothing compared to the “real” AGI. Whoever builds AGI has a decisive strategic advantage. Growth accelerates from 3%/year to 3000%/year without stopping at 30%/year. And so on.

This again shows failure to engage with the Other’s real viewpoint. My mainline view is that growth stays at 5%/year and then everybody falls over dead in 3 seconds and the world gets transformed into paperclips; there’s never a point with 3000%/year.

There will be a complete 4 year interval in which world output doubles, before the first 1 year interval in which world output doubles.

If we allow that consuming and transforming the solar system over the course of a few days is “the first 1 year interval in which world output doubles”, then I’m happy to argue that there won’t be a 4-year interval with world economic output doubling before then. This, indeed, seems like a massively overdetermined point to me. That said, again, the phrasing is not conducive to conveying the Other’s real point of view.

I believe that before we have incredibly powerful AI, we will have AI which is merely very powerful.

Statements like these are very often “true, but not the way the person visualized them”. Before anybody built the first critical nuclear pile in a squash court at the University of Chicago, was there a pile that was almost but not quite critical? Yes, one hour earlier. Did people already build nuclear systems and experiment with them? Yes, but they didn’t have much in the way of net power output. Did the Wright Brothers build prototypes before the Flyer? Yes, but they weren’t prototypes that flew but 80% slower.

I guarantee you that, whatever the fast takeoff scenario, there will be some way to look over the development history, and nod wisely and say, “Ah, yes, see, this was not unprecedented, here are these earlier systems which presaged the final system!” Maybe you could even look back to today and say that about GPT-3, yup, totally presaging stuff all over the place, great. But it isn’t transforming society because it’s not over the social-transformation threshold.

AlphaFold presaged AlphaFold 2 but AlphaFold 2 is good enough to start replacing other ways of determining protein conformations and AlphaFold is not; and then neither of those has much impacted the real world, because in the real world we can already design a vaccine in a day and the rest of the time is bureaucratic time rather than technology time, and that goes on until we have an AI over the threshold to bypass bureaucracy.

Before there’s an AI that can act while fully concealing its acts from the programmers, there will be an AI (albeit perhaps only 2 hours earlier) which can act while only concealing 95% of the meaning of its acts from the operators.

And that AI will not actually originate any actions, because it doesn’t want to get caught; there’s a discontinuity in the instrumental incentives between expecting 95% obscuration, being moderately sure of 100% obscuration, and being very certain of 100% obscuration.

Before that AI grasps the big picture and starts planning to avoid actions that operators detect as bad, there will be some little AI that partially grasps the big picture and tries to avoid some things that would be detected as bad; and the operators will (mainline) say “Yay what a good AI, it knows to avoid things we think are bad!” or (death with unrealistic amounts of dignity) say “oh noes the prophecies are coming true” and back off and start trying to align it, but they will not be able to align it, and if they don’t proceed anyways to destroy the world, somebody else will proceed anyways to destroy the world.

There is always some step of the process that you can point to which is continuous on some level.

The real world is allowed to do discontinuous things to you anyways.

There is not necessarily a presage of 9/11 where somebody flies a small plane into a building and kills 100 people, before anybody flies 4 big planes into 3 buildings and kills 3000 people; and even if there is some presaging event like that, which would not surprise me at all, the rest of the world’s response to the two cases was evidently discontinuous. You do not necessarily wake up to a news story that is 10% of the news story of 2001/09/11, one year before 2001/09/11, written in 10% of the font size on the front page of the paper.

Physics is continuous but it doesn’t always yield things that “look smooth to a human brain”. Some kinds of processes converge to continuity in strong ways where you can throw discontinuous things in them and they still end up continuous, which is among the reasons why I expect world GDP to stay on trend up until the world ends abruptly; because world GDP is one of those things that wants to stay on a track, and an AGI building a nanosystem can go off that track without being pushed back onto it.

In particular, this means that incredibly powerful AI will emerge in a world where crazy stuff is already happening (and probably everyone is already freaking out).

Like the way they’re freaking out about Covid (itself a nicely smooth process that comes in locally pretty predictable waves) by going doobedoobedoo and letting the FDA carry on its leisurely pace; and not scrambling to build more vaccine factories, now that the rich countries have mostly got theirs? Does this sound like a statement from a history book, or from an EA imagining an unreal world where lots of other people behave like EAs? There is a pleasure in imagining a world where suddenly a Big Thing happens that proves we were right and suddenly people start paying attention to our thing, the way we imagine they should pay attention to our thing, now that it’s attention-grabbing; and then suddenly all our favorite policies are on the table!

You could, in a sense, say that our world is freaking out about Covid; but it is not freaking out in anything remotely like the way an EA would freak out; and all the things an EA would immediately do if an EA freaked out about Covid, are not even on the table for discussion when politicians meet. They have their own ways of reacting. (Note: this is not commentary on hard vs soft takeoff per se, just a general commentary on the whole document seeming to me to… fall into a trap of finding self-congruent things to imagine and imagining them.)

(At Rob’s request I’ll try to keep this brief, but this was an experimental format and some issues cropped up that seem large enough to deserve notes.)

Especially when coming in to the early parts of this dialogue, I had some backed-up hypotheses about “What might be the main sticking point? and how can I address that?” which from the standpoint of a pure dialogue might seem to be causing me to go on digressions, relative to if I was just trying to answer Richard’s own questions.  On reading the dialogue, I notice that this looks evasive or like point-missing, like I’m weirdly not just directly answering Richard’s questions.

Often the questions are answered later, or at least I think they are, though it may not be in the first segment of the dialogue.  But the larger phenomenon is that I came in with some things I wanted to say, and Richard came in asking questions, and there was a minor accidental mismatch there.  It would have looked better if we’d both stated positions first without question marks, say, or if I’d just confined myself to answering questions from Richard.  (This is not a huge catastrophe, but it’s something for the reader to keep in mind as a minor hiccup that showed up in the early parts of experimenting with this new format.)

(Prompted by some later stumbles in attempts to summarize this dialogue.  Summaries seem plausibly a major mode of propagation for a sprawling dialogue like this, and the following request seems like it needs to be very prominent to work – embedded requests later on didn’t work.)

Please don’t summarize this dialogue by saying, “and so Eliezer’s MAIN idea is that” or “and then Eliezer thinks THE KEY POINT is that” or “the PRIMARY argument is that” etcetera.  From my perspective, everybody comes in with a different set of sticking points versus things they see as obvious, and the conversation I have changes drastically depending on that.  In the old days this used to be the Orthogonality Thesis, Instrumental Convergence, and superintelligence being a possible thing at all; today most OpenPhil-adjacent folks have other sticking points instead.

Please transform:

• “Eliezer’s main reply is…” -> “Eliezer replied that…”
• “Eliezer thinks the key point is…” -> “Eliezer’s point in response was…”
• “Eliezer thinks a major issue is…”  -> “Eliezer replied that one issue is…”
• “Eliezer’s primary argument against this is…” -> “Eliezer tried the counterargument that…”
• “Eliezer’s main scenario for this is…” -> “In a conversation in September of 2021, Eliezer sketched a hypothetical where…”

Note also that the transformed statements say what you observed, whereas the untransformed statements are (often incorrect) inferences about my latent state of mind.

(Though “distinguishing relatively unreliable inference from more reliable observation” is not necessarily the key idea here or the one big reason I’m asking for this.  That’s just one point I tried making – one argument that I hope might help drive home the larger thesis.)

Hi and welcome all.  My name is Eliezer and I think alignment is really actually quite extremely difficult.  Some people seem to not think this!  It’s an important issue so ought to be resolved somehow, which we can hopefully fully do today.  (I will however want to take a break after the first 90 minutes, if it goes that far and if Ngo is in sleep-cycle shape to continue past that.)

On my model of the Other Person, a lot of times when somebody thinks alignment shouldn’t be that hard, they think there’s some particular thing you can do to align an AGI, which isn’t that hard, and their model is missing one of the foundational difficulties for why you can’t do (easily or at all) one step of their procedure.  So one of my own conversational processes might be to poke around looking for a step that the other person doesn’t realize is hard.  That said, I’ll try to directly answer your own question first.

I think that after AGI becomes possible at all and then possible to scale to dangerously superhuman levels, there will be, in the best-case scenario where a lot of other social difficulties got resolved, a 3-month to 2-year period where only a very few actors have AGI, meaning that it was socially possible for those few actors to decide to not just scale it to where it automatically destroys the world.

During this step, if humanity is to survive, somebody has to perform some feat that causes the world to not be destroyed in 3 months or 2 years when too many actors have access to AGI code that will destroy the world if its intelligence dial is turned up. This requires that the first actor or actors to build AGI, be able to do something with that AGI which prevents the world from being destroyed; if it didn’t require superintelligence, we could go do that thing right now, but no such human-doable act apparently exists so far as I can tell.

So we want the least dangerous, most easily aligned thing-to-do-with-an-AGI, but it does have to be a pretty powerful act to prevent the automatic destruction of Earth after 3 months or 2 years. It has to “flip the gameboard” rather than letting the suicidal game play out. We need to align the AGI that performs this pivotal act, to perform that pivotal act without killing everybody.

Parenthetically, no act powerful enough and gameboard-flipping enough to qualify is inside the Overton Window of politics, or possibly even of effective altruism, which presents a separate social problem. I usually dodge around this problem by picking an exemplar act which is powerful enough to actually flip the gameboard, but not the most alignable act because it would require way too many aligned details: Build self-replicating open-air nanosystems and use them (only) to melt all GPUs.

Since any such nanosystems would have to operate in the full open world containing lots of complicated details, this would require tons and tons of alignment work, is not the pivotal act easiest to align, and we should do some other thing instead. But the other thing I have in mind is also outside the Overton Window, just like this is. So I use “melt all GPUs” to talk about the requisite power level and the Overton Window problem level, both of which seem around the right levels to me, but the actual thing I have in mind is more alignable; and this way, I can reply to anyone who says “How dare you?!” by saying “Don’t worry, I don’t actually plan on doing that.”

This sounds to me like it requires extreme levels of alignment and operating in extremely dangerous regimes, such that, if you could do that, it would seem much more sensible to do some other pivotal act first, using a lower level of alignment tech.

In particular, I would hope that – in unlikely cases where we survive at all – we were able to survive by operating a superintelligence only in the lethally dangerous, but still less dangerous, regime of “engineering nanosystems”.

Whereas “solve alignment for us” seems to require operating in the even more dangerous regimes of “write AI code for us” and “model human psychology in tremendous detail”.

Every AI output effectuates outcomes in the world.  If you have a powerful unaligned mind hooked up to outputs that can start causal chains that effectuate dangerous things, it doesn’t matter whether the comments on the code say “intellectual problems” or not.

The danger of “solving an intellectual problem” is when it requires a powerful mind to think about domains that, when solved, render very cognitively accessible strategies that can do dangerous things.

I expect the first alignment solution you can actually deploy in real life, in the unlikely event we get a solution at all, looks like 98% “don’t think about all these topics that we do not absolutely need and are adjacent to the capability to easily invent very dangerous outputs” and 2% “actually think about this dangerous topic but please don’t come up with a strategy inside it that kills us”.

I would certainly have learned very new and very exciting facts about intelligence, facts which indeed contradict my present model of how intelligences liable to be discovered by present research paradigms work, if you showed me… how can I put this in a properly general way… that problems I thought were about searching for states that get fed into a result function and then a result-scoring function, such that the input gets an output with a high score, were in fact not about search problems like that. I have sometimes given more specific names to this problem setup, but I think people have become confused by the terms I usually use, which is why I’m dancing around them.

In particular, just as I have a model of the Other Person’s Beliefs in which they think alignment is easy because they don’t know about difficulties I see as very deep and fundamental and hard to avoid, I also have a model in which people think “why not just build an AI which does X but not Y?” because they don’t realize what X and Y have in common, which is something that draws deeply on having deep models of intelligence. And it is hard to convey this deep theoretical grasp. But you can also see powerful practical hints that these things are much more correlated than, eg, Robin Hanson was imagining during the FOOM debate, because Robin did not think something like GPT-3 should exist; Robin thought you should need to train lots of specific domains that didn’t generalize. I argued then with Robin that it was something of a hint that humans had visual cortex and cerebellar cortex but not Car Design Cortex, in order to design cars. Then in real life, it proved that reality was far to the Eliezer side of Eliezer on the Eliezer-Robin axis, and things like GPT-3 were built with less architectural complexity and generalized more than I was arguing to Robin that complex architectures should generalize over domains.

The metaphor I sometimes use is that it is very hard to build a system that drives cars painted red, but is not at all adjacent to a system that could, with a few alterations, prove to be very good at driving a car painted blue.  The “drive a red car” problem and the “drive a blue car” problem have too much in common.  You can maybe ask, “Align a system so that it has the capability to drive red cars, but refuses to drive blue cars.”  You can’t make a system that is very good at driving red-painted cars, but lacks the basic capability to drive blue-painted cars because you never trained it on that.  The patterns found by gradient descent, by genetic algorithms, or by other plausible methods of optimization, for driving red cars, would be patterns very close to the ones needed to drive blue cars.  When you optimize for red cars you get the blue car capability whether you like it or not.

If it were possible to perform some pivotal act that saved the world with an AI that just made progress on proving mathematical theorems, without, eg, needing to explain those theorems to humans, I’d be extremely interested in that as a potential pivotal act. We wouldn’t be out of the woods, and I wouldn’t actually know how to build an AI like that without killing everybody, but it would immediately trump everything else as the obvious line of research to pursue.

Parenthetically, there is very very little which my model of intelligence rules out. I think we all die because we cannot do certain dangerous things correctly, on the very first try in the dangerous regimes where one mistake kills you, and do them before proliferation of much easier technologies kills us. If you have the Textbook From 100 Years In The Future that gives the simple robust solutions for everything, that actually work, you can write a superintelligence that thinks 2 + 2 = 5 because the Textbook gives the methods for doing that which are simple and actually work in practice in real life.

(The Textbook has the equivalent of “use ReLUs instead of sigmoids” everywhere, and avoids all the clever-sounding things that will work at subhuman levels and blow up when you run them at superintelligent levels.)

I put to you that there is a predictable bias in your estimates, where you don’t know about the Deep Stuff that is required to prove theorems, so you imagine that certain cognitive capabilities are more disjoint than they actually are.  If you knew about the things that humans are using to reuse their reasoning about chipped handaxes and other humans, to prove math theorems, you would see it as more plausible that proving math theorems would generalize to chipping handaxes and manipulating humans.

GPT-3 is a… complicated story, on my view of it and intelligence.  We’re looking at an interaction between tons and tons of memorized shallow patterns.  GPT-3 is very unlike the way that natural selection built humans.

My current suspicion is that Stack More Layers alone is not going to take us to GPT-6 which is a true AGI; and this is because of the way that GPT-3 is, in your own terminology, “not agentic”, and which is, in my terminology, not having gradient descent on GPT-3 run across sufficiently deep problem-solving patterns.

I agree.

This seems indisputable.

I continue to nod.

So here’s the part that I agree with: If there were one theorem only mildly far out of human reach, like proving the ABC Conjecture (if you think it hasn’t already been proven), and providing a machine-readable proof of this theorem would immediately save the world – say, aliens will give us an aligned superintelligence, as soon as we provide them with this machine-readable proof – then there would exist a plausible though not certain road to saving the world, which would be to try to build a shallow mind that proved the ABC Conjecture by memorizing tons of relatively shallow patterns for mathematical proofs learned through self-play; without that system ever abstracting math as deeply as humans do, but the sheer width of memory and sheer depth of search sufficing to do the job. I am not sure, to be clear, that this would work. But my model of intelligence does not rule it out.

Parts I disagree with: That “help us solve alignment” bears any significant overlap with “provide us a machine-readable proof of the ABC Conjecture without thinking too deeply about it”. That humans want to take over the world only because it resembles things we evolved to do.

Alas, eliminating 5 reasons why something would go wrong doesn’t help much if there’s 2 remaining reasons something would go wrong that are much harder to eliminate!

I think this is one of the very rare cases where the intelligence difference between “village idiot” and “Einstein”, which I’d usually see as very narrow, makes a structural difference! I think you can get some outputs from a village-idiot-level AGI, which got there by training on domains exclusively like math, and this will proooobably not destroy the world (if you were right about that, about what was going on inside). I have more concern about the Einstein level.

I think there’s a certain broad sense in which I agree with the literal truth of what you just said. You will systematically overestimate how much easier, or how far you can push the science part without getting the taking-over-the-world part, for as long as your model is ignorant of what they have in common.

I feel like I haven’t had much luck with trying to explain that on previous occasions. Not to you, to others too.

There are shallow topics like why p-zombies can’t be real and how quantum mechanics works and why science ought to be using likelihood functions instead of p-values, and I can barely explain those to some people, but then there are some things that are apparently much harder to explain than that and which defeat my abilities as an explainer.

That’s why I’ve been trying to point out that, even if you don’t know the specifics, there’s an estimation bias that you can realize should exist in principle.

Of course, I also haven’t had much luck in saying to people, “Well, even if you don’t know the truth about X that would let you see Y, can you not see by abstract reasoning that knowing any truth about X would predictably cause you to update in the direction of Y” – people don’t seem to actually internalize that much either. Not you, other discussions.

Yup. If you didn’t know about visual cortex and auditory cortex, or about eyes and ears, you would assume much more that any sentience ought to both see and hear.

If there’s a pivotal act you can get just by supreme acts of pattern recognition, that’s right up there with “pivotal act composed solely of math” for things that would obviously instantly become the prime direction of research.

(One occasionally-argued line of research can be summarized from a certain standpoint as “how about a pivotal act composed entirely of predicting text” and to this my reply is “you’re trying to get fully general AGI capabilities by predicting text that is about deep / ‘agentic’ reasoning, and that doesn’t actually help”.)

Human math is very much about goals. People want to prove subtheorems on the way to proving theorems. We might be able to make a different kind of mathematician that works more like GPT-3 in the dangerously inscrutable parts that are all noninspectable vectors of floating-point numbers, but even there you’d need some Alpha-Zero-like outer framework to supply the direction of search.

That outer framework might be able to be powerful enough without being reflective, though. So it would plausibly be much easier to build a mathematician that was capable of superhuman formal theorem-proving but not agentic. The reality of the world might tell us “lolnope” but my model of intelligence doesn’t mandate that. That’s why, if you gave me a pivotal act composed entirely of “output a machine-readable proof of this theorem and the world is saved”, I would pivot there! It actually does seem like it would be a lot easier!

Remember, a lot of my concern is about what happens first, especially if it happens soon enough that future AGI bears any resemblance whatsoever to modern ML; not about what can be done in principle.

I think that obvious-to-me future outgrowths of modern ML paradigms are extremely liable to, if they can learn how to do sufficiently superhuman X, generalize to taking over the world. How fast this happens does depend on X. It would plausibly happen relatively slower (at higher levels) with theorem-proving as the X, and with architectures that carefully stuck to gradient-descent-memorization over shallow network architectures to do a pattern-recognition part with search factored out (sort of, this is not generally safe, this is not a general formula for safe things!); rather than imposing anything like the genetic bottleneck you validly pointed out as a reason why humans generalize. Profitable X, and all X I can think of that would actually save the world, seem much more problematic.

Yup. This is my lunch break (actually my first-food-of-day break on a 600-calorie diet) so I can be back in 45min if you’re still up for that.

I’m also cheerful about spectators posting suggestions or comments during the break.

I seem to be done early if people (mainly Richard) want to resume in 10min (30m break)

I should also say out loud that I’ve been working a bit with Ajeya on making an attempt to convey the intuitions behind there being deep patterns that generalize and are liable to be learned, which covered a bunch of ground, taught me how much ground there was, and made me relatively more reluctant to try to re-cover the same ground in this modality.

I could still try briefly and see what happens.

I’m not sure I could summarize your position in a non-straw way. To me there’s a huge visible distance between “solve alignment for us” and “output machine-readable proofs of theorems” where I can’t give a good account of why you think talking about the latter would tell us much about the former. I don’t know what other pivotal act you think might be easier.

Can you predict in advance why Eliezer thinks “solving scientific problems” is significantly thornier? (Where alignment is like totally not “a particularly hard example of a scientific problem” except in the sense that it has science in it at all; which is maybe the real crux; but also a more difficult issue.)

That’s one factor. Should I state the other big one or would you rather try to state it first?

That’s one way of stating it. The way I’d put it is that it’s about making up hypotheses about the real world.

Like, the real world is then a thing that the AI is modeling, at all.

Factor 3: On many interpretations of doing science, you would furthermore need to think up experiments. That’s planning, value-of-information, search for an experimental setup whose consequences distinguish between hypotheses (meaning you’re now searching for initial setups that have particular causal consequences).

There’s a problem of inferring the causes of sensory experience in cognition-that-does-science. (Which, in fact, also appears in the way that humans do math, and is possibly inextricable from math in general; but this is an example of the sort of deep model that says “Whoops I guess you get science from math after all”, not a thing that makes science less dangerous because it’s more like just math.)

You can build an AI that only ever drives red cars, and which, at no point in the process of driving a red car, ever needs to drive a blue car in order to drive a red car. That doesn’t mean its red-car-driving capabilities won’t be extremely close to blue-car-driving capabilities if at any point the internal cognition happens to get pointed towards driving a blue car.

The fact that there’s a deep car-driving pattern which is the same across red cars and blue cars doesn’t mean that the AI has ever driven a blue car, per se, or that it has to drive blue cars to drive red cars. But if blue cars are fire, you sure are playing with that fire.

So there’s separate questions about “does an AGI inseparably need to model itself inside the world to do science” and “did we build something that would be very close to modeling itself, and could easily stumble across that by accident somewhere in the inscrutable floating-point numbers, especially if that was even slightly useful for solving the outer problems”.

If you’re trying to build an AI that literally does science only to observations collected without the AI having had a causal impact on those observations, that’s legitimately “more dangerous than math but maybe less dangerous than active science”.

You might still stumble across an active scientist because it was a simple internal solution to something, but the outer problem would be legitimately stripped of an important structural property the same way that pure math not describing Earthly objects is stripped of important structural properties.

And of course my reaction again is, “There is no pivotal act which uses only that cognitive capability.”

The amount doesn’t seem like the key thing, nearly so much as what underlying facilities you need to do whatever amount of it you need.

If you go back to the 16th century and ask for just one mRNA vaccine, that’s not much of a difference from asking for a million hundred of them.

If you have reduced a pivotal act to “look over the data from this hadron collider you neither built nor ran yourself”, that really is a structural step down from “do science” or “build a nanomachine”. But I can’t see any pivotal acts like that, so is that question much of a crux?

If there’s intermediate steps they might be described in my native language like “reason about causal impacts across only this one preprogrammed domain which you didn’t learn in a general way, in only this part of the cognitive architecture that is separable from the rest of the cognitive architecture”.

What is “shallow” to you?

So it’s memorized a ton of shallow causal-impact-inducing patterns from a large dataset, and this can be verified by, for example, presenting it with an example mildly outside the dataset and watching it fail, which we think will confirm our hypothesis that it didn’t learn any deep ways of solving that dataset.

Eg, it wouldn’t surprise us at all if GPT-4 had learned to predict “27 * 18” but not “what is the area of a rectangle 27 meters by 18 meters”… is what I’d like to say, but Codex sure did demonstrate those two were kinda awfully proximal.

There’s a set of obvious-to-me tactics for doing a pivotal act with minimal danger, which I do not think collectively make the problem safe, and one of these sets of tactics is indeed “Put a limit on the ‘attention window’ or some other internal parameter, ramp it up slowly, don’t ramp it any higher than you needed to solve the problem.”

(Any time one imagines a specific tactic of this kind, if one has the security mindset, one can also imagine all sorts of ways it might go wrong; for example, an attention window can be defeated if there’s any aspect of the attended data or the internal state that ended up depending on past events in a way that leaked info about them. But, depending on how much superintelligence you were throwing around elsewhere, you could maybe get away with that, some of the time.)

I’ll observe that TransformerXL built an attention window that generalized, trained it on I think 380 tokens or something like that, and then found that it generalized to 4000 tokens or something like that.

Having observed one order of magnitude, I would personally not be surprised by two orders of magnitude either, after seeing that.

I have a sense that this is all circling back to the question, “But what is it we do with the intelligence thus weakened?” If you can save the world using a rock, I can build you a very safe rock.

I mostly expect – albeit with some chance that reality says “So what?” to me and surprises me, because it is not as solidly determined as some other things – that we do not hang around very long in the “weirdly ~human AGI” phase before we get into the “if you crank up this AGI it destroys the world” phase. Less than 5 years, say, to put numbers on things.

It would not surprise me in the least if the world ends before self-driving cars are sold on the mass market. On some quite plausible scenarios which I think have >50% of my probability mass at the moment, research AGI companies would be able to produce prototype car-driving AIs if they spent time on that, given the near-world-ending tech level; but there will be Many Very Serious Questions about this relatively new unproven advancement in machine learning being turned loose on the roads. And their AGI tech will gain the property “can be turned up to destroy the world” before Earth gains the property “you’re allowed to sell self-driving cars on the mass market” because there just won’t be much time.

On other scenarios, of course, self-driving becomes possible by limited AI well before things start to break (further) on AGI. And on some scenarios, the way you got to AGI was via some breakthrough that is already scaling pretty fast, so by the time you can use the tech to get self-driving cars, that tech already ends the world if you turn up the dial, or that event follows very swiftly.

Running it with larger bounds on the for loops, over more GPUs, to be concrete about it.

I do not quite think that gradient descent on Stack More Layers alone – as used by OpenAI for GPT-3, say, and as opposed to Deepmind which builds more complex artifacts like Mu Zero or AlphaFold 2 – is liable to be the first path taken to AGI. I am reluctant to speculate more in print about clever ways to AGI, and I think any clever person out there will, if they are really clever and not just a fancier kind of stupid, not talk either about what they think is missing from Stack More Layers or how you would really get AGI. That said, the way that you cannot just run GPT-3 at a greater search depth, the way you can run Mu Zero at a greater search depth, is part of why I think that AGI is not likely to look exactly like GPT-3; the thing that kills us is likely to be a thing that can get more dangerous when you turn up a dial on it, not a thing that intrinsically has no dials that can make it more dangerous.

Well, obviously, in the limit of alignment not being accessible to our civilization, and my successfully building a model weaker than reality which nonetheless correctly rules out alignment being accessible to our civilization, I could spend the rest of my short remaining lifetime arguing with people whose models are weak enough to induce some area of ignorance where for all they know you could align a thing. But that is predictably how conversations go in possible worlds where the Earth is doomed; so somebody wiser on the meta-level, though also ignorant on the object-level, might prefer to ask: “Where do you think your knowledge, rather than your ignorance, says that alignment ought to be doable and you will be surprised if it is not?”

We could talk about that, if you think it’s a crux. Though I’m also not thinking that this whole conversation gets done in a day, so maybe for publishability reasons we should try to focus more on one line of discussion?

But I do think that lots of people get their optimism by supposing that the world can be saved by doing less dangerous things with an AGI. So it’s a big ol’ crux of mine on priors.

(Though it’s something of a restatement, a reason I’m not going into “my intuitions about how cognition works” is that past experience has led me to believe that conveying this info in a form that the Other Mind will actually absorb and operate, is really quite hard and takes a long discussion, relative to my current abilities to Actually Explain things; it is the sort of thing that might take doing homework exercises to grasp how one structure is appearing in many places, as opposed to just being flatly told that to no avail, and I have not figured out the homework exercises.)

I’m cheerful about hearing your own claims about cognition and disagreeing with them.

(“If that were true,” he thought at once, “bureaucracies and books of regulations would be a lot more efficient than they are in real life.”)

Heh. You mean of comparable strength to the human instinct to explicitly maximize inclusive genetic fitness?

Even in that paradigm, except insofar as you expect gradient descent to work very differently from gene-search optimization – which, admittedly, it does – when you optimize really hard on a thing, you get contextual correlates to it, not the thing you optimized on.

This is of course one of the Big Fundamental Problems that I expect in alignment.

That said, I also see basic structural reasons why paperclips would be much easier to train than “obedience”, even if we could magically instill simple inner desires that perfectly reflected the simple outer algorithm we saw ourselves as running over many particular instances of a loss function.

well, first of all, why is a book of regulations so much more unwieldy than a hunter-gatherer?

if deontology is just as good as consequentialism, y’know.

(do you want to try replying or should I just say?)

I reply that reality is complicated, so when you pump a simple goal through complicated reality you get complicated behaviors required to achieve the goal. If you think of reality as a complicated function Input->Probability(Output), then even to get a simple Output or a simple partition on Output or a high expected score in a simple function over Output, you may need very complicated Input.

Humans don’t trust each other. They imagine, “Well, if I just give this bureaucrat a goal, perhaps they won’t reason honestly about what it takes to achieve that goal! Oh no! Therefore I will instead, being the trustworthy and accurate person that I am, reason myself about constraints and requirements on the bureaucrat’s actions, such that, if the bureaucrat obeys these regulations, I expect the outcome of their action will be what I want.”

But (compared to a general intelligence that observes and models complicated reality and does its own search to pick actions) an actually-effective book of regulations (implemented by some nonhuman mind with a large enough and perfect enough memory to memorize it) would tend to involve a (physically unmanageable) vast number of rules saying “if you observe this, do that” to follow all the crinkles of complicated reality as it can be inferred from observation.

(As a side note: do you have a rough guess for when your work with Ajeya will be made public? If it’s still a while away, I’m wondering whether it’s still useful to have a rough outline of these intuitions even if it’s in a form that very few people will internalise)

Plausibly useful, but not to be attempted today, I think?

(We are now theoretically in overtime, which is okay for me, but for you it is 11:30pm (I think?) and so it is on you to call when to halt, now or later.)

Yup, and a place to resume next time might be why I think “obedience” is unnatural compared to “paperclips” – though that is a thing that probably requires taking that stab at what underlies surface competencies.

Not yet afaik, and I don’t want to point you to Ajeya’s stuff even if she were ok with that, because then this in-context conversation won’t make sense to others.

could try Tuesday at this same time – though I may be in worse shape for dietary reasons, still, seems worth trying.

Wednesday?

let’s call it tentatively for that

Thanks, Richard!

will not be good enough

Are not presently on course to be good enough, missing by not a little.  “Will not be good enough” is literally declaring for lying down and dying.

will [be very far from adequate]

Same problem as the last time I commented.  I am not making an unconditional prediction about future failure as would be implied by the word “will”.  Conditional on current courses of action or their near neighboring courses, we seem to be well over an order of magnitude away from surviving, unless a miracle occurs.  It’s still in the end a result of people doing what they seem to be doing, not an inevitability.

Yes.

There’s a deep connection between solving intellectual problems and taking over the world

There’s a deep connection by default between chipping flint handaxes and taking over the world, if you happen to learn how to chip handaxes in a very general way.  “Intellectual” problems aren’t special in this way.  And maybe you could avert the default, but that would take some work and you’d have to do it before easier default ML techniques destroyed the world.

I don’t think a female Eliezer Yudkowsky doesn’t try to save / optimize / takeover the world.  Men may do that for nonsmart reasons; smart men and women follow the same reasoning when they are smart enough.  Eg Anna Salamon and many others.

(I am here.)

made a couple of comments about phrasings in the doc

So, from my perspective, there’s this thing where… it’s really quite hard to teach certain general points by talking at people, as opposed to more specific points. Like, they’re trying to build a perpetual motion machine, and even if you can manage to argue them into believing their first design is wrong, they go looking for a new design, and the new design is complicated enough that they can no longer be convinced that they’re wrong because they managed to make a more complicated error whose refutation they couldn’t keep track of anymore.

Teaching people to see an underlying structure in a lot of places is a very hard thing to teach in this way. Richard Feynman gave an example of the mental motion in his story that ends “Look at the water!”, where people learned in classrooms about how “a medium with an index” is supposed to polarize light reflected from it, but they didn’t realize that sunlight coming off of water would be polarized. My guess is that doing this properly requires homework exercises; and that, unfortunately from my own standpoint, it happens to be a place where I have extra math talent, the same way that eg Marcello is more talented at formally proving theorems than I happen to be; and that people without the extra math talent, have to do a lot more exercises than I did, and I don’t have a good sense of which exercises to give them.

There’s a general insight you can have about how arithmetic is commutative, and for some people you can show them 1 + 2 = 2 + 1 and their native insight suffices to generalize over the 1 and the 2 to any other numbers you could put in there, and they realize that strings of numbers can be rearranged and all end up equivalent. For somebody else, when they’re a kid, you might have to show them 2 apples and 1 apple being put on the table in a different order but ending up with the same number of apples, and then you might have to show them again with adding up bills in different denominations, in case they didn’t generalize from apples to money. I can actually remember being a child young enough that I tried to add 3 to 5 by counting “5, 6, 7” and I thought there was some clever enough way to do that to actually get 7, if you tried hard.

Being able to see “consequentialism” is like that, from my perspective.

I don’t know what homework exercises to give people to make them able to see “consequentialism” all over the place, instead of inventing slightly new forms of consequentialist cognition and going “Well, now that isn’t consequentialism, right?”

Trying to say “searching for states that get fed into an input-result function and then a result-scoring function” was one attempt of mine to describe the dangerous thing in a way that would maybe sound abstract enough that people would try to generalize it more.

Ok, so, part of the problem is that… before you do enough homework exercises for whatever your level of talent is (and even I, at one point, had done little enough homework that I thought there might be a clever way to add 3 and 5 in order to get to 7), you tend to think that only the very crisp formal thing that’s been presented to you, is the “real” thing.

Why would your engine have to obey the laws of thermodynamics? You’re not building one of those Carnot engines you saw in the physics textbook!

Humans contain fragments of consequentialism, or bits and pieces whose interactions add up to partially imperfectly shadow consequentialism, and the critical thing is being able to see that the reason why humans’ outputs ‘work’, in a sense, is because these structures are what is doing the work, and the work gets done because of how they shadow consequentialism and only insofar as they shadow consequentialism.

Put a human in one environment, it gets food. Put a human in a different environment, it gets food again. Wow, different initial conditions, same output! There must be things inside the human that, whatever else they do, are also along the way somehow effectively searching for motor signals such that food is the end result!

More like… from my perspective, even after I talk people out of one specific perpetual motion machine being possible, they go off and try to invent a different, more complicated perpetual motion machine.

And I am not sure what to do about that. It has been going on for a very long time from my perspective.

In the end, a lot of what people got out of all that writing I did, was not the deep object-level principles I was trying to point to – they did not really get Bayesianism as thermodynamics, say, they did not become able to see Bayesian structures any time somebody sees a thing and changes their belief. What they got instead was something much more meta and general, a vague spirit of how to reason and argue, because that was what they’d spent a lot of time being exposed to over and over and over again in lots of blog posts.

Maybe there’s no way to make somebody understand why corrigibility is “unnatural” except to repeatedly walk them through the task of trying to invent an agent structure that lets you press the shutdown button (without it trying to force you to press the shutdown button), and showing them how each of their attempts fails; and then also walking them through why Stuart Russell’s attempt at moral uncertainty produces the problem of fully updated (non-)deference; and hope they can start to see the informal general pattern of why corrigibility is in general contrary to the structure of things that are good at optimization.

Except that to do the exercises at all, you need them to work within an expected utility framework. And then they just go, “Oh, well, I’ll just build an agent that’s good at optimizing things but doesn’t use these explicit expected utilities that are the source of the problem!”

And then if I want them to believe the same things I do, for the same reasons I do, I would have to teach them why certain structures of cognition are the parts of the agent that are good at stuff and do the work, rather than them being this particular formal thing that they learned for manipulating meaningless numbers as opposed to real-world apples.

And I have tried to write that page once or twice (eg “coherent decisions imply consistent utilities“) but it has not sufficed to teach them, because they did not even do as many homework problems as I did, let alone the greater number they’d have to do because this is in fact a place where I have a particular talent.

I don’t know how to solve this problem, which is why I’m falling back on talking about it at the meta-level.

That (Nate’s suggestion) is probably the correct thing to do. I name it out loud because sometimes being told about the meta-problem actually does help on the object problem. It seems to help me a lot and others somewhat less, but it does help others at all, for many others.

So, do you have a particular question you would ask about input-seeking cognitions? I did try to say why I mentioned those at all (it’s a different road to Rome on “consequentialism”).

Yup! And it is no coincidence that there are no whole animals formed entirely out of nothing but a visual cortex!

Also by the time you get to like the temporal lobes or something, there is probably some significant amount of “what could I be seeing that would produce this visual field?” that is searching through hypothesis-space for hypotheses with high plausibility scores, and for sure at the human level, humans will start to think, “Well, could I be seeing this? No, that theory has the following problem. How could I repair that theory?” But it is plausible that there is no low-level analogue of this in a monkey’s temporal cortex; and even more plausible that the parts of the visual cortex, if any, which do anything analogous to this, are doing it in a relatively local and definitely very domain-specific way.

Oh, that’s the cerebellum and motor cortex and so on, if we’re talking about a cat or whatever. They have to find motor plans that result in their catching the mouse.

Just because the visual cortex isn’t (obviously) running a search doesn’t mean the rest of the animal isn’t running any searches.

(On the meta-level, I notice myself hiccuping “But how could you not see that when looking at a cat?” and wondering what exercises would be required to teach that.)

It doesn’t directly parse in my ontology because (a) I don’t know what you mean by ‘high-level’ and (b) whole Cartesian agents can be viewed as functions, that doesn’t mean all agents can be viewed as non-searching pattern-recognizers.

That said, all parts of the cerebral cortex have surprisingly similar morphology, so it wouldn’t be at all surprising if the motor cortex is doing something similar to visual cortex. (The cerebellum, on the other hand…)

Still less of a native distinction in my ontology, but there’s an informal thing it can sort of wave at, and I can hopefully take that as understood and run with it.

The part between the cat’s eyes seeing the mouse, and the part where the cat’s limbs move to catch the mouse, is the whole cat-agent. The whole cat agent sure is a baby consequentialist / searches for mouse-catching motor patterns / gets similarly high-scoring end results even as you vary the environment.

The visual cortex is a particular part of this system-viewed-as-a-feedforward-function that is, plausibly, by no means surely, either not very searchy, or does only small local visual-domain-specific searches not aimed per se at catching mice; it has the epistemic nature rather than the planning nature.

Then from one perspective you could reason that “well, most of the consequentialism is in the remaining cat after visual cortex has sent signals onward”. And this is in general a dangerous mode of reasoning that is liable to fail in, say, inspecting every particular neuron for consequentialism and not finding it; but in this particular case, there are significantly more consequentialist parts of the cat than the visual cortex, so I am okay running with it.

Disagree, I’m rusty on my neuroanatomy but I think the motor cortex may send signals on to the cerebellum rather than the other way around.

(I may also disagree with the actual underlying notion you’re trying to hint at, so possibly not just a “well include the cerebellum then” issue, but I think I should let you respond first.)

So since cats are not (obviously) (that I have read about) cross-domain consequentialists with imaginations, their consequentialism is in bits and pieces of consequentialism embedded in them all over by the more purely pseudo-consequentialist genetic optimization loop that built them.

A cat who fails to catch a mouse may then get little bits and pieces of catbrain adjusted all over.

And then those adjusted bits and pieces get a pattern lookup later.

Why do these pattern-lookups with no obvious immediate search element, all happen to point towards the same direction of catching the mouse? Because of the past causal history about how what gets looked up, which was tweaked to catch the mouse.

So it is legit harder to point out “the consequentialist parts of the cat” by looking for which sections of neurology are doing searches right there. That said, to the extent that the visual cortex does not get tweaked on failure to catch a mouse, it’s not part of that consequentialist loop either.

And yes, the same applies to humans, but humans also do more explicitly searchy things and this is part of the story for why humans have spaceships and cats do not.

In biological agents we’ve got evolution + local evolved system-rules that in the past promoted genetic fitness. Two kinds of local rules like this are “operant-conditioning updates from success or failure” and “search through visualized plans”. I wouldn’t characterize these two kinds of rules as “levels”.

Planning is one way to succeed at search. I think for purposes of understanding alignment difficulty, you want to be thinking on the level of abstraction where you see that in some sense it is the search itself that is dangerous when it’s a strong enough search, rather than the danger seeming to come from details of the planning process.

One of my early experiences in successfully generalizing my notion of intelligence, what I’d later verbalize as “computationally efficient finding of actions that produce outcomes high in a preference ordering”, was in writing an (unpublished) story about time-travel in which the universe was globally consistent.

The requirement of global consistency, the way in which all events between Paradox start and Paradox finish had to map the Paradox’s initial conditions onto the endpoint that would go back and produce those exact initial conditions, ended up imposing strong complicated constraints on reality that the Paradox in effect had to navigate using its initial conditions. The time-traveler needed to end up going through certain particular experiences that would produce the state of mind in which he’d take the actions that would end up prodding his future self elsewhere into having those experiences.

The Paradox ended up killing the people who built the time machine, for example, because they would not otherwise have allowed that person to go back in time, or kept the temporal loop open that long for any other reason if they were still alive.

Just having two examples of strongly consequentialist general optimization in front of me – human intelligence, and evolutionary biology – hadn’t been enough for me to properly generalize over a notion of optimization. Having three examples of homework problems I’d worked – human intelligence, evolutionary biology, and the fictional Paradox – caused it to finally click for me.

[Yudkowsky][12:10][12:11]

I would later try to tell people to “imagine a paperclip maximizer as not being a mind at all, imagine it as a kind of malfunctioning time machine that spits out outputs which will in fact result in larger numbers of paperclips coming to exist later”. I don’t think it clicked because people hadn’t done the same homework problems I had, and didn’t have the same “Aha!” of realizing how part of the notion and danger of intelligence could be seen in such purely material terms.

But the convergent instrumental strategies, the anticorrigibility, these things are contained in the true fact about the universe that certain outputs of the time machine will in fact result in there being lots more paperclips later. What produces the danger is not the details of the search process, it’s the search being strong and effective at all. The danger is in the territory itself and not just in some weird map of it; that building nanomachines that kill the programmers will produce more paperclips is a fact about reality, not a fact about paperclip maximizers!

Yup! Alas, the story was written in 2002-2003 when I was a worse writer and the real story that inspired the Outcome Pump never did get published.

In one sense, they are! A paperclip-maximizing superintelligence is nowhere near as powerful as a paperclip-maximizing time machine. The time machine can do the equivalent of buying winning lottery tickets from lottery machines that have been thermodynamically randomized; a superintelligence can’t, at least not directly without rigging the lottery or whatever.

But a paperclip-maximizing strong general superintelligence is epistemically and instrumentally efficient, relative to you, or to me. Any time we see it can get at least X paperclips by doing Y, we should expect that it gets X or more paperclips by doing Y or something that leads to even more paperclips than that, because it’s not going to miss the strategy we see.

So in that sense, searching our own brains for how a time machine would get paperclips, asking ourselves how many paperclips are in principle possible and how they could be obtained, is a way of getting our own brains to consider lower bounds on the problem without the implicit stupidity assertions that our brains unwittingly use to constrain story characters. Part of the point of telling people to think about time machines instead of superintelligences was to get past the ways they imagine superintelligences being stupid. Of course that didn’t work either, but it was worth a try.

I don’t think that’s quite what you were asking about, but I want to give you a chance to see if you want to rephrase anything before I try to answer your me-reformulated questions.

It depends on how hard you optimize! And whether gradient descent on a particular system can even successfully optimize that hard! Many current AIs are trained by gradient descent and yet not superintelligences at all.

But the answer is that some problems are difficult in that they require solving lots of subproblems, and an easy way to solve all those subproblems is to use patterns which collectively have some coherence and overlap, and the coherence within them generalizes across all the subproblems. Lots of search orderings will stumble across something like that before they stumble across separate solutions for lots of different problems.

I suspect that you cannot get this out of small large amounts of gradient descent on small large layered transformers, and therefore I suspect that GPT-N does not approach superintelligence before the world is ended by systems that look differently, but I could be wrong about that.

I am confused by what you think it means to have an “epistemic subsystem” that “makes plans much better than any human’s”. If it searches paths through time and selects high-scoring ones for output, what makes it “epistemic”?

If it can in fact do the thing that a paperclipping time machine does, what makes it any safer than a paperclipping time machine because we called it “epistemic” or by some other such name?

By what criterion is it selecting the plans that humans look at?

Why did it make a difference that its output was fed through the causal systems called humans on the way to the causal systems called protein synthesizers or the Internet or whatever? If we build a superintelligence to design nanomachines, it makes no obvious difference to its safety whether it sends DNA strings directly to a protein synthesis lab, or humans read the output and retype it manually into an email. Presumably you also don’t think that’s where the safety difference comes from. So where does the safety difference come from?

(note: lunchtime for me in 2 minutes, propose to reconvene in 30m after that)

This does feel to me like another “look at the water” moment, so what do you predict I’ll say about that?

So the answer you expected from me, translated into my terms, would be, “If you select for the consequence of the humans hitting ‘approve’ on the plan, you’re still navigating the space of inputs for paths through time to probable outcomes (namely the humans hitting ‘approve’), so you’re still doing consequentialism.”

But suppose you manage to avoid that. Suppose you get exactly what you ask for. Then the system is still outputting plans such that, when humans follow them, they take paths through time and end up with outcomes that score high in some scoring function.

My answer is, “What the heck would it mean for a planning system to be non-consequentialist? You’re asking for nonwet water! What’s consequentialist isn’t the system that does the work, it’s the work you’re trying to do! You could imagine it being done by a cognition-free material system like a time machine and it would still be consequentialist because the output is a plan, a path through time!”

And this indeed is a case where I feel a helpless sense of not knowing how I can rephrase things, which exercises you have to get somebody to do, what fictional experience you have to walk somebody through, before they start to look at the water and see a material with an index, before they start to look at the phrase “why couldn’t a planning system be non-consequentialist about which plans it outputs” and go “um”.

My imaginary listener now replies, “Ah, but what if we have plans that don’t end up with outcomes that score high in some function?” and I reply “Then you lie on the ground randomly twitching because any outcome you end up with which is not that is one that you wanted more than that meaning you preferred it more than the outcome of random motor outputs which is optimization toward higher in the preference function which is taking a path through time that leads to particular destinations more than it leads to random noise.”

So I’d preface by saying that, if you could build such a system, which is indeed a coherent thing (it seems to me) to describe for the purpose of building it, then there would possibly be a safety difference on the margins, it would be noticeably less dangerous though still dangerous. It would need a special internal structural property that you might not get by gradient descent on a loss function with that structure, just like natural selection on inclusive genetic fitness doesn’t get you explicit fitness optimizers; you could optimize for planning in hypothetical situations, and get something that didn’t explicitly care only and strictly about hypothetical situations. And even if you did get that, the outputs that would kill or brain-corrupt the operators in hypothetical situations might also be fatal to the operators in actual situations. But that is a coherent thing to describe, and the fact that it was not optimizing our own universe, might make it safer.

With that said, I would worry that somebody would think there was some bone-deep difference of agentiness, of something they were empathizing with like personhood, of imagining goals and drives being absent or present in one case or the other, when they imagine a planner that just solves “hypothetical” problems. If you take that planner and feed it the actual world as its hypothetical, tada, it is now that big old dangerous consequentialist you were imagining before, without it having acquired some difference of psychological agency or ‘caring’ or whatever.

So I think there is an important homework exercise to do here, which is something like, “Imagine that safe-seeming system which only considers hypothetical problems. Now see that if you take that system, don’t make any other internal changes, and feed it actual problems, it’s very dangerous. Now meditate on this until you can see how the hypothetical-considering planner was extremely close in the design space to the more dangerous version, had all the dangerous latent properties, and would probably have a bunch of actual dangers too.”

“See, you thought the source of the danger was this internal property of caring about actual reality, but it wasn’t that, it was the structure of planning!”

“What if this potion we’re brewing has a Good Part and a Bad Part, and we could just keep the Good Parts…”

(That is in fact a difference! The insight is in realizing that the hypothetical planner is only one line of outer shell command away from being a Big Scary Thing and is therefore also liable to be Big and Scary in many ways.)

My position is that the AI does not neatly separate its internals into a Part You Think Of As Good and a Part You Think Of As Bad, because that distinction is sharp in your map but not sharp in the territory or the AI’s map.

From the perspective of a paperclip-maximizing-action-outputting-time-machine, its actions are not “object-level making paperclips” or “manipulating the humans next to the time machine to deceive them about what the machine does”, they’re just physical outputs that go through time and end up with paperclips.

The danger is in how hard the agent has to work to come up with the plan. I can, for instance, build an agent that very safely outputs a high-level plan for saving the world:

echo “Hey Richard, go save the world!”

So I do have to ask what kind of “high-level” planning output, that saves the world, you are envisioning, and why it was hard to cognitively come up with such that we didn’t just make that high-level plan right now, if humans could follow it. Then I’ll look at the part where the plan was hard to come up with, and say how the agent had to understand lots of complicated things in reality and accurately navigate paths through time for those complicated things, in order to even invent the high-level plan, and hence it was very dangerous if it wasn’t navigating exactly where you hoped. Or, alternatively, I’ll say, “That plan couldn’t save the world: you’re not postulating enough superintelligence to be dangerous, and you’re also not using enough superintelligence to flip the tables on the currently extremely doomed world.”

Look at the water; it’s not the way you’re doing the work that’s dangerous, it’s the work you’re trying to do. What work are you trying to do, never mind how it gets done?

Well, for one thing, I’d note that a system which is designing nanosystems, and spending 1% of its time thinking about how to kill the operators, is lethal. It has to be such a small fraction of thinking that it, like, never completes the whole thought about “well, if I did X, that would kill the operators!”

An AI that can solve these difficult tasks will need to be able to

Very very little is universally necessary over the design space. The first AGI that our tech becomes able to build is liable to work in certain easier and simpler ways.

Can you, in principle, build a red-car-driver that is totally incapable of driving blue cars? In principle, sure! But the first red-car-driver that gradient descent stumbles over is liable to be a blue-car-driver too.

Indubitably and indeed, this is so.

That doesn’t have a very simple answer, but one aspect there is domain generality which in turn is achieved through novel domain learning.

Humans, you will note, were not aggressively optimized by natural selection to be able to breathe underwater or fly into space. In terms of obvious outer criteria, there is not much outer sign that natural selection produced these creatures much more general than chimpanzees, by training on a much wider range of environments and loss functions.

There are complicated aspects of this story where natural selection may metaphorically be said to have “had no idea of what it was doing”, eg, after early rises in intelligence possibly produced by sexual selection on neatly chipped flint handaxes or whatever, all the cumulative brain-optimization on chimpanzees reached a point where there was suddenly a sharp selection gradient on relative intelligence at Machiavellian planning against other humans (even more so than in the chimp domain) as a subtask of inclusive genetic fitness, and so continuing to optimize on “inclusive genetic fitness” in the same old savannah, turned out to happen to be optimizing hard on the subtask and internal capability of “outwit other humans”, which optimized hard on “model other humans”, which was a capability that could be reused for modeling the chimp-that-is-this-chimp, which turned the system on itself and made it reflective, which contributed greatly to its intelligence being generalized, even though it was just grinding the same loss function on the same savannah; the system being optimized happened to go there in the course of being optimized even harder for the same thing.

So one can imagine asking the question: Is there a superintelligent AGI that can quickly build nanotech, which has a kind of passive safety in some if not all respects, in virtue of it solving problems like “build a nanotech system which does X” the way that a beaver solves building dams, in virtue of having a bunch of specialized learning abilities without it ever having a cross-domain general learning ability?

And in this regard one does note that there are many, many, many things that humans do which no other animal does, which you might think would contribute a lot to that animal’s fitness if there were animalistic ways to do it. They don’t make iron claws for themselves. They never did evolve a tendency to search for iron ore, and burn wood into charcoal that could be used in hardened-clay furnaces.

No animal plays chess, but AIs do, so we can obviously make AIs to do things that animals don’t do. On the other hand, the environment didn’t exactly present any particular species with a challenge of chess-playing either.

Even so, though, even if some animal had evolved to play chess, I fully expect that current AI systems would be able to squish it at chess, because the AI systems are on chips that run faster than neurons and doing crisp calculations and there are things you just can’t do with noisy slow neurons. So that again is not a generally reliable argument about what AIs can do.

So we can again ask: Is there a way to make an AI system that is only good at designing nanosystems, which can achieve some complicated but hopefully-specifiable real-world outcomes, without that AI also being superhuman at understanding and manipulating humans?

And I roughly answer, “Perhaps, but not by default, there’s a bunch of subproblems, I don’t actually know how to do it right now, it’s not the easiest way to get an AGI that can build nanotech (and kill you), you’ve got to make the red-car-driver specifically not be able to drive blue cars.” Can I explain how I know that? I’m really not sure I can, in real life where I explain X0 and then the listener doesn’t generalize X0 to X and respecialize it to X1.

It’s like asking me how I could possibly know in 2008, before anybody had observed AlphaFold 2, that superintelligences would be able to crack the protein folding problem on the way to nanotech, which some people did question back in 2008.

Though that was admittedly more of a slam-dunk than this was, and I could not have told you that AlphaFold 2 would become possible at a prehuman level of general intelligence in 2021 specifically, or that it would be synced in time to a couple of years after GPT-2’s level of generality at text.

Definitely, “turns out it’s easier than you thought to use gradient descent’s memorization of zillions of shallow patterns that overlap and recombine into larger cognitive structures, to add up to a consequentialist nanoengineer that only does nanosystems and never does sufficiently general learning to apprehend the big picture containing humans, while still understanding the goal for that pivotal act you wanted to do” is among the more plausible advance-specified miracles we could get.

But it is not what my model says actually happens, and I am not a believer that when your model says you are going to die, you get to start believing in particular miracles. You need to hold your mind open for any miracle and a miracle you didn’t expect or think of in advance, because at this point our last hope is that in fact the future is often quite surprising – though, alas, negative surprises are a tad more frequent than positive ones, when you are trying desperately to navigate using a bad map.

But that’s not where humans came from. We didn’t get to nuclear power by getting a bunch of fitness from nuclear power plants. We got to nuclear power because if you get a bunch of fitness from chipping flint handaxes and Machiavellian scheming, as found by relatively simple and local hill-climbing, that entrains the same genes that build nuclear power plants.

Huh???

OK

What are the most relevant axes of difference between solving protein folding and designing nanotech that, say, self-assembles into a computer?

Though I want to mark that this question seemed potentially cruxy to me, though perhaps not for others. I.e., if building protein factories that built nanofactories that built nanomachines that met a certain deep and lofty engineering goal, didn’t involve cognitive challenges different in kind from protein folding, we could maybe just safely go do that using AlphaFold 3, which would be just as safe as AlphaFold 2.

I don’t think we can do that. And I would note to the generic Other that if, to them, these both just sound like thinky things, so why can’t you just do that other thinky thing too using the thinky program, this is a case where having any specific model of why we don’t already have this nanoengineer right now would tell you there were specific different thinky things involved.

The second question didn’t make much sense in my native ontology? Humans raised without culture don’t have access to environmental constants whose presence their genes assume, so they end up as broken machines and then they’re bad consequentialists.

What is it that varies that you think I think should predict would stay more constant?

With regards to your first point, my first reaction was, “I just have one view of intelligence, what you see me arguing about reflects which points people have proved weirdly obstinate about. In 2008, Robin Hanson was being weirdly obstinate about how capabilities scaled and whether there was even any point in analyzing AIs differently from ems, so I talked about what I saw as the most slam-dunk case for there being Plenty Of Room Above Biology and for stuff going whoosh once it got above the human level.

“It later turned out that capabilities started scaling a whole lot without self-improvement, which is an example of the kind of weird surprise the Future throws at you, and maybe a case where I missed something by arguing with Hanson instead of imagining how I could be wrong in either direction and not just the direction that other people wanted to argue with me about.

“Later on, people were unable to understand why alignment is hard, and got stuck on generalizing the concept I refer to as consequentialism. A theory of why I talked about both things for related reasons would just be a theory of why people got stuck on these two points for related reasons, and I think that theory would mainly be overexplaining an accident because if Yann LeCun had been running effective altruism I would have been explaining different things instead, after the people who talked a lot to EAs got stuck on a different point.”

Returning to your second point, humans are broken things; if it were possible to build computers while working even worse than humans, we’d be having this conversation at that level of intelligence instead.

Among the things to bear in mind about that, is that we then get tons of weird phenomena that are specific to humans, and you may be very out of luck if you start wishing for the same weird phenomena in AIs. Yes, even if you make some sort of attempt to train it using a loss function.

However, it does seem to me like as we start getting towards the Einstein level instead of the village-idiot level, even though this is usually not much of a difference, we do start to see the atmosphere start to thin already, and the turbulence start to settle down already. Von Neumann was actually a fairly reflective fellow who knew about, and indeed helped generalize, utility functions. The great achievements of von Neumann were not achieved by some very specialized hypernerd who spent all his fluid intelligence on crystallizing math and science and engineering alone, and so never developed any opinions about politics or started thinking about whether or not he had a utility function.

I suspect that some of the difference here is that I think you have to be noticeably better than a human at nanoengineering to pull off pivotal acts large enough to make a difference, which is why I am not instead trying to gather the smartest people left alive and doing that pivotal act directly.

I can’t think of anything you can do with somebody just barely smarter than a human, which flips the gameboard, aside of course from “go build a Friendly AI” which I did try to set up to just go do and which would be incredibly hard to align if we wanted an AI to do it instead (full-blown chicken-and-egg, that AI is already fully aligned).

Explicit reflection is one possible later stage of the path; an earlier part of the path is from being optimized to do things difficult enough that you need to stop stepping on your own feet and have different parts of your thoughts work well together.

It’s the sort of path that has only one destination at its end, so there will be many ways to get there.

(Modulo various cases where different decision theories seem reflectively consistent and so on; I want to say “you know what I mean” but maybe people don’t.)

Monitoring all AGI projects – either not politically feasible in real life given the actual way that countries behave in history books instead of fantasy; or at politically feasible levels, does not work well enough to prevent the world from ending once the know-how proliferates. The AI isn’t doing much work here either; why not go do this now, if it’s possible? (Note: please don’t try to go do this now, it backfires badly.)

Provide sufficiently compelling arguments = superhuman manipulation, an incredibly dangerous domain that is just about the worst domain to try to align.

On my first point, it seems to me that your claims about recursive self-improvement were off in a fairly similar way to how I think your claims about consequentialism are off – which is that they defer too much to one very high-level abstraction.

I suppose that is what it could potentially feel like from the inside to not get an abstraction. Robin Hanson kept on asking why I was trusting my abstractions so much, when he was in the process of trusting his worse abstractions instead.

Can you explain a little more what you mean by “have different parts of your thoughts work well together”? Is this something like the capacity for metacognition; or the global workspace; or self-control; or…?

No, it’s like when you don’t, like, pay five apples for something on Monday, sell it for two oranges on Tuesday, and then trade an orange for an apple.

I have still not figured out the homework exercises to convey to somebody the Word of Power which is “coherence” by which they will be able to look at the water, and see “coherence” in places like a cat walking across the room without tripping over itself.

When you do lots of reasoning about arithmetic correctly, without making a misstep, that long chain of thoughts with many different pieces diverging and ultimately converging, ends up making some statement that is… still true and still about numbers! Wow! How do so many different thoughts add up to having this property? Wouldn’t they wander off and end up being about tribal politics instead, like on the Internet?

And one way you could look at this, is that even though all these thoughts are taking place in a bounded mind, they are shadows of a higher unbounded structure which is the model identified by the Peano axioms; all the things being said are true about the numbers. Even though somebody who was missing the point would at once object that the human contained no mechanism to evaluate each of their statements against all of the numbers, so obviously no human could ever contain a mechanism like that, so obviously you can’t explain their success by saying that each of their statements was true about the same topic of the numbers, because what could possibly implement that mechanism which (in the person’s narrow imagination) is The One Way to implement that structure, which humans don’t have?

But though mathematical reasoning can sometimes go astray, when it works at all, it works because, in fact, even bounded creatures can sometimes manage to obey local relations that in turn add up to a global coherence where all the pieces of reasoning point in the same direction, like photons in a laser lasing, even though there’s no internal mechanism that enforces the global coherence at every point.

To the extent that the outer optimizer trains you out of paying five apples on Monday for something that you trade for two oranges on Tuesday and then trading two oranges for four apples, the outer optimizer is training all the little pieces of yourself to be locally coherent in a way that can be seen as an imperfect bounded shadow of a higher unbounded structure, and then the system is powerful though imperfect because of how the power is present in the coherence and the overlap of the pieces, because of how the higher perfect structure is being imperfectly shadowed. In this case the higher structure I’m talking about is Utility, and doing homework with coherence theorems leads you to appreciate that we only know about one higher structure for this class of problems that has a dozen mathematical spotlights pointing at it saying “look here”, even though people have occasionally looked for alternatives.

And when I try to say this, people are like, “Well, I looked up a theorem, and it talked about being able to identify a unique utility function from an infinite number of choices, but if we don’t have an infinite number of choices, we can’t identify the utility function, so what relevance does this have” and this is a kind of mistake I don’t remember even coming close to making so I do not know how to make people stop doing that and maybe I can’t.

Yeah, was thinking the same.

Reconvene… tomorrow? Monday of next week?

Nate, take a minute to think of your own thoughts there?

I think that broadly matches my take. I’m also a bit worried about biases in the text summarizer, and about whether I managed to say anything that Rob or somebody will object to pre-publication, but we ultimately intended this to be seen and I was keeping that in mind, so, yeah, go ahead and summarize.

I admit to being curious as to what you thought was said that was important or new, but that’s a question that can be left open to be answered at your leisure, earlier in your day.

Yeah.

I don’t know if it’s going to help, but trying it currently seems better than to go on saying nothing.

Doesn’t work as fine for me because I’ve been trying to avoid any food until 12:30p my time, but on that particular day I may be more caloried than usual from the previous day, and could possibly get away with it. (That whole day could also potentially fail if a minor medical procedure turns out to take more recovery than it did the last time I had it.)

(seems reasonable! note, though, that i’d be quite happy to have sensitive sections stricken from the record, insofar as that lets us get more convergence than we otherwise would, while we’re already in the area)

I’m a touch reluctant to have discussions that we intend to delete, because then the larger debate will make less sense once those sections are deleted. Let’s dance around things if we can.

I mean, I can that day at 10am my time say how I am doing and whether I’m in shape for that day.

Good night, heroic debater!

The

A concept, which some people have had trouble grasping.  There seems to be an endless list.  I didn’t have to spend much time contemplating consequentialism to derive the consequences.  I didn’t spend a lot of time talking about it until people started arguing.

the

a

[the search process] is [so general]

“is by default”.  The reason I keep emphasizing that things are only true by default is that the work of surviving may look like doing hard nondefault things.  I don’t take fatalistic “will happen” stances, I assess difficulties of getting nondefault results.

it’ll be very hard to

“I don’t currently see how to”

one

the ultimate

second

secondary outcome of evolution

especially when we’re aware of concepts like utility functions

Very slight effect on human effectiveness in almost all cases because humans have very poor reflectivity.

particularly helpful

“attempted explanatory”.  I don’t think most readers got it.

I’m a little puzzled by how often you write my viewpoint as thinking that whatever I happened to say a sentence about is the Key Thing.  It seems to rhyme with a deeper failure of many EAs to pass the MIRI ITT.

To be a bit blunt and impolite in hopes that long-languishing social processes ever get anywhere, two obvious uncharitable explanations for why some folks may systematically misconstrue MIRI/Eliezer as believing much more than in reality that various concepts an argument wanders over are Big Ideas to us, when some conversation forces us to go to that place:

(A)  It paints a comfortably unflattering picture of MIRI-the-Other as weirdly obsessed with these concepts that seem not so persuasive, or more generally paints the Other as a bunch of weirdos who stumbled across some concept like “consequentialism” and got obsessed with it.  In general, to depict the Other as thinking a great deal of some idea (or explanatory thought experiment) is to tie and stake their status to the listener’s view of how much status that idea deserves.  So if you say that the Other thinks a great deal of some idea that isn’t obviously high-status, that lowers the Other’s status, which can be a comfortable thing to do.

(cont.)

(B) It paints a more comfortably self-flattering picture of a continuing or persistent disagreement, as a disagreement with somebody who thinks that some random concept is much higher-status than it really is, in which case there isn’t more to done or understood except to duly politely let the other person try to persuade you the concept deserves its high status. As opposed to, “huh, maybe there is a noncentral point that the other person sees themselves as being stopped on and forced to explain to me”, which is a much less self-flattering viewpoint on why the conversation is staying within a place.  And correspondingly more of a viewpoint that somebody else is likely to have of us, because it is a comfortable view to them, than a viewpoint that it is comfortable to us to imagine them having.

Taking the viewpoint that somebody else is getting hung up on a relatively noncentral point can also be a flattering self-portrait to somebody who believes that, of course.  It doesn’t mean they’re right.  But it does mean that you should be aware of how the Other’s story, told from the Other’s viewpoint, is much more liable to be something that the Other finds sensible and perhaps comfortable, even if it implies an unflattering (and untrue-seeming and perhaps untrue) view of yourself, than something that makes the Other seem weird and silly and which it is easy and congruent for you yourself to imagine the Other thinking.

no training regime

…that the training regimes we come up with first, in the 3 months or 2 years we have before somebody else destroys the world, will not have this property.

I don’t have any particularly complicated or amazingly insightful theories of why I keep getting depicted as a fatalist; but my world is full of counterfactual functions, not constants.  And I am always aware that if we had access to a real Textbook from the Future explaining all of the methods that are actually robust in real life – the equivalent of telling us in advance about all the ReLUs that in real life were only invented and understood a few decades after sigmoids – we could go right ahead and build a superintelligence that thinks 2 + 2 = 5.

All of my assumptions about “I don’t see how to do X” are always labeled as ignorance on my part and a default because we won’t have enough time to actually figure out how to do X.  I am constantly maintaining awareness of this because being wrong about it being difficult is a major place where hope potentially comes from, if there’s some idea like ReLUs that robustly vanquishes the difficulty, which I just didn’t think of.  Which does not, alas, mean that I am wrong about any particular thing, nor that the infinite source of optimistic ideas that is the wider field of “AI alignment” is going to produce a good idea from the same process that generates all the previous naive optimism through not seeing where the original difficulty comes from or what other difficulties surround obvious naive attempts to solve it.

Can I explain how I know that? I’m really not sure I can.

In original text, this sentence was followed by a long attempt to explain anyways; if deleting that, which is plausibly the correct choice, this lead-in sentence should also be deleted, as otherwise it paints a false picture of how much I would try to explain anyways.

From Nate’s notes:

For instance, the plan presumably needs to involve all sorts of mechanisms for refocusing the laser in the case where the environment contains fog, and redirecting the laser in the case where the environment contains mirrors (…the analogy is getting a bit strained here, sorry, bear with me), so that it can in fact hit a narrow and distant target. Refocusing and redirecting to stay on target are part and parcel to plans that can hit narrow distant targets.

But the humans shutting the AI down is like scattering the laser, and the humans tweaking the AI so that it plans in a different direction is like them tossing up mirrors that redirect the laser; and we want the plan to fail to correct for those interferences.

–> GOOD ANALOGY.

…or at least it sure conveys to me why corrigibility is anticonvergent / anticoherent / actually moderately strongly contrary to and not just an orthogonal property of a powerful-plan generator.

But then, I already know why that’s true and how it generalized up to resisting our various attempts to solve small pieces of more important aspects of it – it’s not just true by weak default, it’s true by a stronger default where a roomful of people at a workshop spend several days trying to come up with increasingly complicated ways to describe a system that will let you shut it down (but not steer you through time into shutting it down), and all of those suggested ways get shot down. (And yes, people outside MIRI now and then publish papers saying they totally just solved this problem, but all of those “solutions” are things we considered and dismissed as trivially failing to scale to powerful agents – they didn’t understand what we considered to be the first-order problems in the first place – rather than these being evidence that MIRI just didn’t have smart-enough people at the workshop.)

Eg, “Well, we took a system that only learned from reinforcement on situations it had previously been in, and couldn’t use imagination to plan for things it had never seen, and then we found that if we didn’t update it on shut-down situations it wasn’t reinforced to avoid shutdowns!”