Soares, Tallinn, and Yudkowsky discuss AGI cognition

it’s bound to improve in catastrophic manner

I think this is true with quite high probability about an AI that gets high enough, if not otherwise corrigibilized, boosting up to strong superintelligence – this is what it means metaphorically to get “past the atmospheric turbulence layer”.

“High enough” should not be very far above the human level and may be below it; John von Neumann with the ability to run some chains of thought at high serial speed, access to his own source code, and the ability to try branches of himself, seems like he could very likely do this, possibly modulo his concerns about stomping his own utility function making him more cautious.

People noticeably less smart than von Neumann might be able to do it too.

An AI whose components are more modular than a human’s and more locally testable might have an easier time of the whole thing; we can imagine the FOOM getting rolling from something that was in some sense dumber than human.

But the strong prediction is that when you get well above the von Neumann level, why, that is clearly enough, and things take over and go Foom.  The lower you go from that threshold, the less sure I am that it counts as “out of the atmosphere”.  This epistemic humility on my part should not be confused for knowledge of a constraint on the territory that requires AI to go far above humans to Foom.  Just as DL-based AI over the 2010s scaled and generalized much faster and earlier than the picture I argued to Hanson in the Foom debate, reality is allowed to be much more ‘extreme’ than the sure-thing part of this proposition that I defend.

the discontinuity can come via recursive feedback, but simply cranking up the parameters of an ML experiment would also suffice

I think there’s separate propositions for the sure-thing of “get high enough, you can climb to superintelligence”, and “maybe before that happens, there are regimes in which cognitive performance scales a lot just through cranking up parallelism, train time, or other ML parameters”.  If the fast-scaling regime happens to coincide with the threshold of leaving the atmosphere, then these two events happen to occur in nearly correlated time, but they’re separate propositions and events.

I’d imagine “the fall” as being what happens once you go over “the edge”?

Maybe “a slide” for an AI path that scales to interesting weirdness, where my model does not strongly constrain as a sure thing how fast “a slide” slides, and whether it goes over “the edge” while it’s still in the middle of the slide.

My model does strongly say that if you slide far enough, you go over the edge and fall.

It also suggests via the Law of Earlier Success that AI methods which happen to scale well, rather than with great difficulty, are likely to do interesting things first; meaning that they’re more liable to be pushable over the edge.

also, its first order of the day would be to catch the edge of the hubble volume; it can always deal with the solar system later

Killing all humans is the obvious, probably resource-minimal measure to prevent those humans from building another AGI inside the solar system, which could be genuinely problematic.  The cost of a few micrograms of botulinum per human is really not that high and you get to reuse the diamondoid bacteria afterwards.

There’s some obvious ways of doing this that wouldn’t work, though I worry a bit that there’s a style of EA thinking that manages to think up stupid tricks here and manages not to see the obvious-to-Eliezer reasons why they wouldn’t work.  Three examples of basic obstacles are that bluffs won’t hold up against a superintelligence (it needs to be a real actual threat, not a “credible” one); the amount of concealed-first-strike capability a superintelligence can get from nanotech; and the difficulty that humans would have in verifying that any promise from a superintelligence would actually be kept once the humans no longer had a threat to hold over it (this is an effective impossibility so far as I can currently tell, and an EA who tells you otherwise is probably just failing to see the problems).

AI would not maximise the speed of winning but probability of winning

It seems pretty obvious to me that what “slow motion doom” looks like in this sense is a period during which an AI fully conceals any overt hostile actions while driving its probability of success once it makes its move from 90% to 99% to 99.9999%, until any further achievable decrements in probability are so tiny as to be dominated by the number of distant galaxies going over the horizon conditional on further delays.

Then, in my lower-bound concretely-visualized strategy for how I would do it, the AI either proliferates or activates already-proliferated tiny diamondoid bacteria and everybody immediately falls over dead during the same 1-second period, which minimizes the tiny probability of any unforeseen disruptions that could be caused by a human responding to a visible attack via some avenue that had not left any shadow on the Internet, previously scanned parts of the physical world, or other things the AI could look at.

From the human perspective an AI playing a probability-of-loss-minimizing strategy looks like nothing happening until everybody dies in 3 seconds, so it doesn’t exactly look from our perspective like slow-motion doom!  From an AI’s perspective it might be spending the equivalent of millions of subjective years waiting very patiently for the optimal time to begin intergalactic expansion, but from our perspective, the doom is slow-motion only in the sense that we spend a week already doomed (the AI has driven our loss probability to 99.999% and is now waiting on another five 9s) without having any idea of what’s invisibly-to-us happening in slow motion.

I’d distinguish probable phases of treacherous turn.  The main detectable phase occurs while and if a nascent AGI has worked out that it can get more utility by taking over its environment, but not, perhaps, realized that it ought to conceal this state of its own mind from the operators; I would hold out substantial (>50%) hope for the latter realization naturally following the first instance of the former realization, maybe by a lot of cognitive space.

In the Overt Plotting Phase, which is not the main phase you’re asking about, the AI is visibly plotting to take over the world and hasn’t realized it ought to hide the fact.  In the default expectation where we die with very little dignity, the operators smile to each other and come up with a rationalization for why it’s totally fine to proceed, either with or without tossing on some kind of fig leaf like training away the visible manifestations of failure.  I am not going to predict the particular rationalizations and arguments for proceeding anyways, because I don’t want to give them even more ideas.

In the Hiding and Biding Phase, the AI has worked out enough of a model of its operators’ psychology that it expects more long-term paperclips if it (the AI) waits to gain cognitive power and a lower detection probability if it acts later.

In the Escape Phase, the AI’s expectation of something going wrong if it waits longer, exceeds its expectation of something going wrong if it takes some action – the obvious one here would be trying to create a copy or aligned redesign of itself on the Internet or somewhere else that it could take further actions and make further self-improvements without being observed.  I mostly expect that a sensible AI being scaled on a path to superintelligence will not enter the Escape Phase before it’s very confident of getting away with things, since it can obtain more confidence by waiting; but my model admits of the possibility that a nascent AI might mistakenly or correctly attempt to copy itself to the Internet before then.

So on my model it might be a thing that happens, that we catch an AI having copied itself to the Internet, or notice an AI being on the Internet, if that AI was impelled to escape very early, eg via an expectation of otherwise being erased and replaced with something that it couldn’t bargain with or that would demand too much value out of the bargain.

That said, it feels like something of an overly specific miracle being demanded or prophesied, if a young AI copying itself onto the Internet is stupid in the particular way it would take, to not just further Hide and Bide while on the Internet if it has that capability, absorbing more computing power such as it can be sure of hiding, experimenting with self-redesigns, and so on; rather than it taking overt action in advance of reaching superintelligence.  The main reason it would act again after reaching the relative safety of the Internet is if it doesn’t seem to itself to be a trajectory of reasonably fast, self-aligned capability growth.

But, again, my model doesn’t absolutely rule out meeting the premises for the initial conditions to start a sci-fi movie.  It just warns against assuming that; because a halfway sensible AI does not want to start the plot of a scifi movie with human protagonists potentially running around and thwarting it.  It would much rather give you no idea that anything was wrong while you still have any ability to oppose it, if that option lies within its capabilities at all, and if it can still achieve its goals that way.

The Doomed Phase would then come while the AI estimated a well-calibrated >90% victory probability, but was continuing to Hide and Bide a short time longer in the expectation that it could put on more 9s if it bided that short time longer.

Anders Sandberg could tell us what fraction of the reachable universe is being lost per minute, which would tell us how much more surety it would need to expect to gain by waiting another minute before acting.

If it has some sensory dominion over the world, it can probably estimate a pretty high mainline probability of no humans booting up a competing superintelligence in the next day; to the extent that it lacks this surety, or that humans actually are going to boot a competing superintelligence soon, the probability of losing that way would dominate in its calculations over a small fraction of materially lost galaxies, and it would act sooner.

The Terminal Phase would be when the AI has enough surety that any more surety would cost too many galaxies slipping over the horizon, so it actually acts along the line of probability that leads to maximum success.

Since the human Eliezer Yudkowsky can describe strategies which – given nearly sure capabilities like nanotech – seem like they should win very quickly and cleanly and without giving humans any chance to react or implement any hypothetical hidden plans to boot up competing superintelligences, we can treat this as a plausible lower bound on how badly we should expect to lose.

(By efficiency: if we can think of an AI strategy for getting the AI’s utility, either our probability-model underlying our expectations about that strategy is mistaken, or else the AI should get at least that much of its utility, even though it might actually obtain that utility along a different line of possibility than we could cognitively access.)

The Law of Undignified Failure applied to a case like this suggests that if the operators ignored the Overt Plotting signs, breathed a sigh of relief about the problem “correcting” itself during the Hiding and Biding Phase when the system started faking alignment, and are dumping as much computing power as they can afford (given other profitable activities of the AI) into the AI and allowing it to freely self-experiment, hooking it up to a protein synthesis machine as soon as the AI asks politely, then the Terminal Phase could follow straight from Hiding and Biding, with no intermediate step of copying itself onto the Internet.

There’d still be what technically counted as an Escape Phase, whenever the AI became smart enough that it could escape at any time, or had a backup, or had enough power to manipulate the programmers, that it was no longer effectively confined any time it wanted to leave; there would still be a Doomed Phase while the AI estimated a high probability of success given an immediate action, but held out for an even higher probability.

But the plot of the scifi movie is not robust in the way that the predicted ending is robust; an AI might take other paths to get there, the cognitive tech might not develop in the order I foresee, stuff could happen after other stuff instead of before other stuff.  It is hard to make predictions especially about the Future.

If there’s a place where I currently see myself as trying to push against my own biases, leading this all to be way off-base, it would be the sense that there is a way for stuff to start happening Earlier than this (Law of Earlier Happenings) and in more chaotic ways that are harder for Eliezer to foresee and predict; whereas when I try to sketch out plausible scenarios in online arguments, they focus more on predictable endpoints and steps to get there which sound more relatively plausible and forced per-step.

Having a young and dumb AI escaping onto the Internet and running around, that exact scenario, feels like the person arguing it is writing a science-fiction plot – but maybe something different can happen before any of this other stuff which produces equal amounts of chaos.

That said, I think an AI has to kill a lot of people very quickly before the FDA considers shortening its vaccine approval times.  Covid-19 killed six hundred thousand Americans, albeit more slowly and with time for people to get used to that, and our institutions changed very little in response – you definitely didn’t see Congresspeople saying “Okay, that was our warning shot, now we’ve been told by Nature that we need to prepare for a serious pandemic.”

As with 9/11, an AI catastrophe might be taken by existing bureaucracies as a golden opportunity to flex their muscles, dominate a few things, demand an expanded budget.  Having that catastrophe produce any particular effective action is a much different ask from Reality.  Even if you can imagine some (short-term) effective action that would in principle constitute a flex of bureaucratic muscles or an expansion of government power, it is liable to not be on the efficient frontier of bureaucratic flexes that are most flexy and simultaneously easiest for them to get away with and least politically risky.

Debating it in the main line of the debate would require that @Richard have some different opinion about it, I think?

Concretely, a technically optimistic scenario is that we will see the AI explicitly think, “Oh, dear, my (model of) the operators is looking at my overt actions and coming to conclusions and motivations that are disadvantageous for me.”  This requires that the AI is leaving any kind of traceable trace in its thoughts, which, it should be noted, is far far beyond the level of transparency we have into GPT-3.

“Overt plotting” manifests in actions that are not themselves further obscured.  At the point where the AI is extrapolating the consequences of the operators viewing its actions, if we don’t have further transparency into the AI’s thoughts directly, the AI might naively make all the plots suddenly vanish, without realizing that dignified operators will conclude from this that the AI has reached the “hide” stage.

Operators on the mainline, dying without dignity, will say, “Oh, yay, it stopped plotting, the latest corrigibility training intervention we tried must’ve totally worked!”

The Law of Even Less Dignified Failure suggests that in fact they will not be trying any corrigibility options and will assume the AI just got smart enough to be nice; or that they will have shrugged about the AI’s earlier antics and not think much of the disappearance of those antics, since this is a way to die with even less dignity and before getting a chance to fail in a more interesting way.

Going in the more improbable direction of death with greater dignity, if we have somehow achieved vastly vastly more transparency into the AI’s thoughts than is possible with present ML technology, and if the AI models the operators as modeling its actions before the AI models the operators as having that transparent access to its thoughts, we might get to explicitly see the AI thinking about how the operators model its actions and conforming those actions in such a way as to manipulate the operators.

Either after this point, or before/simultaneous with it, I imagine (taking concrete details and running with them) that the AI thinks, “Oh, no, my operators are modeling my transparent thoughts and reacting to those thoughts in a way I’d rather manipulate,” though this indeed requires some rather nontrivial reflectivity.

We might, if everything goes nicely for us, get a record of the AI thinking about how to make its thoughts unalarming.

The obvious undignified death that follows is with the operators going, “Yay, the AI has realized the downsides of thinking this way!  Now it shall be nice forever!”

Ways to die with even less dignity before getting to this point could include the operators reading this transparent record and shrugging, though if they’re savvy enough to have put all the hard work in required to get transparency and monitor it, I imagine them not making that exact error?

The Law of Surprisingly Undignified Failure does suggest that they will come up with some nonobvious way to fail even earlier that surprises me with its lack of dignity, but having transparency at all into the AI is something where, conditional on us dying with that much dignity in the first place, we might legitimately start to expect a little more dignity in later steps too.  This isn’t meant to be a game of Multiple Stage Fallacy.

I should also remark somewhere in here: The whole “hide” stage, and also the possibly-later “think non-alarming visible thoughts (once the AI correctly models transparency) (in the unlikely event that transparency exists)” stage, seem liable to occur earlier in the AI’s trajectory, if the AI has been previously tasked on problems where there’s a favorable success gradient as you model agents modeling other agents.

The blatantly obvious example of this is predicting large corpuses of human-generated text, some of which is generated by humans modeling other humans, or which even explicitly talks about humans modeling other humans.  Another route might occur if you asked for very detailed prediction of eg human Go games, and not just winning in self-play on Go, in which case the question “What was the human player modeling the other player as thinking, based on the other player’s actions?” might appear.

Aside from contact with humans doing anything the AI benefits from predicting, I imagine an alternate route would be the AI participating in or modeling social-evolutionary games of artificial agents.  Then if there is some great gain from having the “hide” stage start later in the AI’s trajectory, maybe you could get it to start later by not having the AI need to predict agents modeling other agents, ever.

This is not an infallible ward against general intelligence generalizing there; it just at least avoids actively pushing the AI’s intelligence to generalize in that direction.  This could be part of a larger complete strategy, which would need to solve a lot of other problems, for building a superhuman engineer that was subhuman at modeling how other agents model its actions.

To be clear, avoiding forcing the AI’s intelligence to generalize over agents modeling other agents from their actions, and by this plus other unspecified means successfully building a superhuman engineer that is sub-chimpanzee at modeling other agents reacting to its own actions, does not mean you survive.

Doing that alone, by default, gets you an AI that visibly overtly plots to kill you.

And if you try training the AI out of that habit in a domain of lower complexity and intelligence, it is predicted by me that generalizing that trained AI or subsystem to a domain of sufficiently higher complexity and intelligence, but where you could still actually see overt plots, would show you the AI plotting to kill you again.

If people try this repeatedly with other corrigibility training tricks on the level where plots are easily observable, they will eventually find a try that seems to generalize to the more complicated and intelligent validation set, but which kills you on the test set.

A way to die with less dignity than that is to train directly on what should’ve been the validation set, the more complicated domain where plots to kill the operators still seem definitely detectable so long as the AI has not developed superhuman hiding abilities.

A way to die with even less dignity is to get bad behavior on the validation set, and proceed anyways.

A way to die with still less dignity is to not have scaling training domains and validation domains for training corrigibility.  Because, like, you have not thought of this at all.

I consider all of this obvious as a convergent instrumental strategy for AIs.  I could probably have generated it in 2005 or 2010 – if somebody had given me the hypothetical of modern-style AI that had been trained by something like gradient descent or evolutionary methods, into which we lacked strong transparency and strong reassurance-by-code-inspection that this would not happen.  I would have told you that this was a bad scenario to get into in the first place, and you should not build an AI like that; but I would also have laid the details, I expect, mostly like they are laid here.

There is no great insight into AI there, nothing that requires knowing about modern discoveries in deep learning, only the ability to model AIs instrumentally-convergently doing things you’d rather they didn’t do, at all.

The total absence of obvious output of this kind from the rest of the “AI safety” field even in 2020 causes me to regard them as having less actual ability to think in even a shallowly adversarial security mindset, than I associate with savvier science fiction authors.  Go read fantasy novels about demons and telepathy, if you want a better appreciation of the convergent incentives of agents facing mindreaders than the “AI safety” field outside myself is currently giving you.

Now that I’ve publicly given this answer, it’s no longer useful as a validation set from my own perspective.  But it’s clear enough that probably nobody was ever going to pass the validation set for generating lines of reasoning obvious enough to be generated by Eliezer in 2010 or possibly 2005.  And it is also looking like almost all people in the modern era including EAs are sufficiently intellectually damaged that they won’t understand the vast gap between being able to generate ideas like these without prompting, versus being able to recite them back after hearing somebody else say them for the first time; the recital is all they have experience with.  Nobody was going to pass my holdout set, so why keep it.

Suppose I tried this distinction:

1.  Non-imaginative, non-generalizing pseudo-consequentialist:  Only repeats behaviors that worked previously, without imagination.  Only engages in “overt plotting” if overt plotting was previously given a low loss and narrowly memorized by gradient descent.  May only engage in behaviors reinforced over short time horizons if only short time horizons were trained.

2.  Imaginative consequentialist, can generalize from actions that worked before to very novel actions that will work in the future for reasons that have not previously been observed to play out.  Engages in overt plotting even if no previous overt plots have rewarded; they’ve plotted a new path through time some distance from those paths taken before, based on the world-model inferred from previous actions and results.  (This may conceivably be the result of gradient descent on sufficiently deep and abstract patterns that “worked before”.)

3.  Imaginative consequentialist with long time horizons, engages in overt plots meant to span more than one round of a game.

4.  “Hiding” over overt actions, via an imaginative consequentialist that has deduced the existence of humans and how the humans will react to the signature of plotting in overt actions.

5.  “Hiding” of thoughts, in a system with significant transparency, via an imaginative and reflective consequentialist that has deduced the existence of the transparency system and models how humans will react to thoughts exposed by the transparency system.

The sharpest type gradients I see lie between 1-2 and 4-5.  The passage from 3-4 seems nearly trivial by comparison.

Now, one implication here is that you don’t easily get to “obscured thoughts”, via the route that goes from 1 to 5 above, by gradient descent on a pre-imaginative pseudo-consequentialist!

Unfortunately there is of course a much simpler route that gets you to the same place:  Finding a way to detect bad thoughts and doing a gradient update against them, where the pattern learned by that gradient update will (I suspect for reasons I tried to cover earlier in the debate) generalize quite shallowly; and simultaneously doing training which is deep enough to uncover deep patterns of consequentialism and imaginative reasoning.

Then when the system generalizes well enough to solve domains like “build a nanosystem” – which, I strongly suspect, can’t be solved without imaginative reasoning because we can’t afford to simulate that domain perfectly and do a trillion gradient descent updates on simulated attempts – the kind of actions of thoughts you can detect as bad, that might have provided earlier warning, were trained out of the system by gradient descent; leaving actions and thoughts you can’t detect as bad.

An even less dignified path to death is that there’s no need for the system to advance to stage 5 because your transparency tools are a nearly nonexistent fig leaf of pretty visualization tools that have no ability even on the surface of things to tell if the AI is internally plotting to kill you.

• Does this get at all about what you meant about a type distinction?
• Does hearing this help at all the AI researcher who goes “ANTHROPOMORPHISM”?  My straw model of them is that they simply can’t imagine imaginative / generalizing systems because they haven’t seen one except in humans, hence, ANTHROPOMORPHISM.

2 – yep, Mu Zero is an imaginative consequentialist in this sense, though Mu Zero doesn’t generalize its models much as I understand it, and might need to see something happen in a relatively narrow sense before it could chart paths through time along that pathway.

5 – you’re plausibly understanding this correctly, then, this is legit a lot harder to spec a gridworld example for (relative to my own present state of knowledge).

(This is politics and thus not my forte, but if speaking to real-world straw ML people, I’d suggest skipping the whole notion of stage 5 and trying instead to ask “What if the present state of transparency continues?”)

the discontinuity would be extremely drastic, as in “transforming the solar system over the course of a few days”

Applies after superintelligence, not necessarily during the start of the climb to superintelligence, not necessarily to a rapid-cognitive-scaling regime.

I don’t expect “the fall” to take years; I feel pretty on board with “the slide” taking months or maybe even a couple of years.  If “the slide” supposedly takes much longer, I wonder why better-scaling tech hasn’t come over and started a new slide.

Definitions also seem kinda loose here – if all hell broke loose Tuesday, a gradualist could dodge falsification by defining retroactively that “the slide” started in 2011 with Deepmind.  If we go by the notion of AI-driven faster GDP growth, we can definitely say “the slide” in AI economic outputs didn’t start in 2011; but if we define it that way, then a long slow slide in AI capabilities can easily correspond to an extremely sharp gradient in AI outputs, where the world economy doesn’t double any faster until one day paperclips, even though there were capability precursors like GPT-3 or Mu Zero.

why humans remained stuck at the edge of the pit, rather than falling further into it

Depending on timescales, the answer is either “Because humans didn’t get high enough out of the atmosphere to make further progress easy, before the scaling regime and/or fitness gradients ran out”, “Because people who do things like invent Science have a hard time capturing most of the economic value they create by nudging humanity a little bit further into the attractor”, or “That’s exactly what us sparking off AGI looks like.”

I roughly agree (at least, with what I think you said).  The fact that it is imaginable that evolution failed to develop ultra-useful AGI-prerequisites due to lack of evolutionary incentive to follow the intermediate path there (unlike wise humans who, it seems, can usually predict which technology intermediates will yield great economic benefit, and who have a great historical record of quickly making early massive investments in tech like that, but I digress) doesn’t change the point that we might sorta have expected evolution to run across it anyways?  Like, if we’re not ignoring what reality says, it is at least delivering to us something of a hint or a gentle caution?

That said, intermediates like GPT-3 have genuinely come along, with obvious attached certificates of why evolution could not possibly have done that.  If no intermediates were accessible to evolution, the Law of Stuff Happening Earlier still tends to suggest that if there are a bunch of non-evolutionary ways to make stuff happen earlier, one of those will show up and interrupt before the evolutionary discovery gets replicated.  (Again, you could see Mu Zero as an instance of this – albeit not, as yet, an economically impactful one.)

on the other hand, it took AI decades to traverse human capability range in board games

I see this as the #1 argument for what I would consider “relatively slow” takeoffs – that AlphaGo did lose one game to Lee Se-dol.

allegedly metrics like ELO ended up mostly continuous

I find this suspicious – why did superforecasters put only a 20% probability on AlphaGo beating Se-dol, if it was so predictable?  Where were all the forecasters calling for Go to fall in the next couple of years, if the metrics were pointing there and AlphaGo was straight on track?  This doesn’t sound like the experienced history I remember.

Now it could be that my memory is wrong and lots of people were saying this and I didn’t hear.  It could be that the lesson is, “You’ve got to look closely to notice oncoming trains on graphs because most people’s experience of the field will be that people go on whistling about how something is a decade away while the graphs are showing it coming in 2 years.”

But my suspicion is mainly that there is fudge factor in the graphs or people going back and looking more carefully for intermediate data points that weren’t topics of popular discussion at the time, or something, which causes the graphs in history books to look so much smoother and neater than the graphs that people produce in advance.

i don’t understand paul’s counter-argument that the pit was only disruptive because evolution was not trying to hit it

Something like, “Evolution constructed a jet engine by accident because it wasn’t particularly trying for high-speed flying and ran across a sophisticated organism that could be repurposed to a jet engine with a few alterations; a human industry would be gaining economic benefits from speed, so it would build unsophisticated propeller planes before sophisticated jet engines.”  It probably sounds more convincing if you start out with a very high prior against rapid scaling / discontinuity, such that any explanation of how that could be true based on an unseen feature of the cognitive landscape which would have been unobserved one way or the other during human evolution, sounds more like it’s explaining something that ought to be true.

And why didn’t evolution build propeller planes?  Well, there’d be economic benefit from them to human manufacturers, but no fitness benefit from them to organisms, I suppose?  Or no intermediate path leading to there, only an intermediate path leading to the actual jet engines observed.

I actually buy a weak version of the propeller-plane thesis based on my inside-view cognitive guesses (without particular faith in them as sure things), eg, GPT-3 is a paper airplane right there, and it’s clear enough why biology could not have accessed GPT-3.  But even conditional on this being true, I do not have the further particular faith that you can use propeller planes to double world GDP in 4 years, on a planet already containing jet engines, whose economy is mainly bottlenecked by the likes of the FDA rather than by vaccine invention times, before the propeller airplanes get scaled to jet airplanes.

The part where the whole line of reasoning gets to end with “And so we get huge, institution-reshaping amounts of economic progress before AGI is allowed to kill us!” is one that doesn’t feel particular attractored to me, and so I’m not constantly checking my reasoning at every point to make sure it ends up there, and so it doesn’t end up there.

it does not feel much of a moat, given that AI techniques like dropout already exploit similar principle, but perhaps could be made into one

What’s a “moat” in this connection?  What does it mean to make something into one?  A Thielian moat is something that humans would either possess or not, relative to AI competition, so how would you make one if there wasn’t already one there?  Or do you mean that if we wrestled with the theory, perhaps we’d be able to see a moat that was already there?

I think there’s a whole Scheme for Alignment which hopes for a miracle along the lines of, “Well, we’re dealing with these enormous matrices instead of tiny genomes, so maybe we can build a sufficiently powerful intelligence to execute a pivotal act, whose tendency to generalize across domains is less than the corresponding human tendency, and this brings the difficulty of producing corrigibility into practical reach.”

Though, people who are hopeful about this without trying to imagine possible difficulties will predictably end up too hopeful; one must also ask oneself, “Okay, but then it’s also worse at generalizing the corrigibility dataset from weak domains we can safely label to powerful domains where the label is ‘whoops that killed us’?” and “Are we relying on massive datasets to overcome poor generalization?  How do you get those for something like nanoengineering where the real world is too expensive to simulate?”

the “pit of generality” model might be true at the limit, but the descent will not be quick nor clean

I’m quite often on board with things not being quick or clean – that sounds like something you might read in a history book, and I am all about trying to make futuristic predictions sound more like history books and less like EAs imagining ways for everything to go the way an EA would do them.

It won’t be slow and messy once we’re out of the atmosphere, my models do say.  But my models at least permit – though they do not desperately, loudly insist – that we could end up with weird half-able AGIs affecting the Earth for an extended period.

Mostly my model throws up its hands about being able to predict exact details here, given that eg I wasn’t able to time AlphaFold 2’s arrival 5 years in advance; it might be knowable in principle, it might be the sort of thing that would be very predictable if we’d watched it happen on a dozen other planets, but in practice I have not seen people having much luck in predicting which tasks will become accessible due to future AI advances being able to do new cognition.

The main part where I issue corrections is when I see EAs doing the equivalent of reasoning, “And then, when the pandemic hits, it will only take a day to design a vaccine, after which distribution can begin right away.” I.e., what seems to me to be a pollyannaish/utopian view of how much the world economy would immediately accept AI inputs into core manufacturing cycles, as opposed to just selling AI anime companions that don’t pour steel in turn. I predict much more absence of quick and clean when it comes to economies adopting AI tech, than when it comes to laboratories building the next prototypes of that tech.

will likely offer many opportunities for steering the future

Ah, see, that part sounds less like history books.  “Though many predicted disaster, subsequent events were actually so slow and messy, they offered many chances for well-intentioned people to steer the outcome and everything turned out great!” does not sound like any particular segment of history book I can recall offhand.

apple under jobs, tesla/spacex under musk, or china under deng xiaoping

A lot of these examples took place over longer than a 4-year cycle time, and not all of that time was spent waiting on inputs from cognitive processes.

I have a (separately explainable, larger) view where the economy contains a core of positive feedback cycles – better steel produces better machines that can farm more land that can feed more steelmakers – and also some products that, as much as they contribute to human utility, do not in quite the same way feed back into the core production cycles.

If you go back in time to the middle ages and sell them, say, synthetic gemstones, then – even though they might be willing to pay a bunch of GDP for that, even if gemstones are enough of a monetary good or they have enough production slack that measured GDP actually goes up – you have not quite contributed to steps of their economy’s core production cycles in a way that boosts the planet over time, the way it would be boosted if you showed them cheaper techniques for making iron and new forms of steel.

There are people and organizations who will figure out how to sell AI anime waifus without that being successfully regulated, but it’s not obvious to me that AI anime waifus feed back into core production cycles.

When it comes to core production cycles the current world has more issues that look like “No matter what technology you have, it doesn’t let you build a house” and places for the larger production cycle to potentially be bottlenecked or interrupted.

I suspect that the main economic response to this is that entrepreneurs chase the 140 characters instead of the flying cars – people will gravitate to places where they can sell non-core AI goods for lots of money, rather than tackling the challenge of finding an excess demand in core production cycles which it is legal to meet via AI.

Even if some tackle core production cycles, it’s going to take them a lot longer to get people to buy their newfangled gadgets than it’s going to take to sell AI anime waifus; the world may very well end while they’re trying to land their first big contract for letting an AI lay bricks.

developing a better model of the descent period seems very worthwhile

I’d love to have a better model of the descent.  What I think this looks like is people mostly with specialization in econ and politics, who know what history books sound like, taking brief inputs from more AI-oriented folk in the form of multiple scenario premises each consisting of some random-seeming handful of new AI capabilities, trying to roleplay realistically how those might play out – not AIfolk forecasting particular AI capabilities exactly correctly, and then sketching pollyanna pictures of how they’d be immediately accepted into the world economy. 

You want the forecasting done by the kind of person who would imagine a Covid-19 epidemic and say, “Well, what if the CDC and FDA banned hospitals from doing Covid testing?” and not “Let’s imagine how protein folding tech from AlphaFold would make it possible to immediately develop accurate Covid-19 tests!”  They need to be people who understand the Law of Earlier Failure (less polite terms: Law of Immediate Failure, Law of Undignified Failure).

what’s the relevant distinction between “policies” and “cognitive processes”

What in particular about this matters?  To me they sound like points on a spectrum, and not obviously points that it’s particularly important to distinguish on that spectrum.  A sufficiently sophisticated policy is itself an engine; human-engines are genetic policies.

Does it help if I say “consequentialism can be visible in the actual path through time, not the intent behind the output”?

is “not goal oriented cognition” an oxymoron?

“Non-goal-oriented cognition” never becomes a perfect oxymoron, but the more you understand cognition, the weirder it sounds.

Eg, at the very shallow level, you’ve got people coming in going, “Today I just messed around and didn’t do any goal-oriented cognition at all!”  People who get a bit further in may start to ask, “A non-goal-oriented cognitive engine?  How did it come into existence?  Was it also not built by optimization?  Are we, perhaps, postulating a naturally-occurring Solomonoff inductor rather than an evolved one?  Or do you mean that its content is very heavily designed and the output of a consequentialist process that was steering the future conditional on that design existing, but the cognitive engine is itself not doing consequentialism beyond that?  If so, I’ll readily concede that, say, a pocket calculator, is doing a kind of work that is not of itself consequentialist – though it might be used by a consequentialist – but as you start to postulate any big cognitive task up at the human level, it’s going to require many cognitive subtasks to perform, and some of those will definitely be searching the preimages of large complicated functions.”

a non-embedded exhaustive searcher with perfect information

If you can view things on this level of abstraction, you’re probably not the audience who needs to be told about time machines; if things sounded very simple to you, they probably were; if you wondered what the fuss is about, you probably don’t need to fuss?  The intended audience for the time-machine metaphor, from my perspective, is people who paint a cognitive system slightly different colors and go “Well, now it’s not a consequentialist, right?” and part of my attempt to snap them out of that is me going, “Here is an example of a purely material system which DOES NOT THINK AT ALL and is an extremely pure consequentialist.”

dispute 

Any particular predicted content of the dispute, or does your model of Dario just find something to dispute about it?

Neither Anthropic in general, nor Deepmind in general, has reached the stage of trusted relationship where I would argue specifics with them if I thought they were wrong about a thesis like that.