Ngo and Yudkowsky on scientific reasoning and pivotal acts

1 – I’d say that it’s some evidence for the Dario viewpoint which seems close to the Paul viewpoint.  I say it’s some evidence for the Dario viewpoint because Dario seems to be the person who made something like an advance prediction about it.  It’s not enough to make me believe that you can straightforwardly extend the GPT architecture to 3e14 parameters and train it on 1e13 samples and get human-equivalent performance.

not especially that come to mind?  on my model of the future this is not particularly something I am supposed to know unless there is a rare flash of predictability.

2 – in lots of cases where I learned more specifics about X, and updated about Y, I had the experience of looking back and realizing that knowing anything specific about X would have predictably produced a directional update about Y.  like, knowing anything in particular about how the first AGI eats computation, would cause you to update far away from thinking that biological analogies to the computation consumed by humans were a good way to estimate how many computations an AGI needs to eat.  you know lots of details about how humans consume watts of energy, and you know lots of details about how modern AI consumes watts, so it’s very visible that these quantities are so incredibly different and go through so many different steps that they’re basically unanchored from each other.

I have specific ideas about how you get AGI that isn’t just scaling up Stack More Layers, which lead me to think that the way to estimate the computational cost of it is not “3e14 parameters trained at 1e16 ops per step for 1e13 steps, because that much computation and parameters seems analogous to human biology and 1e13 steps is given by past scaling laws”, a la recent OpenPhil publication.  But it seems to me that it should be possible to have the abstract insight that knowing more about general intelligence in AGIs or in humans would make the biological analogy look less plausible, because you wouldn’t be matching up an unknown key to an unknown lock.

Unfortunately I worry that this depends on some life experience with actual discoveries to get something this abstract-sounding on a gut level, because people basically never seem to make abstract updates of this kind when I try to point to them as predictable directional updates?

But, in principle, I’d hope there would be aspects of this where I could figure out how to show that any knowledge of specifics would probably update you in a predictable direction, even if it doesn’t seem best for Earth for me to win that argument by giving specifics conditional on those specifics actually being correct, and it doesn’t seem especially sound to win that argument by giving specifics that are wrong.

sure, and other factors will be present in AIs but not in humans

eg, human brains don’t scale and mesh, while it’s far more plausible that with AI you could just run more and more of it

that’s a huge factor leading one to expect AI to scale faster than human brains did

it’s like communication between humans, but squared!

this is admittedly a specific argument and I’m not sure how it would abstract out to any specific argument

I suppose the genericized version of my actual response to that would be, “architectures that have a harder time eating more compute are architectures which, for this very reason, are liable to need better versions invented of them, and this in particular seems like something that plausibly happens before scaling to general intelligence is practically possible”

I’m not saying I know how to make that abstractized argument for exactly what Richard cares about, in part because I don’t understand Richard’s exact model, just that it’s one way to proceed past the point where the obvious dilemma crops up of, “If a theory about AGI capabilities is true, it is a disservice to Earth to speak it, and if a theory about AGI capabilities is false, an argument based on it is not sound.”

possible viewpoint to try: that systems in general often have threshold functions as well as smooth functions inside them.

only in ignorance, then, do we imagine that the whole thing is one smooth function.

the history of humanity has a threshold function of, like, communication or culture or whatever.

the correct response to this is not, “ah, so this was the unique, never-to-be-seen-again sort of fact which cropped up in the weirdly complicated story of humanity in particular, which will not appear in the much simpler story of AI”

this only sounds plausible because you don’t know the story of AI so you think it will be a simple story

the correct generalization is “guess some weird thresholds will also pop up in whatever complicated story of AI will appear in the history books”

something like this indeed appears in my model of why I expect not much impact on GDP before AGI is powerful enough to bypass human economies entirely

during the runup phase, pre-AGI won’t be powerful to do “whole new things” that depend on doing lots of widely different things that humans can’t do

just marginally new things that depend on doing one thing humans can’t do, or can do but a bunch worse

there will be sufficient economic reward for building out industries that are mostly human plus one thing that pre-AGI does, and people will pocket those economic rewards, go home, and not be more ambitious than that.  I have trouble empathically grasping why almost all the CEOs are like this in our current Earth, because I am very much not like that myself, but observationally, the current Earth sure does seem to behave like rich people would almost uniformly rather not rock the boat too much.

I did not understand the whole thing about village idiots actually

do you want to copy and paste the document, or try rephrasing the argument?

My models do not have an easy time of visualizing “as generally intelligent as a chimp, but specialized to science research, gives you superhuman scientific capability and the ability to make progress in novel areas of science”.

(this is a reference back to the pre-rephrase in the document)

it seems like, I dunno, “gradient descent can make you generically good at anything without that taking too much general intelligence” must be a core hypothesis there?

I do credit the Village Idiot to Einstein Interval with being a little broader as a target than I used to think, since the Alpha series of Go-players took a couple of years to go from pro to world-beating even once they had a scalable algorithm.  Still seems to me that, over time, the wall clock time to traverse those ranges has been getting shorter, like Go taking less time than Chess.  My intuitions still say that it’d be quite weird to end up hanging out for a long time with AGIs that conduct humanlike conversations and are ambitious enough to run their own corporations while those AGIs are still not much good at science.

But on my present model, I suspect the limitations of “gradient-descent-without-much-general-intelligence” to fall underneath the village idiot side?

Your model, as I understand it, is saying, “But surely, GD-without-GI must suffice to produce better scientists than village idiots, by specializing chimps on science” and my current reply, though it’s not a particular question I’ve thought a lot about before, is, “That… does not quite seem to me like a thing that should happen along the mainline?”

though, as always, in the limit of superintelligences doing things, or our having the Textbook From The Future, we could build almost any kind of mind on purpose if we knew how, etc.

eg, a superintelligence could load up chimps with lots of domain-specific knowledge they were not generally intelligent enough to learn themselves.

ehhhhhh, it is not clear to me that GPT-3 is better than a village idiot at advanced science, even in this narrow sense, especially if the village idiot is allowed some training

I wonder if we’re visualizing different village idiots

my choice of “village idiot” originally was probably not the best target for visualization, because in a lot of cases, a village idiot – especially the stereotype of a village idiot – is, like, a damaged general intelligence with particular gears missing?

whereas it seems like what you want is something more like “Homo erectus but it has language”

oh, wow, 10th percentile intelligence?

that’s super high

GPT-3 is far far out of its league

I think that if scientific reasoning were as easy as you seem to be imagining(?), the publication factories of the modern world would be much more productive of real progress.

Like, on my current model of how the world really works, China pours vast investments into universities and sober-looking people with PhDs and classes and tests and postdocs and journals and papers; but none of this is the real way of Science which is actually, secretly, unbeknownst to China, passed down in rare lineages and apprenticeships from real scientist mentor to real scientist student, and China doesn’t have much in the way of lineages so the extra money they throw at stuff doesn’t turn into real science.

Like… make sense… at all?  Invent a handaxe when nobody had ever seen a handaxe before?

The activity of rearranging scientific sentences into new plausible-sounding paragraphs is well within the reach of publication factories, in fact, they often use considerably more semantic sophistication than that, and yet, this does not cumulate into real scientific progress even in quite large amounts.

I think GPT-3 is basically just Not Science Yet to a much greater extent than even these empty publication factories.

If 10th percentile humans don’t invent handaxes, GPT-3 sure as hell doesn’t.

I mean, while it is usually weird for something to be barely within a species’s capabilities while being within those capabilities at all, such that only relatively smarter individual organisms can do it, in the case of something that a social species has only very recently started to do collectively, it’s plausible that the thing appeared at the point where it was barely accessible to the smartest members.  Eg, it wouldn’t be surprising if it would have taken a long time or forever for humanity to invent science from scratch, if all the Francis Bacons and Newtons and even average-intelligence people were eliminated leaving only the bottom 10%.  Because our species just started doing that, at the point where our species was barely able to start doing that, meaning, at the point where some rare smart people could spearhead it, historically speaking.  It’s not obvious whether or not less smart people can do it over a longer time.

I’m not sure we disagree much about the human part of this model.

My guess is that our disagreement is more about GPT-3.

“Talk ‘cogently’ about a complex abstract topic” doesn’t seem like much of anything significant to me, if GPT-3 is ‘cogent’.  It fails to pass the threshold for inventing science and, I expect, for most particular sciences.

I also expect that if you offer a 10th-percentile human lots of money, they can learn to talk more cogently than GPT-3 about narrower science areas.  GPT-3 is legitimately more well-read at its lower level of intelligence, but train the 10-percentiler in a narrow area and they will become able to write better nonsense about that narrow area.

Like, what we’ve got going on here is a real breadth advantage that GPT-3 has in some areas, but the breadth doesn’t add up because it lacks the depth of a 10%er.

AI has figured out how to do a subhumanly shallow kind of thinking, and it is to be expected that when AI can do anything at all, it can soon do more of that thing than the whole human species could do.

No, that’s nothing remotely like giving the human the brief training the human needs to catch up to GPT-3’s longer training.

A 10%er does not learn in an instant – they learn faster than GPT-3, but not in an instant.

This is more like a scenario of paying somebody to, like, sit around for a year with an editor, learning how to mix-and-match economics sentences until they can learn to sound more like they’re making an argument than GPT-3 does, despite still not understanding any economics.

A lot of the learning would just go into producing sensible-sounding nonsense at all, since lots of 10%ers have not been to college and have not learned how to regurgitate rearranged nonsense for college teachers.

¯\_(ツ)_/¯

Humans cannot in general pass the Turing Test for posing as AIs!

What percentage of humans can pass as a calculator by reading an arithmetic textbook?

Zero!

Then it depends on what kind of answers!

I think a lot of 10%ers could learn to do wedding-cake multiplication, if sufficiently well-paid as adults rather than being tortured in school, out to 6 digits, thus handily beating the current GPT-3 at ‘multiplication’.

GPT-3 did not learn to produce its responses by reading textbooks.

You’re not matching the human’s data to GPT-3’s data.

You would want to collect, like, 1,000 Reddit arguments about inflation, and have the human read that, and have the human produce their own Reddit arguments, and have somebody tell them whether they sounded like real Reddit arguments or not.

The textbook is just not the same thing at all.

I’m not sure we’re at the core of the argument, though.

To me it seems like GPT-3 is allowed to be superhuman at producing remixed and regurgitated sentences about economics, because this is about as relevant to Science talent as a calculator being able to do perfect arithmetic, only less so.

Are these points that GPT-N read elsewhere on the Internet, or are they new good points that no professor of economics on Earth has ever made before?

Then I think that you can do this without being able to do science.  It’s a lot like if somebody with a really good memory was lucky enough to have read that exact argument on the Internet yesterday, and to have a little talent for paraphrasing.  Not by coincidence, having this ability gives you – on my model – no ability to do science, invent science, be the first to build handaxes, or design nanotechnology.

I admit, this does reflect my personal model of how Science works, presumably not shared by many leading bureaucrats, where in fact the papers full of regurgitated scientific-sounding sentences are not accomplishing much.

I’m trying to remember whether or not I’ve heard of that happening, like, 10 years ago.

My vague recollection is that things in the Sokal Hoax genre where the submissions succeeded, used humans to hand-generate the nonsense rather than any submissions in the genre having been purely machine-generated.

Obviously, that doesn’t mean it couldn’t have been done 10 years ago, because 10 years ago it’s plausibly a lot easier to hand-generate passing nonsense than to write an AI program that does it.

oh, wait, I’m wrong!

https://news.mit.edu/2015/how-three-mit-students-fooled-scientific-journals-0414

In April of 2005 the team’s submission, “Rooter: A Methodology for the Typical Unification of Access Points and Redundancy,” was accepted as a non-reviewed paper to the World Multiconference on Systemics, Cybernetics and Informatics (WMSCI), a conference that Krohn says is known for “being spammy and having loose standards.”

in 2013 IEEE and Springer Publishing removed more than 120 papers from their sites after a French researcher’s analysis determined that they were generated via SCIgen

The creators continue to get regular emails from computer science students proudly linking to papers they’ve snuck into conferences, as well as notes from researchers urging them to make versions for other disciplines.

Sure! Resume 5p?

👋

(My model is something like: there are some very shallow steps involved in doing science, lots of medium steps, occasional very deep steps, assembling the whole thing into Science requires having all the lego blocks available.  As soon as you look at anything with details, it ends up ‘heavy-tailed’ because it has multiple pieces and says how things don’t work if all the pieces aren’t there.)

It’s not obvious to me that science proceeds significantly past its present point.  I would not have the right to be surprised if Reality told me the correct answer was that a civilization like that just doesn’t reach AGI, ever.

No, humanity is exactly the species that finds it barely possible to do science.

If it were possible to do it with less intelligence, we’d be having this conversation over the Internet that we’d developed with less intelligence.

Yes, it does seem that there’s quite a narrow band between “barely smart enough to develop agriculture” and “barely smart enough to develop computers”! Though there were genuinely fewer people in the preagricultural world, with worse nutrition and no Ashkenazic Jews, and there’s the whole question about to what degree the reproduction of the shopkeeper class over several centuries was important to the Industrial Revolution getting started.

I model you as taking a lesson from this which is something like… you can train up a villager to be John von Neumann by spending some evolutionary money on giving them science-specific brain features, since John von Neumann couldn’t have been much more deeply or generally intelligent, and you could spend even more money and make a chimp a better scientist than John von Neumann.

My model is more like, yup, the capabilities you need to invent aqueducts sure do generalize the crap out of things, though also at the upper end of cognition there are compounding returns which can bring John von Neumann into existence, and also also there’s various papers suggesting that selection was happening really fast over the last few millennia and real shifts in cognition shouldn’t be ruled out.  (This last part is an update to what I was thinking when I wrote Intelligence Explosion Microeconomics, and is from my own perspective a more gradualist line of thinking, because it means there’s a wider actual target to traverse before you get to von Neumann.)

My skepticism here is that there’s a version of, like, “invent nanotechnology” which routes through just the shallow places, which humanity stumbles over before we stumble over deep AGI.

It should not be overlooked that there’s a very valid sibling of the old complaint “Anything you can do ceases to be AI”, which is that “Things you can do with surprisingly-to-your-model shallow cognition are precisely the things that Reality surprises you by telling you that AI can do earlier than you expected.”  When we see GPT-3, we were getting some amount of real evidence about AI capabilities advancing faster than I expected, and some amount of evidence about GPT-3’s task being performable using shallower cognition than expected.

Many people were particularly surprised by Go because they thought that Go was going to require deeper real thought than chess.

And I think AlphaGo probably was thinking in a legitimately deeper way than Deep Blue.  Just not as much deeper as Douglas Hofstadter thought it would take.

Conversely, people thought a few years ago that driving cars really seemed to be the sort of thing that machine learning would be good at, and were unpleasantly surprised by how the last 0.1% of driving conditions were resistant to shallow techniques.

Despite the inevitable fact that some surprises of this kind now exist, and that more such surprises will exist in the future, it continues to seem to me that science-and-engineering on the level of “invent nanotech” still seems pretty unlikely to be easy to do with shallow thought, by means that humanity discovers before AGI tech manages to learn deep thought?

What actual cognitive steps?  Outside-the-box thinking, throwing away generalizations that governed your previous answers and even your previous questions, inventing new ways to represent your questions, figuring out which questions you need to ask and developing plans to answer them; these are some answers that I hope will be sufficiently useless to AI developers that it is safe to give them, while still pointing in the direction of things that have an un-GPT-3-like quality of depth about them.

Doing this across unfamiliar domains that couldn’t be directly trained in by gradient descent because they were too expensive to simulate a billion examples of

If you have something this powerful, why is it not also noticing that the world contains humans?  Why is it not noticing itself?

Uhhhh… is there a reason why “Eric Drexler’s Nanosystems but, like, the real thing, modulo however much Drexler did not successfully Predict the Future about how to do that, which was probably a lot” is not the obvious answer here?

that’s… not historically true at all?

getting a grip on quantities of heat and their flow was critical to getting steam engines to work

it didn’t happen until the math was there

maybe you can be a mechanic banging on an engine that somebody else designed, around principles that somebody even earlier invented, without a physics degree

but, like, engineers have actually needed math since, like, that’s been a thing, it wasn’t just a prestige trick

or other abstract concepts than equations, which have never existed before

like, maybe not with a type signature unknown to humanity, but with specific instances unknown to present humanity

that’s what I’d expect to see from humanly designed nanosystems

yeeeessss, that is why DeepMind did not take over the world last year

it’s not just that AlphaFold lacks the concepts but that it lacks the machinery to invent those concepts and the machinery to do anything with such concepts

(and they will accept their trivial billion-dollar-payouts and World GDP will continue largely undisturbed, on my mainline model, because it will be easiest to find ways to make money by leveraging narrow systems on the less regulated, less real parts of the economy, instead of trying to build houses or do medicine, etc.)

real tests being expensive, simulation being impossibly expensive, and not having enough samples to train your civilization’s current level of AI technology, is not a problem you can solve by training a new AI to generate samples, because you do not have enough samples to train your civilization’s current level of AI technology to generate more samples

well, I’ve been debating Paul about that separately in the timelines channel, not sure about recapitulating it here

but in broad summary, since I expect the future to look like it was drawn from the “history book” barrel and not the “futurism” barrel, I expect huge barriers to doing huge things with narrow AI in small amounts of time; you can sell waifutech because it’s unregulated and hard to regulate, but that doesn’t feed into core mining and steel production.

we could already have double the GDP if it was legal to build houses and hire people, etc., and the change brought by pre-AGI will perhaps be that our GDP could quadruple instead of just double if it was legal to do things, but that will not make it legal to do things, and why would anybody try to do things and probably fail when there are easier $36 billion profits to be made in waifutech.

my belief is not so much that this training can never happen, but that this probably means the system was trained beyond the point of safe shallowness

not in principle over all possible systems a superintelligence could build, but in practice when it happens on Earth

my only qualm about this is that current techniques make it possible to buy shallowness in larger quantities than this Earth has ever seen before, and people are looking for surprising ways to make use of that

so I weigh in my mind the thought of Reality saying Gotcha! by handing me a headline I read tomorrow about how GPT-4 has started producing totally reasonable science papers that are actually correct

and I am pretty sure that exact thing doesn’t happen

and I ask myself about GPT-5 in a few more years, which had the same architecture as GPT-3 but more layers and more training, doing the same thing

and it’s still largely “nope”

then I ask myself about people in 5 years being able to use the shallow stuff in any way whatsoever to produce the science papers

and of course the answer there is, “okay, but is it doing that without having shallowly learned stuff that adds up to deep stuff which is why it can now do science“

and I try saying back “no, it was born of shallowness and it remains shallow and it’s just doing science because it turns out that there is totally a way to be an incredibly mentally shallow skillful scientist if you think 10,000 shallow thoughts per minute instead of 1 deep thought per hour”

and my brain is like, “I cannot absolutely rule it out but it really seems like trying to call the next big surprise in 2014 and you guess self-driving cars instead of Go because how the heck would you guess that Go was shallower than self-driving cars”

like, that is an imaginable surprise

but there’s a world of difference between saying that a surprise is imaginable, and that it wouldn’t surprise you

the Chess techniques had already failed at Go

actual new techniques were required

the people around at the time had witnessed sudden progress on self-driving cars a few years earlier

self-driving cars were of obviously greater economic interest as well

my recollection is that talk of the time was about self-driving

heh! I have the same sense.

that is, math being shallower than science.

though perhaps not as shallow as Go, and you will note that Go has fallen and Math has not

if Reality came back and was like “Wrong! Keeping up with the far reaches of human mathematics is harder than being able to develop your own nanotech,” I would be like “What?” to about the same degree as being “What?” on “You can build nanotech just by thinking trillions of thoughts that are too shallow to notice humans!”

How do you suspect your beliefs would shift if you had any detailed model of intelligence?

Consider trying to imagine a particular wrong model of intelligence and seeing what it would say differently?

(not sure this is a useful exercise and we could indeed try to move on)

The first model is true and the second one is false, if that helps.  You can tell this by contemplating where you would update if you learned any model, by considering that things look more disconnected when you can’t see the machinery behind them.  If you don’t know what moves the second hand on a watch and the minute hand on a watch, they could just be two things that move at different rates for completely unconnected reasons; if you can see inside the watch, you’ll see that the battery is shared and the central timing mechanism is shared and then there’s a few gears to make the hands move at different rates.

Like, in my ontology, the notion of “effortful” doesn’t particularly parse as anything basic, because it doesn’t translate over into paperclip maximizers, which are neither effortful nor effortless.

But in a human scientist you’ve got thoughts being shoved around by all sorts of processes behind the curtains, created by natural selection, some of them reflecting shards of Consequentialism / shadowing paths through time

The flashes of insight come to people who were looking in nonrandom places

If they didn’t plan deliberately and looked on pure intuition, they looked with an intuition trained by past success and failure

Somebody walking doesn’t plan to walk, but long ago as a baby they learned from falling over, and their ancestors who fell over more didn’t reproduce

Once we knew specific things about bioweapons, countries developed specific treaties for controlling them, which failed (according to @CarlShulman)

Mmmmmmaybe.  Do note that I tend to be more optimistic than the average human about, say, global warming, or everything in transhumanism outside of AGI.

Nukes have going for them that, in fact, nobody has an incentive to start a global thermonuclear war.  Eliezer is not in fact pessimistic about everything and views his AGI pessimism as generalizing to very few other things, which are not, in fact, as bad as AGI.

But yeah, compared to pre-1946 history, nukes actually kind of did go really surprisingly well!

Like, this planet used to be a huge warring snakepit of Great Powers and Little Powers and then nukes came along and people actually got serious and decided to stop having the largest wars they could fuel.

The analog would be an international agreement to sign a nice unenforced statement of AI safety principles and then all just building AGI in doomy ways without explicitly saying they’re doing it..

This scenario sure sounds INCREDIBLY PLAUSIBLE, yes

I mostly don’t consider the anthropic shadow stuff

I’m happy making no appeal to anthropics here.

There may come a day when the strength of nonanthropic reasoning fails… but that is not this day!

[So yeah, I picture 1930s-Eliezer pointing to technological trends and being like “by default, 30 years after the first nukes are built, you’ll be able to build one in your back yard. And governments aren’t competent enough to stop that happening.”

And I don’t think I could have come up with a compelling counterargument back then.]

So, I mean, in fact, I don’t prophesize doom from very many trends at all!  It’s literally just AGI that is anywhere near that unmanageable!  Many people in EA are more worried about biotech than I am, for example.

[I do believe that AI control will be more difficult than nuclear control, because AI is so much more useful. But I also expect that there will be many more details about AI development that we don’t currently understand, that will allow us to influence it (because AGI is a much more complicated concept than “really really big bomb”).]

Er… I think this is not a correct use of the Way I was attempting to gesture at; things being more complicated when known than unknown, does not mean you have more handles to influence them because each complication has the potential to be a handle.  It is not in general true that very complicated things are easier for humanity in general, and governments in particular, to control, because they have so many exposed handles.

I think there’s a valid argument about it maybe being more possible to control the supply chain for AI training processors if the global chip supply chain is narrow (also per Carl).

It is in fact a big deal about nuclear tech that uranium can’t be mined in every country, as I understand it, and that centrifuges stayed at the frontier of technology and were harder to build outside the well-developed countries, and that the world ended up revolving around a few Great Powers that had no interest in nuclear tech proliferating any further.

Unfortunately, before you let that encourage you too much, I would also note it was an important fact about nuclear bombs that they did not produce streams of gold and then ignite the atmosphere if you turned up the stream of gold too high with the actual thresholds involved being unpredictable.

I would be a lot more cheerful about a few Great Powers controlling AGI if AGI produced wealth, but more powerful AGI produced no more wealth; if AGI was made entirely out of hardware, with no software component that could be keep getting orders of magnitude more efficient using hardware-independent ideas; and if the button on AGIs that destroyed the world was clearly labeled.

That does take AGI to somewhere in the realm of nukes.

And Eliezer is capable of being less concerned about things when they are intrinsically less concerning, which is why my history does not, unlike some others in this field, involve me running also being Terribly Concerned about nuclear war, global warming, biotech, and killer drones.

Well, if you’re a superintelligence, you can probably do human-equivalent human-speed general intelligence on a 286, though it might possibly have less fine motor control, or maybe not, I don’t know.

I wouldn’t be surprised if human ingenuity asymptoted out at AGI on a home computer from 1995.

Don’t know if it’d take more like a hundred years or a thousand years to get fairly close to that.

The question of how efficiently you can perform a fixed algorithm doing fixed things, often pales compared to the gains on switching to different algorithms doing different things.

Given government control of all the neural net training chips and no more public GPU farms, I buy that they could keep a nuke!AGI (one that wasn’t tempting to crank up and had clearly labeled Doom-Causing Buttons whose thresholds were common knowledge) under lock of the Great Powers for 7 years, during which software decreased hardware requirements by 44x.  I am a bit worried about how long it takes before there’s a proper paradigm shift on the level of deep learning getting started in 2006, after which the Great Powers need to lock down on individual GPUs.

I’ve noticed that opinion around OpenPhil!  It makes sense if you have short timelines and expect the world to end before there’s another paradigm shift, but OpenPhil doesn’t seem to expect that either.

Yeah, uh, there was kinda a paradigm shift in AI between say 2000 and now.  There really, really was.

Would you believe that at one point Netflix ran a prize contest to produce better predictions of their users’ movie ratings, with a $1 million prize, and this was one of the largest prizes ever in AI and got tons of contemporary ML people interested, and neural nets were not prominent on the solutions list at all, because, back then, people occasionally solved AI problems not using neural nets?

I suppose that must seem like a fairy tale, as history always does, but I lived it!

The world might otherwise end before the next paradigm shift, but if the world keeps on ticking for 10 years, 20 years, there will not always be the paradigm of training massive networks by even more massive amounts of gradient descent; I do not think that is actually the most efficient possible way to turn computation into intelligence.

Neural networks stayed stuck at only a few layers for a long time, because the gradients would explode or die out if you made the networks any deeper.

There was a critical moment in 2006(?) where Hinton and Salakhutdinov(?) proposed training Restricted Boltzmann machines unsupervised in layers, and then ‘unrolling’ the RBMs to initialize the weights in the network, and then you could do further gradient descent updates from there, because the activations and gradients wouldn’t explode or die out given that initialization.  That got people to, I dunno, 6 layers instead of 3 layers or something? But it focused attention on the problem of exploding gradients as the reason why deeply layered neural nets never worked, and that kicked off the entire modern field of deep learning, more or less.

If anybody goes back and draws a graph claiming the whole thing was continuous if you measure the right metric, I am not really very impressed unless somebody at the time was using that particular graph and predicting anything like the right capabilities off of it.

[Okay, so are you claiming that that neural networks were mostly bottlenecked by algorithmic improvements, not compute availability, for a significant part of their history?]

Er… yeah?  There was a long time when, even if you threw a big neural network at something, it just wouldn’t work.

Good night, btw?

Nuclear plants might be like narrow AI in this analogy; some designs potentially contribute to proliferation, and you can get more economic wealth by building more of them, but they have no Unlabeled Doom Dial where you can get more and more wealth out of them by cranking them up until at some unlabeled point the atmosphere ignites.

Also a thought: I don’t think you just want somebody with more knowledge of AI history, I think you might need to ask an actual old fogey who was there at the time, and hasn’t just learned an ordered history of just the parts of the past that are relevant to the historian’s theory about how the present happened.

Two of them, independently, to see if the answers you get are reliable-as-in-statistical-reliability.