As promised, here’s a write-up of some thoughts from my end. In particular, since I’ve spent a lot of the debate poking Eliezer about his views, I’ve tried here to put forward more positive beliefs of my own in this doc (along with some more specific claims): [GDocs link]

We take as a starting observation that a number of “grand challenges” in AI have been solved by AIs that are very far from the level of generality which people expected would be needed. Chess, once considered to be the pinnacle of human reasoning, was solved by an algorithm that’s essentially useless for real-world tasks. Go required more flexible learning algorithms, but policies which beat human performance are still nowhere near generalising to anything else; the same for StarCraft, DOTA, and the protein folding problem. Now it seems very plausible that AIs will even be able to pass (many versions of) the Turing Test while still being a long way from AGI.

Perhaps we can quantify the Turing test by asking something like:

• What percentile of competence is the judge?
• What percentile of competence are the humans who the AI is meant to pass as?
• How much effort does the judge put in (measured in, say, hours of strategic preparation)?

Does this framing seem reasonable to you? And if so, what are the highest numbers for each of these metrics that correspond to a Turing test which an AI could plausibly pass before the world ends?

I expect this trend to continue until after we have AIs which are superhuman at mathematical theorem-proving, programming, many other white-collar jobs, and many types of scientific research. It seems like Eliezer doesn’t. I’ll highlight two specific disagreements which seem to play into this.

Quoth Paul:

seems like you have to make the wright flyer much better before it’s important, and that it becomes more like an industry as that happens, and that this is intimately related to why so few people were working on it

Is this basically saying ‘the Wright brothers didn’t personally capture much value by inventing heavier-than-air flying machines, and this was foreseeable, which is why there wasn’t a huge industry effort already underway to try to build such machines as fast as possible.’ ?

My maybe-wrong model of Eliezer says here ‘the Wright brothers knew a (Thielian) secret’, while my maybe-wrong model of Paul instead says:

• They didn’t know a secret — it was obvious to tons of people that you could do something sorta like what the Wright brothers did and thereby invent airplanes; the Wright brothers just had unusual non-monetary goals that made them passionate to do a thing most people didn’t care about.
• Or maybe it’s better to say: they knew some specific secrets about physics/engineering, but only because other people correctly saw ‘there are secrets to be found here, but they’re stamp-collecting secrets of little economic value to me, so I won’t bother to learn the secrets’. ~Everyone knows where the treasure is located, and ~everyone knows the treasure won’t make you rich.

I mean, I think they knew a bit of stuff, but it generally takes a lot of stuff to make something valuable, and the more people have been looking around in an area the more confident you can be that it’s going to take a lot of stuff to do much better, and it starts to look like an extremely strong regularity for big industries like ML or semiconductors

it’s pretty rare to find small ideas that don’t take a bunch of work to have big impacts

I don’t know exactly what a thielian secret is (haven’t read the reference and just have a vibe)

straightening it out a bit, I have 2 beliefs that combine disjunctively: (i) generally it takes a lot of work to do stuff, as a strong empirical fact about technology, (ii) generally if the returns are bigger there are more people working on it, as a slightly-less-strong fact about sociology

secrets = important undiscovered information (or information that’s been discovered but isn’t widely known), that you can use to get an edge in something. https://www.lesswrong.com/posts/ReB7yoF22GuerNfhH/thiel-on-secrets-and-indefiniteness

There seems to be a Paul/Eliezer disagreement about how common these are in general. And maybe a disagreement about how much more efficiently humanity discovers and propagates secrets as you scale up the secret’s value?

My guess is that everyone agrees with claims 1, 2, and 3 here (please let me know if I’m wrong!):

1. The history of humanity looks less like Long Series of Cheat Codes World, and more like Well-Designed Game World.

In Long Series of Cheat Codes World, human history looks like this, over and over: Some guy found a cheat code that totally outclasses everyone else and makes him God or Emperor, until everyone else starts using the cheat code too (if the Emperor allows it). After which things are maybe normal for another 50 years, until a new Cheat Code arises that makes its first adopters invincible gods relative to the previous tech generation, and then the cycle repeats.

In Well-Designed Game World, you can sometimes eke out a small advantage, and the balance isn’t perfect, but it’s pretty good and the leveling-up tends to be gradual. A level 100 character totally outclasses a level 1 character, and some level transitions are a bigger deal than others, but there’s no level that makes you a god relative to the people one level below you.

2. General intelligence took over the world once. Someone who updated on that fact but otherwise hasn’t thought much about the topic should not consider it ‘bonkers’ that machine general intelligence could take over the world too, even though they should still consider it ‘bonkers’ that eg a coffee startup could take over the world.

(Because beverages have never taken over the world before, whereas general intelligence has; and because our inside-view models of coffee and of general intelligence make it a lot harder to imagine plausible mechanisms by which coffee could make someone emperor, kill all humans, etc., compared to general intelligence.)

(In the game analogy, the situation is a bit like ‘I’ve never found a crazy cheat code or exploit in this game, but I haven’t ruled out that there is one, and I heard of a character once who did a lot of crazy stuff that’s at least suggestive that she might have had a cheat code.’)

3. AGI is arising in a world where agents with science and civilization already exist, whereas humans didn’t arise in such a world. This is one reason to think AGI might not take over the world, but it’s not a strong enough consideration on its own to make the scenario ‘bonkers’ (because AGIs are likely to differ from humans in many respects, and it wouldn’t obviously be bonkers if the first AGIs turned out to be qualitatively way smarter, cheaper to run, etc.).

—

If folks agree with the above, then I’m confused about how one updates from the above epistemic state to ‘bonkers’.

It was to a large extent physics facts that determined how easy it was to understand the feasibility of nukes without (say) decades of very niche specialized study. Likewise, it was physics facts that determined you need rare materials, many scientists, and a large engineering+infrastructure project to build a nuke. In a world where the physics of nukes resulted in it being some PhD’s quiet ‘nobody thinks this will work’ project like Andrew Wiles secretly working on a proof of Fermat’s Last Theorem for seven years, that would have happened.

If an alien came to me in 1800 and told me that totally new physics would let future humans build city-destroying superbombs, then I don’t see why I should have considered it bonkers that it might be lone mad scientists rather than nations who built the first superbomb. The ‘lone mad scientist’ scenario sounds more conjunctive to me (assumes the mad scientist knows something that isn’t widely known, AND has the ability to act on that knowledge without tons of resources), so I guess it should have gotten less probability, but maybe not dramatically less?

‘Mad scientist builds city-destroying weapon in basement’ sounds wild to me, but I feel like almost all of the actual unlikeliness comes from the ‘city-destroying weapons exist at all’ part, and then the other parts only moderately lower the probability.

Likewise, I feel like the prima-facie craziness of basement AGI mostly comes from ‘generally intelligence is a crazy thing, it’s wild that anything could be that high-impact’, and a much smaller amount comes from ‘it’s wild that something important could happen in some person’s basement’.

—

It does structurally make sense to me that Paul might know things I don’t about GPT-3 and/or humans that make it obvious to him that we roughly know the roadmap to AGI and it’s this.

If the entire ‘it’s bonkers that some niche part of ML could crack open AGI in 2026 and reveal that GPT-3 (and the mainstream-in-2026 stuff) was on a very different part of the tech tree’ view is coming from a detailed inside-view model of intelligence like this, then that immediately ends my confusion about the argument structure.

I don’t understand why you think you have the roadmap, and given a high-confidence roadmap I’m guessing I’d still put more probability than you on someone finding a very different, shorter path that works too. But the argument structure “roadmap therefore bonkers” makes sense to me.

If there are meant to be other arguments against ‘high-impact AGI via niche ideas/techniques’ that are strong enough to make it bonkers, then I remain confused about the argument structure and how it can carry that much weight.

I can imagine an inside-view model of human cognition, GPT-3 cognition, etc. that tells you ‘AGI coming from nowhere in 3 years is bonkers’; I can’t imagine an ML-is-a-reasonably-efficient-market argument that does the same, because even a perfectly efficient market isn’t omniscient and can still be surprised by undiscovered physics facts that tell you ‘nukes are relatively easy to build’ and ‘the fastest path to nukes is relatively hard to figure out’.

(Caveat: I’m using the ‘basement nukes’ and ‘Fermat’s last theorem’ analogy because it helps clarify the principles involved, not because I think AGI will be that extreme on the spectrum.)

Oh, I also wouldn’t be confused by a view like “I think it’s 25% likely we’ll see a more Eliezer-ish world. But it sounds like Eliezer is, like, 90% confident that will happen, and that level of confidence (and/or the weak reasoning he’s provided for that confidence) seems bonkers to me.”

The thing I’d be confused by is e.g. “ML is efficient-ish, therefore the out-of-the-blue-AGI scenario itself is bonkers and gets, like, 5% probability.”

I feel confused about the role “innovations are almost always low-impact” plays in slow-takeoff-ish views.

Suppose I think that there’s some reachable algorithm that’s different from current approaches, and can do par-human scientific reasoning without requiring tons of compute.

The existence or nonexistence of such an algorithm is just a fact about the physical world. If I imagine one universe where such an algorithm exists, and another where it doesn’t, I don’t see why I should expect that one of those worlds has more discontinuous change in GWP, ship sizes, bridge lengths, explosive yields, etc. (outside of any discontinuities caused by the advent of humans and the advent of AGI)? What do these CS facts have to do with the other facts?

But AI Impacts seems to think there’s an important connection, and a large number of facts of the form ‘steamships aren’t like nukes’ seem to undergird a lot of Paul’s confidence that the scenario I described —

(“there’s some reachable algorithm that’s different from current approaches, and can do par-human scientific reasoning without requiring tons of compute.”)

— is crazy talk. (Unless I’m misunderstanding. As seems actually pretty likely to me!)

(E.g., Paul says “To me your model just seems crazy, and you are saying it predicts crazy stuff at the end but no crazy stuff beforehand”, and one of the threads of the timelines conversation has been Paul asking stuff like “do you want to give any example other than nuclear weapons of technologies with the kind of discontinuous impact you are describing?”.)

Possibilities that came to mind for me:

1. The argument is ‘reality keeps surprising us with how continuous everything else is, so we seem to have a cognitive bias favoring discontinuity, so we should have a skeptical prior about our ability to think our way to ‘X is discontinuous’ since our brains are apparently too broken to do that well?

(But to get from 1 to ‘discontinuity models are batshit’ we surely need something more probability-mass-concentrating than just a bias argument?)

2. The commonality between steamship sizes, bridge sizes, etc. and AGI is something like ‘how tractable is the world?’. A highly tractable world, one whose principles are easy to understand and leverage, will tend to have more world-shatteringly huge historical breakthroughs in various problems, and will tend to see a larger impact from the advent of humans and the advent of AGI.

Our world looks much less tractable, so even if there’s a secret sauce to building AGI, we should expect the resultant AGI to be a lot less impactful.

I endorse #2 (although I think more weakly than Paul does) and would also add #3: another commonality is something like “how competitive is innovation?”

@RobBensinger It’s showing us a fact about the vast space of ideas and technologies we’ve already explored that they are not so concentrated and lumpy that the law of large numbers doesn’t work well as a first approximation in a world with thousands or millions of people contributing. And that specifically includes past computer science innovation.

So the ‘we find a secret sauce algorithm that causes a massive unprecedented performance jump, without crappier predecessors’ is a ‘separate, additional miracle’ at exactly the same time as the intelligence explosion is getting going. You can get hyperbolic acceleration from increasing feedbacks from AI to AI hardware and software, including crazy scale-up at the end, as part of a default model. But adding on to it that AGI is hit via an extremely large performance jump of a type that is very rare, takes a big probability penalty.

And the history of human brains doesn’t seem to provide strong evidence of a fundamental software innovation, vs hardware innovation and gradual increases in selection applied to cognition/communication/culture.

The fact that, e.g. AIs are mastering so much math and language while still wielding vastly infrahuman brain-equivalents, and crossing human competence in many domains (where there was ongoing effort) over decades is significant evidence for something smoother than the development of modern humans and their culture.

That leaves me not expecting a simultaneous unusual massive human concentrated algorithmic leap with AGI, although I expect wildly accelerating progress from increasing feedbacks at that time. Crossing a given milestone is disproportionately likely to happen in the face of an unusually friendly part/jump of a tech tree (like AlexNet/the neural networks->GPU transition) but still mostly not, and likely not from an unprecedented in computer science algorithmic change.

https://aiimpacts.org/?s=cross+

(Just want to interject that Carl has higher P(doom) than Paul and has also critiqued Paul for not being more concrete, and I doubt that this is the source of the common disagreements that Paul/Carl both have with Eliezer)

From my perspective the thing the AI impacts investigation is asking is something like “When people are putting lots of resources into improving some technology, how often is it the case that someone can find a cool innovation that improves things a lot relative to the baseline?” I think that your response to that is something like “Sure, if the broad AI market were efficient and everyone were investigating the right lines of research, then AI progress might be smooth, but AGI would have also been developed way sooner. We can’t safely assume that AGI is like an industry where lots of people are pushing toward the same thing”

But it’s not assuming a great structural similarity between bridges and AI, except that they’re both things that humans are trying hard to find ways to improve

Sure, if the broad AI market were efficient and everyone were investigating the right lines of research, then AI progress might be smooth

Presumably also “‘AI progress’ subsumes many different kinds of cognition, we don’t currently have baby AGIs, and when we do figure out how to build AGI the very beginning of the curve (the Wright flyer moment, or something very shortly after) will correspond to a huge capability increase.”

I think Paul’s view would say:

• Things certainly happen for the first time
• When they do, they happen at small scale in shitty prototypes, like the Wright Flyer or GPT-1 or AlphaGo or the Atari bots or whatever
• When they’re making a big impact on the world, it’s after a lot of investment and research, like commercial aircrafts in the decades after Kitty Hawk or like the investments people are in the middle of making now with AI that can assist with coding

Paul’s view says that the Kitty Hawk moment already happened for the kind of AI that will be super transformative and could kill us all, and like the historical Kitty Hawk moment, it was not immediately a huge deal

The fact that you express this kind of view about AGI erupting one day is why I thought your thing in IEM was saying there was a major algorithmic innovation from chimps to humans, that humans were qualitatively and not just quantitatively better than chimps and this was not because of their larger brain size primarily. But I’m confused because up thread in the discussion of evolution you were emphasizing much more that there was an innovation between dinosaurs and primates, not that there was an innovation between chimps and humans, and you seemed more open to the chimp/human diff being quantitative and brain-size driven than I had thought you’d be. But being open to the chimp-human diff being quantitative/brain-size-driven suggests to me that you should be more open than you are to AGI being developed by slow grinding on the same shit, instead of erupting without much precedent?

The claim I’m making is that Paul’s view predicts a lag and a lot of investment between the first flight and aircraft making a big impact on the travel industry, and predicts that the first flight wouldn’t have immediately made a big impact on the travel industry. In other words Kitty Hawk isn’t a discontinuity in the Paul view because the metrics he’d expect to be continuous are the ones that large numbers of people are trying hard to optimize, like cost per mile traveled or whatnot, not metrics that almost nobody is trying to optimize, like “height flown.”

In other words, it sounds like you’re saying:

• Kitty Hawk is analogous to AGI erupting
• Previous history of travel is analogous to pre-AGI history of AI

While Paul is saying:

• Kitty Hawk is analogous to e.g. AlexNet
• Later history of aircraft is analogous to the post-AlexNet story of AI which we’re in the middle of living, and will continue on to make huge Singularity-causing impacts on the world

But Paul offered to bet with you about literally any quantity you choose?

It still sounds to me like “take a basket of N performance metrics, bet that the model size to perf trend will break upward in > K of them within e.g. 2 or 3 years” should sound good to you, I’m confused why that didn’t. If it does and it’s just about the legwork then I think we could get someone to come up with the benchmarks and stuff for you

Or maybe the same thing but >K of them will break downward, whatever

We could bet about the human perception of sense in language models, for example

I don’t think you need to bet about calendar times from par-human to super-human, and any meta-trend in that quantity. It sounds like Paul is saying “I’ll basically trust the model size to perf trends and predict a 10x bigger model from the same architecture family will get the perf the trends predict,” and you’re pushing back against that saying e.g. that humans won’t find GPT-4 to be subjectively more coherent than GPT-3 and that Paul is neglecting that there could be major innovations in the future that bring down the FLOP/s to get a certain perf by a lot and bend the scaling laws. So why not bet that Paul won’t be as accurate as he thinks he is by following the scaling laws?

I think Paul’s view would say:

• Things certainly happen for the first time
• When they do, they happen at small scale in shitty prototypes, like the Wright Flyer or GPT-1 or AlphaGo or the Atari bots or whatever
• When they’re making a big impact on the world, it’s after a lot of investment and research, like commercial aircrafts in the decades after Kitty Hawk or like the investments people are in the middle of making now with AI that can assist with coding

Paul’s view says that the Kitty Hawk moment already happened for the kind of AI that will be super transformative and could kill us all, and like the historical Kitty Hawk moment, it was not immediately a huge deal

“When they do, they happen at small scale in shitty prototypes, like the Wright Flyer or GPT-1 or AlphaGo or the Atari bots or whatever”

How shitty the prototype is should depend (to a very large extent) on the physical properties of the tech. So I don’t find it confusing (though I currently disagree) when someone says “I looked at a bunch of GPT-3 behavior and it’s cognitively sophisticated enough that I think it’s doing basically what humans are doing, just at a smaller scale. The qualitative cognition I can see going on is just that impressive, taking into account the kinds of stuff I think human brains are doing.”

What I find confusing is, like, treating ten thousand examples of non-AI, non-cognitive-tech continuities (nukes, building heights, etc.) as though they’re anything but a tiny update about ‘will AGI be high-impact’ — compared to the size of updates like ‘look at how smart and high-impact humans were’ and perhaps ‘look at how smart-in-the-relevant-ways GPT-3 is’.

Like, impactfulness is not a simple physical property, so there’s not much reason for different kinds of tech to have similar scales of impact (or similar scales of impact n years after the first prototype). Mainly I’m not sure to what extent we disagree about this, vs. this just being me misunderstanding the role of the ‘most things aren’t high-impact’ argument.

(And yeah, a random historical technology drawn from a hat will be pretty low-impact. But that base rate also doesn’t seem to me like it has much evidential relevance anymore when I update about what specific tech we’re discussing.)

The question is not “will AGI be high impact” — Paul agrees it will, and for any FOOM quantity (like crossing a chimp-to-human-sized gap in a day or whatever) he agrees that will happen eventually too.

The technologies studies in the dataset spanned a wide range in their peak impact on society, and they’re not being used to forecast the peak impact of mature AI tech

Yeah, I’m specifically confused about how we know that the AGI Wright Flyer and its first successors are low-impact, from looking at how low-impact other technologies are (if that is in fact a meaningful-sized update on your view)

Not drawing a comparison about the overall impactfulness of AI / AGI (e.g., over fifteen years)

I think we should try to find ML predictions where you defer to superforecasters and Paul disagrees, since he said he would bet against superforecasters in ML

I think what Paul was saying last night is you find superforecasters betting on some benchmark performance, and he just figures out which side he’d take (and he expects in most/all superforecaster predictions that he would not be deferential, there’s a side he would take)

not really following along with the conversation, but my desire to bet about “whatever you want” was driven in significant part by frustration with Eliezer repeatedly saying things like “people like Paul get surprised by reality” and me thinking that’s nonsense

I still think that if you want to say “this sort of reasoning is garbage empirically” then you ought to be willing to bet about something. If we are just saying “we agree about all of the empirics, it’s just that somehow we have different predictions about AGI” then that’s fine and symmetrical.

even then presumably if you think it’s garbage you should be able to point to some particular future predictions where it would be garbage?

if you used it

and then I can either say “no, I don’t think that’s a valid application for reason X” or “sure, I’m happy to bet”

and it’s possible you can’t find any places where it sticks its neck out in practice (even in your version), but then I’m again just rejecting the claim that it’s empirically ruled out

I think this might be derailing some broader point, but I am provisionally mostly ignoring your point “this doesn’t work in practice” if we can’t find places where we actually foresee disagreements

(which is fine, I don’t think it’s core to your argument)

I don’t know if superforecasters make public bets on ML topics, I was saying I’m happy to bet on ML topics and if your strategy is “look up what superforecasters say” that’s fine and doesn’t change my willingness to bet

I think this is probably not as promising as either (i) dig in on the arguments that are most in dispute (seemed to be some juicier stuff earlier though I’m just focusing on work today) , or (ii) just talking generally about what we expect to see in the next 5 years so that we can at least get more of a vibe looking back

You can bet on the Metaculus AI Tournament forecasts.

https://www.metaculus.com/ai-progress-tournament/

1/3 that RH has an automated proof before sustained 7%/year GWP growth?

I think the clearest indicator is that we have AI that ought to be able to e.g. run the fully automated factory-building factory (not automating mines or fabs, just the robotic manufacturing and construction), but it’s not being deployed or is being deployed with very mild economic impacts

another indicator is that we have AI systems that can fully replace human programmers (or other giant wins), but total investment in improving them is still small

another indicator is a DeepMind demo that actually creates a lot of value (e.g. 10x larger than DeepMind’s R&D costs? or even comparable to DeepMind’s cumulative R&D costs if you do the accounting really carefully and I definitely believe it and it wasn’t replaceable by Brain), it seems like on your model things should “break upwards” and in mine that just doesn’t happen that much

sounds like you may have >90% on automated proof of RH before a few years of 7%/year growth driven by AI? so that would give a pretty significant odds ratio either way

I think “stack more layers gets stuck but a clever idea makes crazy stuff happen” is generally going to be evidence for your view

That said, I’d mostly reject AlphaGo as an example, because it’s just plugging in neural networks to existing go algorithms in almost the most straightforward way and the bells and whistles don’t really matter. But if AlphaZero worked and AlphaGo didn’t, and the system accomplished something impressive/important (like proving RH, or being significantly better at self-contained programming tasks), then that would be a surprise.

And I’d reject LSTM -> transformer or MoE as an example because the quantitative effect size isn’t that big.

But if something like that made the difference between “this algorithm wasn’t scaling before, and now it’s scaling,” then I’d be surprised.

And the size of jump that surprises me is shrinking over time. So in a few years even getting the equivalent of a factor of 4 jump from some clever innovation would be very surprising to me.

I think RH is not that surprising, it’s not at all clear to me where “do formal math” sits on the “useful stuff AI could do” spectrum, I guess naively I’d put it somewhere “in the middle” (though the analogy to board games makes it seem a bit lower, and there is a kind of obvious approach to doing this that seems to be working reasonably well so that also makes it seem lower), and 7% GDP growth is relatively close to the end (ETA: by “close to the end” I don’t mean super close to the end, just far enough along that there’s plenty of time for RH first)

I do think that performance jumps are maybe more dispositive, but I’m afraid that it’s basically going to go like this: there won’t be metrics that people are tracking that jump up, but you’ll point to new applications that people hadn’t considered before, and I’ll say “but those new applications aren’t that valuable” whereas to you they will look more analogous to a world-ending AGI coming out from the blue

like for AGZ I’ll be like “well it’s not really above the deep learning trend if you run it backwards” and you’ll be like “but no one was measuring it before! you can’t make up the trend in retrospect!” and I’ll be like “OK, but the reason no one was measuring it before was that it was worse than traditional go algorithms until like 2 years ago and the upside is not large enough that you should expect a huge development effort for a small edge”

like I’m surprised if a clever innovation does more good than spending 4x more compute

I’m saying that it gets harder over time, don’t expect wins as big as transformers

I think even transformers probably wouldn’t make this cut though?

certainly not vs CNNs

vs RNNs I think the comparison I’d be using to operationalize it is translation, as measured in the original paper

they do make this cut for translation, looks like the number is like 100 >> 4

100x for english-german, more like 10x for english-french, those are the two benchmarks they cite

but both more than 4x

I’m saying I don’t expect ongoing wins that big

I think the key ambiguity is probably going to be about what makes a measurement established/hard-to-improve

I think the relevant feature of the innovation is that the work to find it is small relative to the work that went into the problem to date (though there may be other work on other avenues)

I’m calling that an innovation because it’s a small amount of work

like, it’s not just a claim about EMH, it’s also a claim about the nature of progress

I think AlphaFold counts and is probably if anything a bigger multiplier, it’s just uncertainty over how many people actually worked on the baselines

I mean, I think they are steadily subsiding

as areas grow

I would be surprised to see a 10x today on machine translation

though not so surprised that we can avoid talking about probabilities

yeah

or to make it more surprising, old sota with 10x less compute

I’m more surprised if they get the old performance with 10x less compute though, so that way around is better on all fronts

like, this view also leads me to predict that if I look at the actual amount of manpower that went into alphafold, it’s going to be pretty big relative to the other people submitting to that protein folding benchmark

not plausible claims on problems people care about

I think the comparison is to contemporary benchmarks from one of the 99 other startups who didn’t find the bright idea

that’s the relevant thing on your view, right?

and then I’m predicting that number is already <10 for machine translation and falling (maybe I shouldn’t talk about machine translation or at least not commit to numbers given that I know very little about it, but whatever that’s my estimate), and for other domains it will be <10 by the time they get as crowded as machine translation, and for transformative tasks they will be <2

isn’t there an open-source replication of alphafold?

we could bet about its performance relative to the original

I agree

anyway, my guess is generally that if you are big relative to previous efforts in the area you can make giant improvements, if you are small relative to previous efforts you might get lucky (or just be much smarter) but that gets increasingly unlikely as the field gets bigger

like alexnet and transformers are big wins by groups who are small relative to the rest of the field, but transformers are much smaller than alexnet and future developments will continue to shrink

I mean, if you are the same size as all the prior effort put together?

I’m not surprised if you can totally dominate in that case, especially if prior efforts aren’t well-coordinated

and for things that are done by hobbyists, I wouldn’t be surprised if you can be a bit bigger than an individual hobbyist and dominate

yeah, I think if DeepMind beats Google Brain by 10x compute next year on translation, that’s a significant strike against Paul

I think not that seriously, but more seriously than 2016 and than anything else where you are seeing big swings

and so I’m less surprised than for TAI, but still surprised

I definitely think there’s a big disagreement here about what to expect for pre-end-of-days ML

but lots of concerns about details like what domains are crowded enough to be surprising and how to do comparisons

I mean, to be clear, I think the transformer paper having giant gains is also evidence against paulverse

it’s just that there are really a lot of datapoints, and some of them definitely go against paul’s view

to me it feels like the relevant thing for making the end-of-days forecast is something like “how much of the progress comes from ‘innovations’ that are relatively unpredictable and/or driven by groups that are relatively small, vs scaleup and ‘business as usual’ progress in small pieces?”

I mostly want to know how good Google’s translation is at that time; and if DeepMind’s product is expensive or only shows gains for long texts, I want to know whether there is actually an economic niche for it that is large relative to the R&D cost.

like I’m not sure whether anyone works at all on long-text translation, and I’m not sure if it would actually make Google $ to work on it

great text chat with contractor in indonesia almost certainly meets that bar though

I’m OK if they don’t have the benchmark graph as long as they have some evaluation that other people were trying at, I think real-time chat probably qualifies

also I’m saying 10x less training compute, not inference (but 10x less inference compute is harder)

yes

in practice it seems almost certain that it’s going to be harder to evaluate

though I agree there are really clean versions where they actually measured a benchmark other people work on and can compare training compute directly

(like in the transformer paper)

I’m more surprised by the clean version

I’d want to look at either human evals or logprob, I think probably not? but it’s possible it was

it’s definitely way harder to win at the old task with 10x less compute

I think it’s usually the case that if you scale up far enough past previous sota, you will be able to find tons of techniques needed to make it work at the new scale

but I’m expecting it to be less of a big deal because all experiments will be roughly at the frontier of what is feasible

and so the new thing won’t be able to afford to go 10x bigger

unlike today when we are scaling up spending so fast

but this does make it harder for the next few years at least, which is maybe the key period

(it makes it hard if we are both close enough to the edge that “10x cheaper to get old results” seems unlikely but “getting new results that couldn’t be achieved with 10x more compute and old method” seems likely)

what I basically expect is to (i) roughly know how much performance you get from making models 10x bigger, (ii) roughly know how much someone beat the competition, and then you can compare the numbers

(but “performance from making models 10x bigger” depends a lot on exactly how big they were and whether you are in a regime with unfavorable scaling)

I am also expecting a general scale up in ML training runs over time, though it’s plausible that you also expect that until the end of days and just expect a much earlier end of days

that’s going to kill the “they spent 100x more compute” announcements soon enough

like, that’s easy when “100x more” means $1M, it’s a bit hard when “100x more” means $100M, it’s not going to happen except on the most important tasks when “100x more” means $10B

I’m not surprised by announcements like protein folding, it’s not that the world overall gets more and more hostile to big wins, it’s that any industry gets more and more hostile as it gets bigger (or across industries, they get more and more hostile as the stakes grow)

(I mean, the protein folding thing is a datapoint against my view, but it’s not that much evidence and it’s not getting bigger over time)

yeah, but doesn’t your view expect more innovations for any given problem?

like, it’s not just that you think the universe of weird profitable applications is larger, you also think AI progress is just more driven by innovations, right?

otherwise it feels like the whole game is about whether you think that AI-automating-AI-progress is a weird application or something that people will try on

yeah, but then don’t you see big wins from the next transformers?

and you think those just keep happening even as fields mature

or you mean that they might slow down because people stop working on them?

it seems clear that it will be more than 10x relative to GPT

I guess I don’t know what GPT-like architecture means, but from what you say it seems like normal progress would result in a non-GPT-like architecture

so I don’t think I’m disagreeing with that

do you think transformers are a big insight?

(is adding soft attention to LSTMs a big insight?)

(I think the intellectual history of transformers is a lot like “take the LSTM out of the LSTM with attention”)

that really really seems like the accumulation of small insights

like, residual connections are the single biggest thing

and relus also help

and batch normalization helps

and attention is better than lstms

you could also call that the accumulation of big insights, but the point is that it’s an accumulation of a lot of stuff

mostly developed in different places

(I think transformers are a significantly smaller jump than previous improvements)

also a thing we could guess about though

also 10-100x is still actually surprising to me for transformers

so I guess lesson learned

in general if you take anything developed at scale X and try to scale it way past X I think it won’t work

or like, it will work much worse than something that continues to get tweaked

yeah, I mean that given everything I know I am surprised that transformers were as large as a 100x improvement on translation

in that paper

yeah, I am definitely interested to understand a bit better what’s up there

but tentatively I’m sticking to my guns on the original prediction

if you have random 10-20 person teams getting 100x speedups versus prior sota

as we approach TAI

that’s so far from paulverse

anyway, seems like a good approach to finding a concrete disagreement

and even looking back at this conversation would be a start for diagnosing who is more right in hindsight

main thing is to say how quickly and in what industries I’m how surprised

I think the economically important prediction doesn’t really need that much of “within a few years”

like the total stakes have just been low to date

none of the deep learning labs are that close to paying for themselves

so we’re not in the regime where “economic niche > R&D budget”

we are still in the paulverse-consistent regime where investment is driven by the hope of future wins

though paul is surprised that R&D budgets aren’t more larger than the economic value

I’ll register considerable skepticism

it seems easier to just talk about ML tasks that people work on

it seems really hard to arbitrate the “all the important niches are invested in” stuff in a way that’s correlated with takeoff

whereas the “we should be making a big chunk of our progress from insights” seems like it’s easier

though I understand that your view could be disjunctive, of either “AI will have hidden secrets that yield great intelligence,” or “there are hidden secret applications that yield incredible profit”

(sorry that statement is crude / not very faithful)

should follow up on this in the future, off for now though

I’ll interject some points re the earlier discussion about how animal data relates to the ‘AI scaling to AGI’ thesis.

1. In humans it’s claimed the IQ-job success correlation varies by job, For a scientist or doctor it might be 0.6+, for a low complexity job more like 0.4, or more like 0.2 for simple repetitive manual labor. That presumably goes down a lot with less in the way of hands, or focused on low density foods like baleen whales or grazers. If it’s 0.1 for animals like orcas or elephants, or 0.05, then there’s 4-10x less fitness return to smarts.

2. But they outmass humans by more than 4-10x. Elephants 40x, orca 60x+. Metabolically (20 watts divided by BMR of the animal) the gap is somewhat smaller though, because of metabolic scaling laws (energy scales with 3/4 or maybe 2/3 power, so ).

https://en.wikipedia.org/wiki/Kleiber%27s_law

If dinosaurs were poikilotherms, that’s a 10x difference in energy budget vs a mammal of the same size, although there is debate about their metabolism.

3. If we’re looking for an innovation in birds and primates, there’s some evidence of ‘hardware’ innovation rather than ‘software.’ Herculano-Houzel reports in The Human Advantage (summarizing much prior work neuron counting) different observational scaling laws for neuron number with brain mass for different animal lineages.

We were particularly interested in cellular scaling differences that might have arisen in primates. If the same rules relating numbers of neurons to brain size in rodents (6)

The brain of the capuchin monkey, for instance, weighing 52 g, contains >3× more neurons in the cerebral cortex and ≈2× more neurons in the cerebellum than the larger brain of the capybara, weighing 76 g.

[Editor’s Note: Quote source is “Cellular scaling rules for primate brains.”]

In rodents brain mass increases with neuron count n^1.6, whereas it’s close to linear (n^1.1) in primates. For cortex neurons and cortex mass 1.7 and 1.0. In general birds and primates are outliers in neuron scaling with brain mass.

Note also that bigger brains with lower neuron density have longer communication times from one side of the brain to the other. So primates and birds can have faster clock speeds for integrated thought than a large elephant or whale with similar neuron count.

4. Elephants have brain mass ~2.5x human, and 3x neurons, but 98% of those are in the cerebellum (vs 80% in or less in most animals; these are generally the tiniest neurons and seem to do a bunch of fine motor control). Human cerebral cortex has 3x the neurons of the elephant cortex (which has twice the mass). The giant cerebellum seems like controlling the very complex trunk.

https://nautil.us/issue/35/boundaries/the-paradox-of-the-elephant-brain

Blue whales get close to human neuron counts with much larger brains.

https://en.wikipedia.org/wiki/List_of_animals_by_number_of_neurons

5. As Paul mentioned, human brain volume correlation with measures of cognitive function after correcting for measurement error on the cognitive side is in the vicinity of 0.3-0.4 (might go a bit higher after controlling for non-functional brain volume variation, lower from removing confounds). The genetic correlation with cognitive function in this study is 0.24:

https://www.nature.com/articles/s41467-020-19378-5

So it accounts for a minority of genetic influences on cognitive ability. We’d also expect a bunch of genetic variance that’s basically disruptive mutations in mutation-selection balance (e.g. schizophrenia seems to be a result of that, with schizophrenia alleles under negative selection, but a big mutational target, with the standing burden set by the level of fitness penalty for it; in niches with less return to cognition the mutational surface will be cleaned up less frequently and have more standing junk).

Other sources of genetic variance might include allocation of attention/learning (curiosity and thinking about abstractions vs immediate sensory processing/alertness), length of childhood/learning phase, motivation to engage in chains of thought, etc.

Overall I think there’s some question about how to account for the full genetic variance, but mapping it onto the ML experience with model size, experience and reward functions being key looks compatible with the biological evidence. I lean towards it, although it’s not cleanly and conclusively shown.

Regarding economic impact of AGI, I do not buy the ‘regulation strangles all big GDP boosts’ story.

The BEA breaks down US GDP by industry here (page 11):

https://www.bea.gov/sites/default/files/2021-06/gdp1q21_3rd_1.pdf

As I work through sectors and the rollout of past automation I see opportunities for large-scale rollout that is not heavily blocked by regulation. Manufacturing is still trillions of dollars, and robotic factories are permitted and produced under current law, with the limits being more about which tasks the robots work for at low enough cost (e.g. this stopped Tesla plans for more completely robotic factories). Also worth noting manufacturing is mobile and new factories are sited in friendly jurisdictions.

Software to control agricultural machinery and food processing is also permitted.

Warehouses are also low-regulation environments with logistics worth hundreds of billions of dollars. See Amazon’s robot-heavy warehouses limited by robotics software.

Driving is hundreds of billions of dollars, and Tesla has been permitted to use Autopilot, and there has been a lot of regulator enthusiasm for permitting self-driving cars with humanlike accident rates. Waymo still hasn’t reached that it seems and is lowering costs.

Restaurants/grocery stores/hotels are around a trillion dollars. Replacing humans in vision/voice tasks to take orders, track inventory (Amazon Go style), etc is worth hundreds of billions there and mostly permitted. Robotics cheap enough to replace low-wage labor there would also be valuable (although a lower priority than high-wage work if compute and development costs are similar).

Software is close to a half trillion dollars and the internals of software development are almost wholly unregulated.

Finance is over a trillion dollars, with room for AI in sales and management.

Sales and marketing are big and fairly unregulated.

In highly regulated and licensed professions like healthcare and legal services, you can still see a licensee mechanically administer the advice of the machine, amplifying their reach and productivity.

Even in housing/construction there’s still great profits to be made by improving the efficiency of what construction is allowed (a sector worth hundreds of billions).

If you’re talking about legions of super charismatic AI chatbots, they could be doing sales, coaching human manual labor to effectively upskill it, and providing the variety of activities discussed above. They’re enough to more than double GDP, even with strong Baumol effects/cost disease, I’d say.

Although of course if you have AIs that can do so much the wages of AI and hardware researchers will be super high, and so a lot of that will go into the intelligence explosion, while before that various weaknesses that prevent full automation of AI research will also mess up activity in these other sectors to varying degrees.

Re discontinuity and progress curves, I think Paul is right. AI Impacts went to a lot of effort assembling datasets looking for big jumps on progress plots, and indeed nukes are an extremely high percentile for discontinuity, and were developed by the biggest spending power (yes other powers could have bet more on nukes, but didn’t, and that was related to the US having more to spend and putting more in many bets), with the big gains in military power per $ coming with the hydrogen bomb and over the next decade.

https://aiimpacts.org/category/takeoff-speed/continuity-of-progress/discontinuous-progress-investigation/

For measurable hardware and software progress (Elo in games, loss on defined benchmarks), you have quite continuous hardware progress, and software progress that is on the same ballpark, and not drastically jumpy (like 10 year gains in 1), moreso as you get to metrics used by bigger markets/industries.

I also agree with Paul’s description of the prior Go trend, and how DeepMind increased $ spent on Go software enormously. That analysis was a big part of why I bet on AlphaGo winning against Lee Sedol at the time (the rest being extrapolation from the Fan Hui version and models of DeepMind’s process for deciding when to try a match).

Little of Open Phil’s opinions are independent of Carl, the source of all opinions