As promised, here’s a write-up of some thoughts from my end. In particular, since I’ve spent a lot of the debate poking Eliezer about his views, I’ve tried here to put forward more positive beliefs of my own in this doc (along with some more specific claims): [GDocs link]

We take as a starting observation that a number of “grand challenges” in AI have been solved by AIs that are very far from the level of generality which people expected would be needed. Chess, once considered to be the pinnacle of human reasoning, was solved by an algorithm that’s essentially useless for real-world tasks. Go required more flexible learning algorithms, but policies which beat human performance are still nowhere near generalising to anything else; the same for StarCraft, DOTA, and the protein folding problem. Now it seems very plausible that AIs will even be able to pass (many versions of) the Turing Test while still being a long way from AGI.

Perhaps we can quantify the Turing test by asking something like:

• What percentile of competence is the judge?
• What percentile of competence are the humans who the AI is meant to pass as?
• How much effort does the judge put in (measured in, say, hours of strategic preparation)?

Does this framing seem reasonable to you? And if so, what are the highest numbers for each of these metrics that correspond to a Turing test which an AI could plausibly pass before the world ends?

I expect this trend to continue until after we have AIs which are superhuman at mathematical theorem-proving, programming, many other white-collar jobs, and many types of scientific research. It seems like Eliezer doesn’t. I’ll highlight two specific disagreements which seem to play into this.

A first disagreement is related to Eliezer’s characterisation of GPT-3 as a shallow pattern-memoriser. I think there’s a continuous spectrum between pattern-memorisation and general intelligence. In order to memorise more and more patterns, you need to start understanding them at a high level of abstraction, draw inferences about parts of the patterns based on other parts, and so on. When those patterns are drawn from the real world, then this process leads to the gradual development of a world-model.

This position seems more consistent with the success of deep learning so far than Eliezer’s position (although my advocacy of it loses points for being post-hoc; I was closer to Eliezer’s position before the GPTs). It also predicts that deep learning will lead to agents which can reason about the world in increasingly impressive ways (although I don’t have a strong position on the extent to which new architectures and algorithms will be required for that). I think that the spectrum from less to more intelligent animals (excluding humans) is a good example of what it looks like to gradually move from pattern-memorisation to increasingly sophisticated world-models and abstraction capabilities.

I expect that Eliezer won’t claim that pattern-memorisation is unrelated to general intelligence, but will claim that a pattern-memoriser needs to undergo a sharp transition in its cognitive algorithms before it can reason reliably about novel domains (like open scientific problems) – with his main argument for that being the example of the sharp transition undergone by humans.

However, it seems unlikely to me that humans underwent a major transition in our underlying cognitive algorithms since diverging from chimpanzees, because our brains are so similar to those of chimps, and because our evolution from chimps didn’t take very long. This evidence suggests that we should favour explanations for our success which don’t need to appeal to big algorithmic changes, if we have any such explanations; and I think we do. More specifically, I’d characterise the three key differences between humans and chimps as:

1. Humans have bigger brains.
2. Humans have a range of small adaptations primarily related to motivation and attention, such as infant focus on language and mimicry, that make us much better at cultural learning.
3. Humans grow up in a rich cultural environment.

bigger brains

I recall a 3-4x difference; but this paper says 5-6x for frontal cortex: https://www.nature.com/articles/nn814

language and mimicry

“apes are unable to ape sounds” claims david deutsch in “the beginning of infinity”

[Humans grow up in a rich cultural environment.]

much richer cultural environment including deliberate teaching

I claim that the discontinuity between the capabilities of humans and chimps is mainly explained by the general intelligence of chimps not being aimed in the direction of learning the skills required for economically valuable tasks, which in turn is mainly due to chimps lacking the “range of small adaptations” mentioned above.

My argument is a more specific version of Paul’s claim that chimp evolution was not primarily selecting for doing things like technological development. In particular, it was not selecting for them because no cumulative cultural environment existed while chimps were evolving, and selection for the application of general intelligence to technological development is much stronger in a cultural environment. (I claim that the cultural environment was so limited before humans mainly because cultural accumulation is very sensitive to transmission fidelity.)

By contrast, AIs will be trained in a cultural environment (including extensive language use) from the beginning, so this won’t be a source of large gains for later systems.

more specific version of Paul’s claim

Based on some of Paul’s recent comments, this may be what he intended all along; though I don’t recall his original writings on takeoff speeds making this specific argument.

(I claim that the cultural environment was so limited before humans mainly because cultural accumulation is very sensitive to transmission fidelity.)

There can be other areas with superlinear effects from repeated application of  a skill. There’s reason to think that the most productive complex industries tend to have that character.

Making individual minds able to correctly execute long chains of reasoning by reducing per-step error rate could plausibly have very superlinear effects in programming, engineering, management, strategy, persuasion, etc. And you could have new forms of ‘super-culture’ that don’t work with humans.

https://ideas.repec.org/a/eee/jeborg/v85y2013icp1-10.htmlhttps://ideas.repec.org/a/eee/jeborg/v85y2013icp1-10.html

If true, this argument would weigh against Eliezer’s claims about agents which possess a core of general intelligence being able to easily apply that intelligence to a wide range of tasks. And I don’t think that Eliezer has a compelling alternative explanation of the key cognitive differences between chimps and humans (the closest I’ve seen in his writings is the brainstorming at the end of this post).

If this is the case, I notice an analogy between Eliezer’s argument against Kurzweil, and my argument against Eliezer. Eliezer attempted to put microfoundations underneath the trend line of Moore’s law, which led to a different prediction than Kurzweil’s straightforward extrapolation. Similarly, my proposed microfoundational explanation of the chimp-human gap gives rise to a different prediction than Eliezer’s more straightforward, non-microfoundational extrapolation.

Good to know; apologies for the incorrect inference.

Given that this seems like a big sticking point in the debate overall, do you have any ideas about how to move forward while avoiding infohazards?

My position makes some predictions about hypothetical cases:

1. If chimpanzees had the same motivational and attention-guiding adaptations towards cultural learning and cooperation that humans do, and were raised in equally culturally-rich environments, then they could become economically productive workers in a range of jobs (primarily as manual laborers, but plausibly also for operating machinery, etc).
   1. Results from chimps raised in human families, like Washoe, seem moderately impressive, although still very uncertain. There’s probably a lot of bias towards positive findings – but on the other hand, it’s only been done a handful of times, and I expect that more practice at it would lead to much better results.
   2. Comparisons between humans and chimps which aren’t raised in similar ways to humans are massively biased towards humans. For the purposes of evaluating general intelligence, comparisons between chimpanzees and feral children seem fairer (although it’s very hard to know how much the latter were affected by non-linguistic childhoods as opposed to abuse or pre-existing disabilities).
2. Consider a hypothetical species which has the same level of “general intelligence” that chimpanzees currently have, but is as well-adapted to the domains of abstract reasoning and technological development as chimpanzee behaviour is to the domain of physical survival (e.g. because they evolved in an artificial environment where their fitness was primarily determined by their intellectual contributions). I claim that this species would have superhuman scientific research capabilities, and would be able to make progress in novel areas of science (analogously to how chimpanzees can currently learn to navigate novel physical landscapes).
   1. Insofar as Eliezer doubts this, but does believe that this species could outperform a society of village idiots at scientific research, then he needs to explain why the village-idiot-to-Einstein gap is so significant in this context but not in others.
   2. However, this is a pretty weird thought experiment, and maybe doesn’t add much to our existing intuitions about AIs. My main intention here is to point at how animal behaviour is really really well-adapted to physical environments, in a way which makes people wonder what it would be like to be really really well-adapted to intellectual environments.
3. I claim that the difficulty of human-level oracle AGIs matching humans Consider an AI which has been trained only to answer questions, and is now human-level at doing so. I claim that the difficulty of this AI matching humans at a range of real-world tasks (without being specifically trained to do so) would be much closer to the difficulty of teaching chimps to do science, than the difficulty of teaching adult humans to do abstract reasoning about a new domain.
   1. The analogy here is: chimps have reasonably general intelligence, but it’s hard for them to apply it to science because they weren’t trained to apply intelligence to that. Likewise, human-level oracle AGIs have general intelligence, but it’ll be hard for them to apply it to influencing the world because they weren’t trained to apply intelligence to that.

village-idiot-to-Einstein gap

I wonder to what extent you can model within-species intelligence differences partly just as something like hyperparameter search – if you have a billion humans with random variation in their neural/cognitive traits, the top human will be a lot better than average. Then you could say something like:

• humans are the dumbest species you could have where the distribution of intelligence in each generation is sufficient for cultural accumulation
• that by itself might not imply a big gap from chimps
• but human society has much larger population, so the smartest individuals are much smarter

I think Eliezer’s response (which I’d agree with) would be that the cognitive difference between the best humans and normal humans is strongly constrained by the fact that we’re all one species who can interbreed with each other. And so our cognitive variation can’t be very big compared with inter-species variation (at the top end at least; although it could at the bottom end via things breaking).

I think that’s not obviously true – it’s definitely possible that there’s a lot of random variation due to developmental variation etc. If that’s the case then population size could create large within-species differences

Edited to clarify what I meant in this context (and remove the word “oracle” altogether).

The analogy here is: chimps have reasonably general intelligence, but it’s hard for them to apply it to science because they weren’t trained to apply intelligence to that. Likewise, human-level oracle AGIs have general intelligence, but it’ll be hard for them to apply it to influencing the world because they weren’t trained to apply intelligence to that.

this is not intuitive to me; it seems pretty plausible that the subtasks of predicting the world and of influencing the world are much more similar than the subtasks of surviving in a chimp society are to the subtasks of doing science

I think Eliezer’s position is that all of these tasks are fairly similar if you have general intelligence. E.g. he argued that the difference between very good theorem-proving and influencing the world is significantly smaller than people expect. So even if you’re right, I think his position is too strong for your claim to help him. (I expect him to say that I’m significantly overestimating the extent to which chimps are running general cognitive algorithms).

I wasn’t trying to defend his position, just disagreeing with you 😛

More specific details

Here are three training regimes which I expect to contribute to AGI:

• Self-supervised training – e.g. on internet text, code, books, videos, etc.
• Task-based RL – agents are rewarded (likely via human feedback, and some version of iterated amplification) for doing well on bounded tasks.
• Open-ended RL – agents are rewarded for achieving long-term goals in rich environments.

Most of my probability of catastrophe comes from AGIs trained primarily via open-ended RL. Although IA makes these scenarios less likely by making task-based RL more powerful, it doesn’t seem to me that IA tackles the hardest case (of aligning agents trained via open-ended RL) head-on. But disaster from open-ended RL also seems a long way away – mainly because getting long-term real-world feedback is very slow, and I expect it to be hard to create sufficiently rich artificial environments. By that point I do expect the strategic landscape to be significantly different, because of the impact of task-based RL.

Perhaps the best way to pin down disagreements in our expectations about the effects of the strategic landscape is to identify some measures that could help to reduce AGI risk, and ask how seriously key decision-makers would need to take AGI risk for each measure to be plausible, and how powerful and competent they would need to be for that measure to make a significant difference. Actually, let’s lump these metrics together into a measure of “amount of competent power applied”. Some benchmarks, roughly in order (and focusing on the effort applied by the US):

• Banning chemical/biological weapons
• COVID
  • Key points: mRNA vaccines, lockdowns, mask mandates
• Nuclear non-proliferation
  • Key points: Nunn-Lugar Act, stuxnet, various treaties
• The International Space Station
  • Cost to US: ~$75 billion
• Climate change
  • US expenditure: >$154 billion (but not very effectively)
• Project Apollo
  • Wikipedia says that Project Apollo “was the largest commitment of resources ($156 billion in 2019 US dollars) ever made by any nation in peacetime. At its peak, the Apollo program employed 400,000 people and required the support of over 20,000 industrial firms and universities.”
• WW1
• WW2

Here are some wild speculations (I just came up with this framework, and haven’t thought about these claims very much):

1. The US and China preventing any other country from becoming a leader in AI requires about as much competent power as banning chemical/biological weapons.
2. The US and China enforcing a ban on AIs above a certain level of autonomy requires about as much competent power as the fight against climate change.
   1. In this scenario, all the standard forces which make other types of technological development illegal have pushed towards making autonomous AGI illegal too.
3. Launching a good-faith joint US-China AGI project requires about as much competent power as launching Project Apollo.
   1. According to this article, Kennedy (and later Johnson) made several offers (some of which were public) of a joint US-USSR Moon mission, which Khrushchev reportedly came close to accepting. Of course this is a long way from actually doing a joint project (and it’s not clear how reliable the source is), but it still surprised me a lot, given that I viewed the “space race” as basically a zero-sum prestige project. If your model predicted this, I’d be interested to hear why.

AI ban much, much harder than chemical weapons ban. Indeed chemical weapons were low military utility, that was central to the deal, and they have still been used subsequently.

An AI ban is about as easy as banning advanced metal-forging techniques in current use so nobody can get ahead of the USA and China in making airplanes. That would be HARD and likewise require credible threats of military action against former allies.

If large amounts of compute relative to today are needed (and presumably Eliezer rejects this), the fact that there is only a single global leading node chip supply chain makes it vastly easier than metal forging, which exists throughout the world and is vastly cheaper.

Sharing with allies (and at least embedding allies to monitor US compliance) also reduces the conflict side.

OTOH, if compute requirements were super low then it gets a lot worse.

And the biological weapons ban failed completely: the Soviets built an enormous bioweapons program, the largest ever, after agreeing to the ban, and the US couldn’t even tell for sure they were doing so.

China has been trying for some time to build its own and has failed with tens of billions of dollars (but has captured some lagging node share), but would be substantially more likely to succeed with a trillion dollar investment. That said, it is hard to throw money at these things and the tons of tacit knowledge/culture/supply chain networks are tough to replicate. Also many ripoffs of the semiconductor subsidies have occurred. Getting more NASA/Boeing and less SpaceX is a plausible outcome even with huge investment.

They are trying to hire people away from the existing supply chain to take its expertise and building domestic skills with the lagging nodes.

I’ll read that as including the many suppliers of ASML (one EUV machine has over 100,000 parts, many incredibly fancy or unique). It’s just a matter of how many years it takes. I think Earth fails to rebuild that capacity in 2 years but succeeds in 10.

“A study this spring by Boston Consulting Group and the Semiconductor Industry Association estimated that creating a self-sufficient chip supply chain would take at least $1 trillion and sharply increase prices for chips and products made with them…The situation underscores the crucial role played by ASML, a once obscure company whose market value now exceeds $285 billion. It is “the most important company you never heard of,” said C.J. Muse, an analyst at Evercore ISI.”

https://www.nytimes.com/2021/07/04/technology/tech-cold-war-chips.html

So far I haven’t talked about how much competent power I actually expect people to apply to AI governance. I don’t think it’s useful for Eliezer and me to debate this directly, since it’s largely downstream from most of the other disagreements we’ve had. In particular, I model him as believing that there’ll be very little competent power applied to prevent AI risk from governments and wider society, partly because he expects a faster takeoff than I do, and partly because he has a lower opinion of governmental competence than I do. But for the record, it seems likely to me that there’ll be as much competent effort put into reducing AI risk by governments and wider society as there has been into fighting COVID; and plausibly (but not likely) as much as fighting climate change.

One key factor is my expectation that arguments about the importance of alignment will become much stronger as we discover more compelling examples of misalignment. I don’t currently have strong opinions about how compelling the worst examples of misalignment before catastrophe are likely to be; but identifying and publicising them seems like a particularly effective form of advocacy, and one which we should prepare for in advance.

The predictable accumulation of easily-accessible evidence that AI risk is important is one example of a more general principle: that it’s much easier to understand, publicise, and solve problems as those problems get closer and more concrete. This seems like a strong effect to me, and a key reason why so many predictions of doom throughout history have failed to come true, even when they seemed compelling at the time they were made.

Upon reflection, however, I think that even taking this effect into account, the levels of competent power required for the interventions mentioned above are too high to justify the level of optimism about AI governance that I started our debate with. On the other hand, I found Eliezer’s arguments about consequentialism less convincing than I expected. Overall I’ve updated that AI risk is higher than I previously believed; though I expect my views to be quite unsettled while I think more, and talk to more people, about specific governance interventions and scenarios.

Quoth Paul:

seems like you have to make the wright flyer much better before it’s important, and that it becomes more like an industry as that happens, and that this is intimately related to why so few people were working on it

Is this basically saying ‘the Wright brothers didn’t personally capture much value by inventing heavier-than-air flying machines, and this was foreseeable, which is why there wasn’t a huge industry effort already underway to try to build such machines as fast as possible.’ ?

My maybe-wrong model of Eliezer says here ‘the Wright brothers knew a (Thielian) secret’, while my maybe-wrong model of Paul instead says:

• They didn’t know a secret — it was obvious to tons of people that you could do something sorta like what the Wright brothers did and thereby invent airplanes; the Wright brothers just had unusual non-monetary goals that made them passionate to do a thing most people didn’t care about.
• Or maybe it’s better to say: they knew some specific secrets about physics/engineering, but only because other people correctly saw ‘there are secrets to be found here, but they’re stamp-collecting secrets of little economic value to me, so I won’t bother to learn the secrets’. ~Everyone knows where the treasure is located, and ~everyone knows the treasure won’t make you rich.

I mean, I think they knew a bit of stuff, but it generally takes a lot of stuff to make something valuable, and the more people have been looking around in an area the more confident you can be that it’s going to take a lot of stuff to do much better, and it starts to look like an extremely strong regularity for big industries like ML or semiconductors

it’s pretty rare to find small ideas that don’t take a bunch of work to have big impacts

I don’t know exactly what a thielian secret is (haven’t read the reference and just have a vibe)

straightening it out a bit, I have 2 beliefs that combine disjunctively: (i) generally it takes a lot of work to do stuff, as a strong empirical fact about technology, (ii) generally if the returns are bigger there are more people working on it, as a slightly-less-strong fact about sociology

secrets = important undiscovered information (or information that’s been discovered but isn’t widely known), that you can use to get an edge in something. https://www.lesswrong.com/posts/ReB7yoF22GuerNfhH/thiel-on-secrets-and-indefiniteness

There seems to be a Paul/Eliezer disagreement about how common these are in general. And maybe a disagreement about how much more efficiently humanity discovers and propagates secrets as you scale up the secret’s value?

My guess is that everyone agrees with claims 1, 2, and 3 here (please let me know if I’m wrong!):

1. The history of humanity looks less like Long Series of Cheat Codes World, and more like Well-Designed Game World.

In Long Series of Cheat Codes World, human history looks like this, over and over: Some guy found a cheat code that totally outclasses everyone else and makes him God or Emperor, until everyone else starts using the cheat code too (if the Emperor allows it). After which things are maybe normal for another 50 years, until a new Cheat Code arises that makes its first adopters invincible gods relative to the previous tech generation, and then the cycle repeats.

In Well-Designed Game World, you can sometimes eke out a small advantage, and the balance isn’t perfect, but it’s pretty good and the leveling-up tends to be gradual. A level 100 character totally outclasses a level 1 character, and some level transitions are a bigger deal than others, but there’s no level that makes you a god relative to the people one level below you.

2. General intelligence took over the world once. Someone who updated on that fact but otherwise hasn’t thought much about the topic should not consider it ‘bonkers’ that machine general intelligence could take over the world too, even though they should still consider it ‘bonkers’ that eg a coffee startup could take over the world.

(Because beverages have never taken over the world before, whereas general intelligence has; and because our inside-view models of coffee and of general intelligence make it a lot harder to imagine plausible mechanisms by which coffee could make someone emperor, kill all humans, etc., compared to general intelligence.)

(In the game analogy, the situation is a bit like ‘I’ve never found a crazy cheat code or exploit in this game, but I haven’t ruled out that there is one, and I heard of a character once who did a lot of crazy stuff that’s at least suggestive that she might have had a cheat code.’)

3. AGI is arising in a world where agents with science and civilization already exist, whereas humans didn’t arise in such a world. This is one reason to think AGI might not take over the world, but it’s not a strong enough consideration on its own to make the scenario ‘bonkers’ (because AGIs are likely to differ from humans in many respects, and it wouldn’t obviously be bonkers if the first AGIs turned out to be qualitatively way smarter, cheaper to run, etc.).

—

If folks agree with the above, then I’m confused about how one updates from the above epistemic state to ‘bonkers’.

It was to a large extent physics facts that determined how easy it was to understand the feasibility of nukes without (say) decades of very niche specialized study. Likewise, it was physics facts that determined you need rare materials, many scientists, and a large engineering+infrastructure project to build a nuke. In a world where the physics of nukes resulted in it being some PhD’s quiet ‘nobody thinks this will work’ project like Andrew Wiles secretly working on a proof of Fermat’s Last Theorem for seven years, that would have happened.

If an alien came to me in 1800 and told me that totally new physics would let future humans build city-destroying superbombs, then I don’t see why I should have considered it bonkers that it might be lone mad scientists rather than nations who built the first superbomb. The ‘lone mad scientist’ scenario sounds more conjunctive to me (assumes the mad scientist knows something that isn’t widely known, AND has the ability to act on that knowledge without tons of resources), so I guess it should have gotten less probability, but maybe not dramatically less?

‘Mad scientist builds city-destroying weapon in basement’ sounds wild to me, but I feel like almost all of the actual unlikeliness comes from the ‘city-destroying weapons exist at all’ part, and then the other parts only moderately lower the probability.

Likewise, I feel like the prima-facie craziness of basement AGI mostly comes from ‘generally intelligence is a crazy thing, it’s wild that anything could be that high-impact’, and a much smaller amount comes from ‘it’s wild that something important could happen in some person’s basement’.

—

It does structurally make sense to me that Paul might know things I don’t about GPT-3 and/or humans that make it obvious to him that we roughly know the roadmap to AGI and it’s this.

If the entire ‘it’s bonkers that some niche part of ML could crack open AGI in 2026 and reveal that GPT-3 (and the mainstream-in-2026 stuff) was on a very different part of the tech tree’ view is coming from a detailed inside-view model of intelligence like this, then that immediately ends my confusion about the argument structure.

I don’t understand why you think you have the roadmap, and given a high-confidence roadmap I’m guessing I’d still put more probability than you on someone finding a very different, shorter path that works too. But the argument structure “roadmap therefore bonkers” makes sense to me.

If there are meant to be other arguments against ‘high-impact AGI via niche ideas/techniques’ that are strong enough to make it bonkers, then I remain confused about the argument structure and how it can carry that much weight.

I can imagine an inside-view model of human cognition, GPT-3 cognition, etc. that tells you ‘AGI coming from nowhere in 3 years is bonkers’; I can’t imagine an ML-is-a-reasonably-efficient-market argument that does the same, because even a perfectly efficient market isn’t omniscient and can still be surprised by undiscovered physics facts that tell you ‘nukes are relatively easy to build’ and ‘the fastest path to nukes is relatively hard to figure out’.

(Caveat: I’m using the ‘basement nukes’ and ‘Fermat’s last theorem’ analogy because it helps clarify the principles involved, not because I think AGI will be that extreme on the spectrum.)

Oh, I also wouldn’t be confused by a view like “I think it’s 25% likely we’ll see a more Eliezer-ish world. But it sounds like Eliezer is, like, 90% confident that will happen, and that level of confidence (and/or the weak reasoning he’s provided for that confidence) seems bonkers to me.”

The thing I’d be confused by is e.g. “ML is efficient-ish, therefore the out-of-the-blue-AGI scenario itself is bonkers and gets, like, 5% probability.”

I feel confused about the role “innovations are almost always low-impact” plays in slow-takeoff-ish views.

Suppose I think that there’s some reachable algorithm that’s different from current approaches, and can do par-human scientific reasoning without requiring tons of compute.

The existence or nonexistence of such an algorithm is just a fact about the physical world. If I imagine one universe where such an algorithm exists, and another where it doesn’t, I don’t see why I should expect that one of those worlds has more discontinuous change in GWP, ship sizes, bridge lengths, explosive yields, etc. (outside of any discontinuities caused by the advent of humans and the advent of AGI)? What do these CS facts have to do with the other facts?

But AI Impacts seems to think there’s an important connection, and a large number of facts of the form ‘steamships aren’t like nukes’ seem to undergird a lot of Paul’s confidence that the scenario I described —

(“there’s some reachable algorithm that’s different from current approaches, and can do par-human scientific reasoning without requiring tons of compute.”)

— is crazy talk. (Unless I’m misunderstanding. As seems actually pretty likely to me!)

(E.g., Paul says “To me your model just seems crazy, and you are saying it predicts crazy stuff at the end but no crazy stuff beforehand”, and one of the threads of the timelines conversation has been Paul asking stuff like “do you want to give any example other than nuclear weapons of technologies with the kind of discontinuous impact you are describing?”.)

Possibilities that came to mind for me:

1. The argument is ‘reality keeps surprising us with how continuous everything else is, so we seem to have a cognitive bias favoring discontinuity, so we should have a skeptical prior about our ability to think our way to ‘X is discontinuous’ since our brains are apparently too broken to do that well?

(But to get from 1 to ‘discontinuity models are batshit’ we surely need something more probability-mass-concentrating than just a bias argument?)

2. The commonality between steamship sizes, bridge sizes, etc. and AGI is something like ‘how tractable is the world?’. A highly tractable world, one whose principles are easy to understand and leverage, will tend to have more world-shatteringly huge historical breakthroughs in various problems, and will tend to see a larger impact from the advent of humans and the advent of AGI.

Our world looks much less tractable, so even if there’s a secret sauce to building AGI, we should expect the resultant AGI to be a lot less impactful.

I endorse #2 (although I think more weakly than Paul does) and would also add #3: another commonality is something like “how competitive is innovation?”

@RobBensinger It’s showing us a fact about the vast space of ideas and technologies we’ve already explored that they are not so concentrated and lumpy that the law of large numbers doesn’t work well as a first approximation in a world with thousands or millions of people contributing. And that specifically includes past computer science innovation.

So the ‘we find a secret sauce algorithm that causes a massive unprecedented performance jump, without crappier predecessors’ is a ‘separate, additional miracle’ at exactly the same time as the intelligence explosion is getting going. You can get hyperbolic acceleration from increasing feedbacks from AI to AI hardware and software, including crazy scale-up at the end, as part of a default model. But adding on to it that AGI is hit via an extremely large performance jump of a type that is very rare, takes a big probability penalty.

And the history of human brains doesn’t seem to provide strong evidence of a fundamental software innovation, vs hardware innovation and gradual increases in selection applied to cognition/communication/culture.

The fact that, e.g. AIs are mastering so much math and language while still wielding vastly infrahuman brain-equivalents, and crossing human competence in many domains (where there was ongoing effort) over decades is significant evidence for something smoother than the development of modern humans and their culture.

That leaves me not expecting a simultaneous unusual massive human concentrated algorithmic leap with AGI, although I expect wildly accelerating progress from increasing feedbacks at that time. Crossing a given milestone is disproportionately likely to happen in the face of an unusually friendly part/jump of a tech tree (like AlexNet/the neural networks->GPU transition) but still mostly not, and likely not from an unprecedented in computer science algorithmic change.

https://aiimpacts.org/?s=cross+

(Just want to interject that Carl has higher P(doom) than Paul and has also critiqued Paul for not being more concrete, and I doubt that this is the source of the common disagreements that Paul/Carl both have with Eliezer)

From my perspective the thing the AI impacts investigation is asking is something like “When people are putting lots of resources into improving some technology, how often is it the case that someone can find a cool innovation that improves things a lot relative to the baseline?” I think that your response to that is something like “Sure, if the broad AI market were efficient and everyone were investigating the right lines of research, then AI progress might be smooth, but AGI would have also been developed way sooner. We can’t safely assume that AGI is like an industry where lots of people are pushing toward the same thing”

But it’s not assuming a great structural similarity between bridges and AI, except that they’re both things that humans are trying hard to find ways to improve

Sure, if the broad AI market were efficient and everyone were investigating the right lines of research, then AI progress might be smooth

Presumably also “‘AI progress’ subsumes many different kinds of cognition, we don’t currently have baby AGIs, and when we do figure out how to build AGI the very beginning of the curve (the Wright flyer moment, or something very shortly after) will correspond to a huge capability increase.”

I think Paul’s view would say:

• Things certainly happen for the first time
• When they do, they happen at small scale in shitty prototypes, like the Wright Flyer or GPT-1 or AlphaGo or the Atari bots or whatever
• When they’re making a big impact on the world, it’s after a lot of investment and research, like commercial aircrafts in the decades after Kitty Hawk or like the investments people are in the middle of making now with AI that can assist with coding

Paul’s view says that the Kitty Hawk moment already happened for the kind of AI that will be super transformative and could kill us all, and like the historical Kitty Hawk moment, it was not immediately a huge deal

The fact that you express this kind of view about AGI erupting one day is why I thought your thing in IEM was saying there was a major algorithmic innovation from chimps to humans, that humans were qualitatively and not just quantitatively better than chimps and this was not because of their larger brain size primarily. But I’m confused because up thread in the discussion of evolution you were emphasizing much more that there was an innovation between dinosaurs and primates, not that there was an innovation between chimps and humans, and you seemed more open to the chimp/human diff being quantitative and brain-size driven than I had thought you’d be. But being open to the chimp-human diff being quantitative/brain-size-driven suggests to me that you should be more open than you are to AGI being developed by slow grinding on the same shit, instead of erupting without much precedent?

The claim I’m making is that Paul’s view predicts a lag and a lot of investment between the first flight and aircraft making a big impact on the travel industry, and predicts that the first flight wouldn’t have immediately made a big impact on the travel industry. In other words Kitty Hawk isn’t a discontinuity in the Paul view because the metrics he’d expect to be continuous are the ones that large numbers of people are trying hard to optimize, like cost per mile traveled or whatnot, not metrics that almost nobody is trying to optimize, like “height flown.”

In other words, it sounds like you’re saying:

• Kitty Hawk is analogous to AGI erupting
• Previous history of travel is analogous to pre-AGI history of AI

While Paul is saying:

• Kitty Hawk is analogous to e.g. AlexNet
• Later history of aircraft is analogous to the post-AlexNet story of AI which we’re in the middle of living, and will continue on to make huge Singularity-causing impacts on the world

But Paul offered to bet with you about literally any quantity you choose?

It still sounds to me like “take a basket of N performance metrics, bet that the model size to perf trend will break upward in > K of them within e.g. 2 or 3 years” should sound good to you, I’m confused why that didn’t. If it does and it’s just about the legwork then I think we could get someone to come up with the benchmarks and stuff for you

Or maybe the same thing but >K of them will break downward, whatever

We could bet about the human perception of sense in language models, for example

I don’t think you need to bet about calendar times from par-human to super-human, and any meta-trend in that quantity. It sounds like Paul is saying “I’ll basically trust the model size to perf trends and predict a 10x bigger model from the same architecture family will get the perf the trends predict,” and you’re pushing back against that saying e.g. that humans won’t find GPT-4 to be subjectively more coherent than GPT-3 and that Paul is neglecting that there could be major innovations in the future that bring down the FLOP/s to get a certain perf by a lot and bend the scaling laws. So why not bet that Paul won’t be as accurate as he thinks he is by following the scaling laws?

I think Paul’s view would say:

• Things certainly happen for the first time
• When they do, they happen at small scale in shitty prototypes, like the Wright Flyer or GPT-1 or AlphaGo or the Atari bots or whatever
• When they’re making a big impact on the world, it’s after a lot of investment and research, like commercial aircrafts in the decades after Kitty Hawk or like the investments people are in the middle of making now with AI that can assist with coding

Paul’s view says that the Kitty Hawk moment already happened for the kind of AI that will be super transformative and could kill us all, and like the historical Kitty Hawk moment, it was not immediately a huge deal

“When they do, they happen at small scale in shitty prototypes, like the Wright Flyer or GPT-1 or AlphaGo or the Atari bots or whatever”

How shitty the prototype is should depend (to a very large extent) on the physical properties of the tech. So I don’t find it confusing (though I currently disagree) when someone says “I looked at a bunch of GPT-3 behavior and it’s cognitively sophisticated enough that I think it’s doing basically what humans are doing, just at a smaller scale. The qualitative cognition I can see going on is just that impressive, taking into account the kinds of stuff I think human brains are doing.”

What I find confusing is, like, treating ten thousand examples of non-AI, non-cognitive-tech continuities (nukes, building heights, etc.) as though they’re anything but a tiny update about ‘will AGI be high-impact’ — compared to the size of updates like ‘look at how smart and high-impact humans were’ and perhaps ‘look at how smart-in-the-relevant-ways GPT-3 is’.

Like, impactfulness is not a simple physical property, so there’s not much reason for different kinds of tech to have similar scales of impact (or similar scales of impact n years after the first prototype). Mainly I’m not sure to what extent we disagree about this, vs. this just being me misunderstanding the role of the ‘most things aren’t high-impact’ argument.

(And yeah, a random historical technology drawn from a hat will be pretty low-impact. But that base rate also doesn’t seem to me like it has much evidential relevance anymore when I update about what specific tech we’re discussing.)

The question is not “will AGI be high impact” — Paul agrees it will, and for any FOOM quantity (like crossing a chimp-to-human-sized gap in a day or whatever) he agrees that will happen eventually too.

The technologies studies in the dataset spanned a wide range in their peak impact on society, and they’re not being used to forecast the peak impact of mature AI tech

Yeah, I’m specifically confused about how we know that the AGI Wright Flyer and its first successors are low-impact, from looking at how low-impact other technologies are (if that is in fact a meaningful-sized update on your view)

Not drawing a comparison about the overall impactfulness of AI / AGI (e.g., over fifteen years)

I think we should try to find ML predictions where you defer to superforecasters and Paul disagrees, since he said he would bet against superforecasters in ML

I think what Paul was saying last night is you find superforecasters betting on some benchmark performance, and he just figures out which side he’d take (and he expects in most/all superforecaster predictions that he would not be deferential, there’s a side he would take)

not really following along with the conversation, but my desire to bet about “whatever you want” was driven in significant part by frustration with Eliezer repeatedly saying things like “people like Paul get surprised by reality” and me thinking that’s nonsense

I still think that if you want to say “this sort of reasoning is garbage empirically” then you ought to be willing to bet about something. If we are just saying “we agree about all of the empirics, it’s just that somehow we have different predictions about AGI” then that’s fine and symmetrical.

even then presumably if you think it’s garbage you should be able to point to some particular future predictions where it would be garbage?

if you used it

and then I can either say “no, I don’t think that’s a valid application for reason X” or “sure, I’m happy to bet”

and it’s possible you can’t find any places where it sticks its neck out in practice (even in your version), but then I’m again just rejecting the claim that it’s empirically ruled out

I think this might be derailing some broader point, but I am provisionally mostly ignoring your point “this doesn’t work in practice” if we can’t find places where we actually foresee disagreements

(which is fine, I don’t think it’s core to your argument)

I don’t know if superforecasters make public bets on ML topics, I was saying I’m happy to bet on ML topics and if your strategy is “look up what superforecasters say” that’s fine and doesn’t change my willingness to bet

I think this is probably not as promising as either (i) dig in on the arguments that are most in dispute (seemed to be some juicier stuff earlier though I’m just focusing on work today) , or (ii) just talking generally about what we expect to see in the next 5 years so that we can at least get more of a vibe looking back

You can bet on the Metaculus AI Tournament forecasts.

https://www.metaculus.com/ai-progress-tournament/

1/3 that RH has an automated proof before sustained 7%/year GWP growth?

I think the clearest indicator is that we have AI that ought to be able to e.g. run the fully automated factory-building factory (not automating mines or fabs, just the robotic manufacturing and construction), but it’s not being deployed or is being deployed with very mild economic impacts

another indicator is that we have AI systems that can fully replace human programmers (or other giant wins), but total investment in improving them is still small

another indicator is a DeepMind demo that actually creates a lot of value (e.g. 10x larger than DeepMind’s R&D costs? or even comparable to DeepMind’s cumulative R&D costs if you do the accounting really carefully and I definitely believe it and it wasn’t replaceable by Brain), it seems like on your model things should “break upwards” and in mine that just doesn’t happen that much

sounds like you may have >90% on automated proof of RH before a few years of 7%/year growth driven by AI? so that would give a pretty significant odds ratio either way

I think “stack more layers gets stuck but a clever idea makes crazy stuff happen” is generally going to be evidence for your view

That said, I’d mostly reject AlphaGo as an example, because it’s just plugging in neural networks to existing go algorithms in almost the most straightforward way and the bells and whistles don’t really matter. But if AlphaZero worked and AlphaGo didn’t, and the system accomplished something impressive/important (like proving RH, or being significantly better at self-contained programming tasks), then that would be a surprise.

And I’d reject LSTM -> transformer or MoE as an example because the quantitative effect size isn’t that big.

But if something like that made the difference between “this algorithm wasn’t scaling before, and now it’s scaling,” then I’d be surprised.

And the size of jump that surprises me is shrinking over time. So in a few years even getting the equivalent of a factor of 4 jump from some clever innovation would be very surprising to me.

I think RH is not that surprising, it’s not at all clear to me where “do formal math” sits on the “useful stuff AI could do” spectrum, I guess naively I’d put it somewhere “in the middle” (though the analogy to board games makes it seem a bit lower, and there is a kind of obvious approach to doing this that seems to be working reasonably well so that also makes it seem lower), and 7% GDP growth is relatively close to the end (ETA: by “close to the end” I don’t mean super close to the end, just far enough along that there’s plenty of time for RH first)

I do think that performance jumps are maybe more dispositive, but I’m afraid that it’s basically going to go like this: there won’t be metrics that people are tracking that jump up, but you’ll point to new applications that people hadn’t considered before, and I’ll say “but those new applications aren’t that valuable” whereas to you they will look more analogous to a world-ending AGI coming out from the blue

like for AGZ I’ll be like “well it’s not really above the deep learning trend if you run it backwards” and you’ll be like “but no one was measuring it before! you can’t make up the trend in retrospect!” and I’ll be like “OK, but the reason no one was measuring it before was that it was worse than traditional go algorithms until like 2 years ago and the upside is not large enough that you should expect a huge development effort for a small edge”

like I’m surprised if a clever innovation does more good than spending 4x more compute

I’m saying that it gets harder over time, don’t expect wins as big as transformers

I think even transformers probably wouldn’t make this cut though?

certainly not vs CNNs

vs RNNs I think the comparison I’d be using to operationalize it is translation, as measured in the original paper

they do make this cut for translation, looks like the number is like 100 >> 4

100x for english-german, more like 10x for english-french, those are the two benchmarks they cite

but both more than 4x

I’m saying I don’t expect ongoing wins that big

I think the key ambiguity is probably going to be about what makes a measurement established/hard-to-improve

I think the relevant feature of the innovation is that the work to find it is small relative to the work that went into the problem to date (though there may be other work on other avenues)

I’m calling that an innovation because it’s a small amount of work

like, it’s not just a claim about EMH, it’s also a claim about the nature of progress

I think AlphaFold counts and is probably if anything a bigger multiplier, it’s just uncertainty over how many people actually worked on the baselines

I mean, I think they are steadily subsiding

as areas grow

I would be surprised to see a 10x today on machine translation

though not so surprised that we can avoid talking about probabilities

yeah

or to make it more surprising, old sota with 10x less compute

I’m more surprised if they get the old performance with 10x less compute though, so that way around is better on all fronts

like, this view also leads me to predict that if I look at the actual amount of manpower that went into alphafold, it’s going to be pretty big relative to the other people submitting to that protein folding benchmark

not plausible claims on problems people care about

I think the comparison is to contemporary benchmarks from one of the 99 other startups who didn’t find the bright idea

that’s the relevant thing on your view, right?

and then I’m predicting that number is already <10 for machine translation and falling (maybe I shouldn’t talk about machine translation or at least not commit to numbers given that I know very little about it, but whatever that’s my estimate), and for other domains it will be <10 by the time they get as crowded as machine translation, and for transformative tasks they will be <2

isn’t there an open-source replication of alphafold?

we could bet about its performance relative to the original

I agree

anyway, my guess is generally that if you are big relative to previous efforts in the area you can make giant improvements, if you are small relative to previous efforts you might get lucky (or just be much smarter) but that gets increasingly unlikely as the field gets bigger

like alexnet and transformers are big wins by groups who are small relative to the rest of the field, but transformers are much smaller than alexnet and future developments will continue to shrink

I mean, if you are the same size as all the prior effort put together?

I’m not surprised if you can totally dominate in that case, especially if prior efforts aren’t well-coordinated

and for things that are done by hobbyists, I wouldn’t be surprised if you can be a bit bigger than an individual hobbyist and dominate

yeah, I think if DeepMind beats Google Brain by 10x compute next year on translation, that’s a significant strike against Paul

I think not that seriously, but more seriously than 2016 and than anything else where you are seeing big swings

and so I’m less surprised than for TAI, but still surprised

I definitely think there’s a big disagreement here about what to expect for pre-end-of-days ML

but lots of concerns about details like what domains are crowded enough to be surprising and how to do comparisons

I mean, to be clear, I think the transformer paper having giant gains is also evidence against paulverse

it’s just that there are really a lot of datapoints, and some of them definitely go against paul’s view

to me it feels like the relevant thing for making the end-of-days forecast is something like “how much of the progress comes from ‘innovations’ that are relatively unpredictable and/or driven by groups that are relatively small, vs scaleup and ‘business as usual’ progress in small pieces?”

I mostly want to know how good Google’s translation is at that time; and if DeepMind’s product is expensive or only shows gains for long texts, I want to know whether there is actually an economic niche for it that is large relative to the R&D cost.

like I’m not sure whether anyone works at all on long-text translation, and I’m not sure if it would actually make Google $ to work on it

great text chat with contractor in indonesia almost certainly meets that bar though

I’m OK if they don’t have the benchmark graph as long as they have some evaluation that other people were trying at, I think real-time chat probably qualifies

also I’m saying 10x less training compute, not inference (but 10x less inference compute is harder)

yes

in practice it seems almost certain that it’s going to be harder to evaluate

though I agree there are really clean versions where they actually measured a benchmark other people work on and can compare training compute directly

(like in the transformer paper)

I’m more surprised by the clean version

I’d want to look at either human evals or logprob, I think probably not? but it’s possible it was

it’s definitely way harder to win at the old task with 10x less compute

I think it’s usually the case that if you scale up far enough past previous sota, you will be able to find tons of techniques needed to make it work at the new scale

but I’m expecting it to be less of a big deal because all experiments will be roughly at the frontier of what is feasible

and so the new thing won’t be able to afford to go 10x bigger

unlike today when we are scaling up spending so fast

but this does make it harder for the next few years at least, which is maybe the key period

(it makes it hard if we are both close enough to the edge that “10x cheaper to get old results” seems unlikely but “getting new results that couldn’t be achieved with 10x more compute and old method” seems likely)

what I basically expect is to (i) roughly know how much performance you get from making models 10x bigger, (ii) roughly know how much someone beat the competition, and then you can compare the numbers

(but “performance from making models 10x bigger” depends a lot on exactly how big they were and whether you are in a regime with unfavorable scaling)

I am also expecting a general scale up in ML training runs over time, though it’s plausible that you also expect that until the end of days and just expect a much earlier end of days

that’s going to kill the “they spent 100x more compute” announcements soon enough

like, that’s easy when “100x more” means $1M, it’s a bit hard when “100x more” means $100M, it’s not going to happen except on the most important tasks when “100x more” means $10B

I’m not surprised by announcements like protein folding, it’s not that the world overall gets more and more hostile to big wins, it’s that any industry gets more and more hostile as it gets bigger (or across industries, they get more and more hostile as the stakes grow)

(I mean, the protein folding thing is a datapoint against my view, but it’s not that much evidence and it’s not getting bigger over time)

yeah, but doesn’t your view expect more innovations for any given problem?

like, it’s not just that you think the universe of weird profitable applications is larger, you also think AI progress is just more driven by innovations, right?

otherwise it feels like the whole game is about whether you think that AI-automating-AI-progress is a weird application or something that people will try on

yeah, but then don’t you see big wins from the next transformers?

and you think those just keep happening even as fields mature

or you mean that they might slow down because people stop working on them?

it seems clear that it will be more than 10x relative to GPT

I guess I don’t know what GPT-like architecture means, but from what you say it seems like normal progress would result in a non-GPT-like architecture

so I don’t think I’m disagreeing with that

do you think transformers are a big insight?

(is adding soft attention to LSTMs a big insight?)

(I think the intellectual history of transformers is a lot like “take the LSTM out of the LSTM with attention”)

that really really seems like the accumulation of small insights

like, residual connections are the single biggest thing

and relus also help

and batch normalization helps

and attention is better than lstms

you could also call that the accumulation of big insights, but the point is that it’s an accumulation of a lot of stuff

mostly developed in different places

(I think transformers are a significantly smaller jump than previous improvements)

also a thing we could guess about though

also 10-100x is still actually surprising to me for transformers

so I guess lesson learned

in general if you take anything developed at scale X and try to scale it way past X I think it won’t work

or like, it will work much worse than something that continues to get tweaked

yeah, I mean that given everything I know I am surprised that transformers were as large as a 100x improvement on translation

in that paper

yeah, I am definitely interested to understand a bit better what’s up there

but tentatively I’m sticking to my guns on the original prediction

if you have random 10-20 person teams getting 100x speedups versus prior sota

as we approach TAI

that’s so far from paulverse

anyway, seems like a good approach to finding a concrete disagreement

and even looking back at this conversation would be a start for diagnosing who is more right in hindsight

main thing is to say how quickly and in what industries I’m how surprised

I think the economically important prediction doesn’t really need that much of “within a few years”

like the total stakes have just been low to date

none of the deep learning labs are that close to paying for themselves

so we’re not in the regime where “economic niche > R&D budget”

we are still in the paulverse-consistent regime where investment is driven by the hope of future wins

though paul is surprised that R&D budgets aren’t more larger than the economic value

I’ll register considerable skepticism

it seems easier to just talk about ML tasks that people work on

it seems really hard to arbitrate the “all the important niches are invested in” stuff in a way that’s correlated with takeoff

whereas the “we should be making a big chunk of our progress from insights” seems like it’s easier

though I understand that your view could be disjunctive, of either “AI will have hidden secrets that yield great intelligence,” or “there are hidden secret applications that yield incredible profit”

(sorry that statement is crude / not very faithful)

should follow up on this in the future, off for now though

I’ll interject some points re the earlier discussion about how animal data relates to the ‘AI scaling to AGI’ thesis.

1. In humans it’s claimed the IQ-job success correlation varies by job, For a scientist or doctor it might be 0.6+, for a low complexity job more like 0.4, or more like 0.2 for simple repetitive manual labor. That presumably goes down a lot with less in the way of hands, or focused on low density foods like baleen whales or grazers. If it’s 0.1 for animals like orcas or elephants, or 0.05, then there’s 4-10x less fitness return to smarts.

2. But they outmass humans by more than 4-10x. Elephants 40x, orca 60x+. Metabolically (20 watts divided by BMR of the animal) the gap is somewhat smaller though, because of metabolic scaling laws (energy scales with 3/4 or maybe 2/3 power, so ).

https://en.wikipedia.org/wiki/Kleiber%27s_law

If dinosaurs were poikilotherms, that’s a 10x difference in energy budget vs a mammal of the same size, although there is debate about their metabolism.

3. If we’re looking for an innovation in birds and primates, there’s some evidence of ‘hardware’ innovation rather than ‘software.’ Herculano-Houzel reports in The Human Advantage (summarizing much prior work neuron counting) different observational scaling laws for neuron number with brain mass for different animal lineages.

We were particularly interested in cellular scaling differences that might have arisen in primates. If the same rules relating numbers of neurons to brain size in rodents (6)

The brain of the capuchin monkey, for instance, weighing 52 g, contains >3× more neurons in the cerebral cortex and ≈2× more neurons in the cerebellum than the larger brain of the capybara, weighing 76 g.

[Editor’s Note: Quote source is “Cellular scaling rules for primate brains.”]

In rodents brain mass increases with neuron count n^1.6, whereas it’s close to linear (n^1.1) in primates. For cortex neurons and cortex mass 1.7 and 1.0. In general birds and primates are outliers in neuron scaling with brain mass.

Note also that bigger brains with lower neuron density have longer communication times from one side of the brain to the other. So primates and birds can have faster clock speeds for integrated thought than a large elephant or whale with similar neuron count.

4. Elephants have brain mass ~2.5x human, and 3x neurons, but 98% of those are in the cerebellum (vs 80% in or less in most animals; these are generally the tiniest neurons and seem to do a bunch of fine motor control). Human cerebral cortex has 3x the neurons of the elephant cortex (which has twice the mass). The giant cerebellum seems like controlling the very complex trunk.

https://nautil.us/issue/35/boundaries/the-paradox-of-the-elephant-brain

Blue whales get close to human neuron counts with much larger brains.

https://en.wikipedia.org/wiki/List_of_animals_by_number_of_neurons

5. As Paul mentioned, human brain volume correlation with measures of cognitive function after correcting for measurement error on the cognitive side is in the vicinity of 0.3-0.4 (might go a bit higher after controlling for non-functional brain volume variation, lower from removing confounds). The genetic correlation with cognitive function in this study is 0.24:

https://www.nature.com/articles/s41467-020-19378-5

So it accounts for a minority of genetic influences on cognitive ability. We’d also expect a bunch of genetic variance that’s basically disruptive mutations in mutation-selection balance (e.g. schizophrenia seems to be a result of that, with schizophrenia alleles under negative selection, but a big mutational target, with the standing burden set by the level of fitness penalty for it; in niches with less return to cognition the mutational surface will be cleaned up less frequently and have more standing junk).

Other sources of genetic variance might include allocation of attention/learning (curiosity and thinking about abstractions vs immediate sensory processing/alertness), length of childhood/learning phase, motivation to engage in chains of thought, etc.

Overall I think there’s some question about how to account for the full genetic variance, but mapping it onto the ML experience with model size, experience and reward functions being key looks compatible with the biological evidence. I lean towards it, although it’s not cleanly and conclusively shown.

Regarding economic impact of AGI, I do not buy the ‘regulation strangles all big GDP boosts’ story.

The BEA breaks down US GDP by industry here (page 11):

https://www.bea.gov/sites/default/files/2021-06/gdp1q21_3rd_1.pdf

As I work through sectors and the rollout of past automation I see opportunities for large-scale rollout that is not heavily blocked by regulation. Manufacturing is still trillions of dollars, and robotic factories are permitted and produced under current law, with the limits being more about which tasks the robots work for at low enough cost (e.g. this stopped Tesla plans for more completely robotic factories). Also worth noting manufacturing is mobile and new factories are sited in friendly jurisdictions.

Software to control agricultural machinery and food processing is also permitted.

Warehouses are also low-regulation environments with logistics worth hundreds of billions of dollars. See Amazon’s robot-heavy warehouses limited by robotics software.

Driving is hundreds of billions of dollars, and Tesla has been permitted to use Autopilot, and there has been a lot of regulator enthusiasm for permitting self-driving cars with humanlike accident rates. Waymo still hasn’t reached that it seems and is lowering costs.

Restaurants/grocery stores/hotels are around a trillion dollars. Replacing humans in vision/voice tasks to take orders, track inventory (Amazon Go style), etc is worth hundreds of billions there and mostly permitted. Robotics cheap enough to replace low-wage labor there would also be valuable (although a lower priority than high-wage work if compute and development costs are similar).

Software is close to a half trillion dollars and the internals of software development are almost wholly unregulated.

Finance is over a trillion dollars, with room for AI in sales and management.

Sales and marketing are big and fairly unregulated.

In highly regulated and licensed professions like healthcare and legal services, you can still see a licensee mechanically administer the advice of the machine, amplifying their reach and productivity.

Even in housing/construction there’s still great profits to be made by improving the efficiency of what construction is allowed (a sector worth hundreds of billions).

If you’re talking about legions of super charismatic AI chatbots, they could be doing sales, coaching human manual labor to effectively upskill it, and providing the variety of activities discussed above. They’re enough to more than double GDP, even with strong Baumol effects/cost disease, I’d say.

Although of course if you have AIs that can do so much the wages of AI and hardware researchers will be super high, and so a lot of that will go into the intelligence explosion, while before that various weaknesses that prevent full automation of AI research will also mess up activity in these other sectors to varying degrees.

Re discontinuity and progress curves, I think Paul is right. AI Impacts went to a lot of effort assembling datasets looking for big jumps on progress plots, and indeed nukes are an extremely high percentile for discontinuity, and were developed by the biggest spending power (yes other powers could have bet more on nukes, but didn’t, and that was related to the US having more to spend and putting more in many bets), with the big gains in military power per $ coming with the hydrogen bomb and over the next decade.

https://aiimpacts.org/category/takeoff-speed/continuity-of-progress/discontinuous-progress-investigation/

For measurable hardware and software progress (Elo in games, loss on defined benchmarks), you have quite continuous hardware progress, and software progress that is on the same ballpark, and not drastically jumpy (like 10 year gains in 1), moreso as you get to metrics used by bigger markets/industries.

I also agree with Paul’s description of the prior Go trend, and how DeepMind increased $ spent on Go software enormously. That analysis was a big part of why I bet on AlphaGo winning against Lee Sedol at the time (the rest being extrapolation from the Fan Hui version and models of DeepMind’s process for deciding when to try a match).

Little of Open Phil’s opinions are independent of Carl, the source of all opinions