Looking at concrete scenarios can serve as a corrective to a number of false assumptions about ASI endgames, such as “Governments would have time to respond effectively to ASI post-deployment”. Some high-level features that seem relatively likely to occur in ASI extinction scenarios include:
- Covert action. Until an ASI is confident that it can succeed in a first-strike attack, it will not want to give its developers or the larger world any cause for concern. If it does engage in visible activity, it will likely be to sow chaos or otherwise distract key actors.
- Self-improvement and hardware acquisition. Improved software, more optimized hardware, and larger quantities of hardware would all allow ASI to think faster and make higher-quality decisions. “Self-improvement” can also shade into building entirely new AIs that optimize the same objective.
- Manipulation. In the event that ASI needs human assistance to bootstrap to free operation in the physical world, or to bootstrap from weaker real-world machinery to stronger machinery, we can expect ASI to use social manipulation to get humans to initially help. This “manipulation” can be as simple as paying someone on the internet to do a task, without revealing the significance of the task or revealing the AI’s involvement.
- Scientific research and experimentation. This can include reading all scientific papers on the internet, drawing new conclusions from and connections between scientific literatures, running experiments in simulation, and running experiments in the physical world.
- Construction of rapidly self-replicating microorganisms or small-scale machinery. Operating on a small scale allows faster (and often less detectable) real-world action. Exponential self-replication allows ASI to bootstrap from a single modified microorganism or nano-scale machine into large-scale infrastructure development (e.g., for self-improvement or resource extraction) or into a decisively powerful weapon (e.g., a fully lethal virus).
- Unforeseen events. ASI explores an enormously multidimensional space far better than any human can, thinks very unlike any human, and can chain scientific advances into further advances that are ever-more-remote from the things we understand today. Moreover, ASI is likely to deliberately favor unforeseen strategies because these are harder to defend against.
All of this makes it a relatively doomed venture to try to predict the details of an ASI takeover event. We can hope to give proofs of concept that certain scenario classes are possible, and set rough lower bounds on what an ASI might do, but this is different from predicting the true chain of events.
We expect any period where ASI needs humans to keep things running to be brief. Given the economic incentives, people are likely to increasingly delegate control of infrastructure to AI (and systems hackable by AI) over the coming years; and we expect ASI to be able to manipulate and/or pay humans to help it develop independent (and faster-running) infrastructure.
This process can be extremely quick, as humans are far from the limits of how quickly engineering, military action, and infrastructure development can occur.
Consider what sorts of machinery are permitted by physics, as demonstrated by structures in nature. Algae can be modeled as solar-powered self-replicating factories that can double in population size in less than a day. Trees assemble bulk construction materials largely out of thin air (carbon capture). And biological organisms are nowhere near the theoretical limits of energy efficiency and material strength.
ASI would at least be able to achieve science and engineering feats commensurate with what already exists in the biological world. E.g., ASI would likely be able to covertly develop and propagate deadly pathogens across the planet, and design organisms to replace the work of humans.
The biological systems we can see in the natural world don’t set an upper limit on the capabilities of ASI systems. More likely, ASI would be able to construct de novo micro- and macro-machines to achieve its ends (to the extent we haven’t handed it such machines ourselves), rather than being limited to biological agents and materials. But this lower capability level is already sufficient for ASI to rapidly displace humans.
