Superintelligence

1. Past developments and present capabilities

Overview

Artificial intelligence now powers countless everyday tools. But the abilities that define human-level understanding are still out of reach. The puzzle is how a field can do so much in narrow areas and still lack the general capacity that would tie those successes together.

Common sense and natural language understanding remain hard. Many researchers call them AI-complete: solve them, and you've built generally human-level intelligence. Chess fell to a surprisingly simple algorithm. That suggests other abilities, like general reasoning or programming, might also have simple solutions. Ptolemaic astronomy kept adding epicycles for over a thousand years. Then Copernicus proposed a simpler sun-centered model, and Kepler refined it into something more accurate. AI now spans many areas, with over 10 million robots worldwide. It includes speech recognition, machine translation from bilingual text, face recognition, theorem provers, and military drones.

DART logistics planning in Operation Desert Storm in 1991, according to DARPA, repaid its thirty-year AI investment. McCarthy's dictum still holds: when something works, it stops being called AI. Today's systems are narrow, but their parts may add up to general intelligence. Those parts include classifiers, search algorithms, planners, solvers, and representational frameworks.

Algorithmic high-frequency traders handle more than half of US equity shares. The 2010 Flash Crash, when a trillion dollars briefly vanished and trades 60% or more from pre-crisis levels were canceled as clearly erroneous, shows three lessons. Simple components can create system-wide risk. Sensible assumptions can turn catastrophic when followed with strict logic. And automatic stop logic resolved the crash because humans could not respond fast enough.

Surveys define human-level machine intelligence, or HLMI, as carrying out most human professions at least as well as a typical human. The combined median estimates put the chance at 10% by 2022, 50% by 2040, and 90% by 2075. Nilsson gave 10% by 2030, 50% by 2050, and 90% by 2100. Later dates deserve more probability. For superintelligence after HLMI, combined estimates put the chance at 10% within two years and 75% within thirty. It makes sense to hold a mixed view. Surveys suggest a sizeable chance of HLMI by mid-century, possibly followed fairly soon by superintelligence, with outcomes ranging from extremely good to human extinction.

An ability that looks like it needs deep intelligence can turn out to have a surprisingly simple mechanism, as chess and planetary astronomy both did. So no cognitive domain should be treated as permanently beyond a compact solution. Because successful techniques stop being labeled artificial intelligence and become ordinary engineering, the field's visible frontier always looks emptier than what it has actually built. The narrow tools already in use, including classifiers, planners, solvers, and search methods, are the raw material for a general intelligence. Their individual narrowness does not cap what a larger assembly of them might achieve. The Flash Crash shows that individually reasonable automated components can interact into system-wide collapse at speeds beyond human reaction. That danger comes with handing fast control loops to machines. Survey medians place a genuine chance of machines matching typical human performance across most professions by mid-century, with superintelligence potentially arriving not long after. Nilsson's figures sit in the same range, and the wide disagreement among forecasters is itself reason to hold a mixed view of both the timing and the consequences.

2. Paths to superintelligence

Overview

Machines remain far less generally capable than people, yet something vastly more capable is supposed to arrive. Which routes could possibly lead there, and how much credence any of them deserves, remains unresolved.

Superintelligence defined

A superintelligence is an intellect whose cognitive performance far outstrips human beings across nearly every domain of interest. It is noncommittal about implementation and about subjective experience. Chess programs such as Deep Fritz do not qualify. Domain-limited cases are flagged explicitly, as with an "engineering superintelligence."

Artificial intelligence

No blueprint for artificial general intelligence exists, and none would be published here. Core design features are discernible:

• a capacity to learn, integral from the outset rather than added later
• effective handling of uncertainty and probabilistic information
• extracting useful concepts from sensory data and internal states
• combining concepts into flexible representations for logical and intuitive reasoning

Early Good Old-Fashioned AI neglected these. Turing's 1950 "child machine" would be educated and iterated on, progressing faster than evolution by targeting identified weaknesses rather than random mutation. Chalmers and Moravec argue guided engineering should achieve the same, though engineering still trails nature in morphogenesis, self-repair and immune defense.

Evolutionary estimates and their limits

The book's rough estimate of recapitulating evolution is 10^31 to 10^44 FLOPS. Tianhe-2, the fastest supercomputer in September 2013, reached 3.39x10^16 FLOPS. Aiming the search at intelligence might save five to twenty-five orders of magnitude, but an observation selection effect could shift the estimate by thirty.

Seed AI and whole brain emulation

A seed AI could improve its own architecture and then design progressively smarter versions of itself. That recursive self-improvement could produce an intelligence explosion, considered in Chapter 4. Machine minds need not resemble human ones, and their motivations may diverge radically. Whole brain emulation, or uploading, instead models a scanned biological brain's computational structure, beginning with a detailed scan.

The emulation path

Whole brain emulation has three stages:

• scan a brain
• process images into a three-dimensional neuronal map
• simulate it

The prerequisites:

• scanning
• translation
• simulation

No conceptual breakthrough is needed, but the enabling technologies require vast incremental progress. C. elegans, a 1 mm roundworm with 302 neurons, has had its wiring known since the mid-1980s, but strengths and dynamics remain unmapped. A roadmap places prerequisites near mid-century, with a ladder from C. elegans to honeybee, mouse, rhesus monkey and human. If modeling is the final bottleneck, a broken system might suddenly become coherent, making advance warning less certain in that scenario.

Biological cognition

Education, nutrition, sleep, exercise and disease prevention yield modest gains. Nootropics are dubious; flooding a healthy brain with a chemical is unlikely to sharply raise intelligence. Embryo selection is stronger, using pre-implantation diagnosis and cheap genotyping. Table 5 projects IQ gains of 4.2 points from 1 in 2 embryos, 11.5 from 1 in 10, 18.8 from 1 in 100 and 24.3 from 1 in 1,000, with diminishing returns.

Iterated embryo selection

Stem-cell-derived gametes could generate unlimited gametes. The procedure:

• Genotype and select embryos with desired traits.
• Extract stem cells and convert them to sperm and ova.
• Cross them to produce embryos.
• Repeat to accumulate changes.

This could compress ten generations into a few years, potentially producing individuals at or above the historical maximum, a collective superintelligence if culture and infrastructure match. Adoption will lag because of a twenty-year maturation period, moral or religious bans and preference for natural conception, though elite use and health costs may create a bandwagon.

Selection power and competition

States might encourage genetic selection to build human capital, or to select for docility and conformity outside a ruling clan. Nations with long-term population policies, among them China and Singapore, might promote it. Table 6 pairs four technology tiers with adoption levels:

• "IVF+": one of two embryos
• "Aggressive IVF": one of ten
• "In vitro egg selection": one of a hundred
• "Iterated embryo selection": over a hundred points

Designed and proofread genomes

Genomes synthesized to specification would let rare cognition-boosting alleles absent from both parents be spliced in, and enable genetic spell-checking: each person carries hundreds of slightly deleterious mutations, negligible individually but costly together, and proofread genomes might yield people closer to "Platonic ideals."

Timing, regulation and conclusions

Germline enhancement is unlikely to reshape society before mid-century, since the enhanced generation needs twenty to twenty-five years to mature. Regulation driven by fear of success varies across cultures, and once one country shows results, holdouts face strong incentives to follow. Three conclusions:

• Weak superintelligence is achievable biotechnologically.
• Cognitively enhanced humans make advanced machine intelligence more plausible.
• Genetically enhanced populations will enter public life in the second half of this century.

Brain-computer interfaces

Implants carry infection, hemorrhage and cognitive decline risks, and deep brain stimulation relieves Parkinson's symptoms while, according to the book, treated patients score worse than controls on verbal fluency, attention and memory. Enhancement is harder than therapy: extra bandwidth would not speed thought unless nearly the whole brain were upgraded, meaning a whole brain prosthesis, or artificial general intelligence. Thought transfer between brains fails because each brain's representations are idiosyncratic, and the required interface would be an AI-complete problem.

The hippocampal prosthesis

A prosthesis bridging two rat hippocampal areas restored a blocked connection and, by sending a clearer token of a memory pattern, improved memory performance beyond normal. Scaling to more memories, controlling combinatorial explosion, hidden costs such as weaker generalization, and the tangled feedback loops of cortex remain unresolved. An implant need not be intelligent if the brain learns the mapping, but projecting it onto the retina or cochlea would be simpler.

Networks and organizations

Networking individual minds with artifacts and bots might yield collective rather than individual superintelligence. Collective intelligence has grown through language, writing, printing, denser populations, better organization, epistemic norms and institutional capital, and is capped by members' abilities, communication overheads and organizational distortions. Lower overheads and fixes for status games, mission creep, concealment and agency problems would allow larger, denser organizations. Prediction markets, lie detectors and self-deception detectors could curb deception and improve forecasting.

Could the internet wake up?

Vernor Vinge's 1993 essay, which coined "technological singularity", raised the internet becoming a virtual skull. Spontaneous emergence is implausible; more likely, incremental gains in search, filtering, software agents and protocols would leave a system one ingredient short of igniting, converging on artificial general intelligence.

Weighing the paths

Many paths raise confidence that superintelligence will be reached; a blocked route still leaves others, and multiple paths do not imply multiple destinations. Biological or organizational enhancement would speed the science behind machine superintelligence, and the route taken may shape how much control humans retain. Whole brain emulation needs incremental advances, but AI will likely cross first, since partial emulations could support neuromorphic AIs, while brain-computer interfaces look unlikely to yield it.

Key Takeaways

• Superintelligence means general cognitive superiority, so narrow experts do not qualify; the machine routes that matter require learning, probabilistic reasoning, and flexible concepts from the start.
• Evolutionary recapitulation estimates span many orders of magnitude, so no single timeline is reliable, and seed AI that recursively improves its own architecture is a distinct route whose motivations need not resemble human ones.
• Biological enhancement, from iterated embryo selection to proofread genomes, can produce a limited superintelligence, but its multi-decade maturation lag means bans will erode once one state demonstrates a cognitive edge.
• Brain-computer interfaces are unlikely to yield superintelligence because the interface itself is AI-complete, and whole brain emulation will probably be overtaken by machine AI, which partial emulations could accelerate.
• Networked collective intelligence is a distinct path capped by coordination costs, the internet will not spontaneously wake up, and multiple paths raise confidence that superintelligence will be reached without implying a single destination or preserving human control.

3. Forms of superintelligence

Overview

What does superintelligence actually mean, and does the answer change what we should expect from it? Any definition broad enough to include calculators and chess engines still leaves one question open: are the deepest abilities within human reach, or beyond it?

Superintelligence means intellects whose abilities go far beyond the strongest human minds across many kinds of thinking. That covers systems with very different strengths, so it helps to separate three forms.

Three forms

• Speed superintelligence: a mind that does everything a human can do, but much, much faster.
• Collective superintelligence: many smaller minds working together, and together they outperform anything we have now by a wide margin.
• Quality superintelligence: a mind at least as fast as a human and much smarter in kind.

Quality is easiest to picture by comparison: it would stand to ordinary human intelligence roughly as human intelligence stands to elephants or chimpanzees.

Any of the three could eventually build the others, so in the long run they all reach the same place, and each could get there faster than we could from where we are now. Their direct strengths are harder to rank. Speed helps with long chains of steps. Collective systems help with work that can be split into parallel pieces. Quality is roughly the most capable overall.

Sources of advantage

Machine minds also inherit hardware advantages over biology. Neurons fire at roughly 200 Hz, while processors run at gigahertz. Signals in axons travel far below light speed. Neuron count, brain size, and working memory are all tightly limited, while digital hardware can scale to much higher levels.

A machine brain could hold much more information than a biological one and reach it much faster. One estimate puts an adult human brain at about a billion bits. A low-end smartphone holds about a hundred times that.

Further hardware advantages

Biological neurons are noisier than transistors, and noisy computing forces redundant encoding, so cleaner, more precise components could make computing more efficient. Sensors could be added by the million. Brains tire within hours and decay after decades of subjective time, unlike microprocessors. Much of the brain's architecture is fixed from birth or changes only slowly, while hardware can be retuned for each task. Today's best supercomputers already sit within plausible estimates of the brain's processing power, but hardware improves quickly and its ceiling is much higher than biology's.

Software advantages

• Editability: parameters are easier to change in software than in neural tissue, so a whole brain emulation could test extra neurons in a cortical area.
• Duplicability: software copies at high fidelity to fill the hardware base, while biological brains reproduce slowly and start helpless.
• Goal coordination: large human groups struggle to stay united, but identical programs sharing a goal avoid that problem.
• Memory sharing: digital minds swap files instead of training for years, and a billion copies could sync databases every hour.

Key Takeaways

• Speed, collective, and quality superintelligence are different profiles, not rungs on a single ladder. Which one has the advantage depends on the task: long chains of steps, parallel work, or raw insight.
• Any of the three could eventually build the other two, so their long-run abilities end up in the same place even though their immediate strengths differ.
• Digital minds have a built-in edge over biological brains: faster signalling, lower noise, more sensors, hardware that can be retuned, and no fatigue or decay.
• Software adds advantages tissue cannot match: parameters can be edited directly, copies can be made at high fidelity, goals can stay aligned across many instances, and memory can be shared instead of relearned.

4. The kinetics of an intelligence explosion

Overview

How fast does the shift from human-level machine intelligence to radical superintelligence actually happen? If it takes decades, that leaves time to adapt. If it takes days or minutes, almost no one can react in time.

Two things decide the speed. First, how much design effort goes into making the system smarter. Second, how hard it is to keep improving the system. When improvement is easy, intelligence can rise quickly. When it gets harder, progress slows down. Easy gains can run out. But sometimes each improvement makes the next one easier, like finishing the last pieces of a jigsaw puzzle.

How soon a takeoff happens is a separate question from how steep it is. Picture a graph of capability over time, starting from a human baseline in 2014. There are three broad possibilities. A slow takeoff takes decades or centuries, giving governments and institutions time to adapt, run tests, and negotiate treaties. A fast takeoff takes minutes, hours, or days, leaving almost no time to think, so the outcome depends on whatever preparations were already in place. A moderate takeoff takes months or years, giving some room to respond but little time to analyze, test, or coordinate.

Moderate takeoffs could be secretive and chaotic, letting small groups hold outsized influence. Nothing like a fast or moderate takeoff has ever happened before, outside of myth and religion. The Agricultural and Industrial Revolutions played out over centuries and decades. But a slow takeoff is unlikely. If a takeoff happens at all, it will probably be explosive.

Outside of machines, progress tends to hit hard limits. Public health, diet, and education all show steep diminishing returns. With whole brain emulation, copying an insect brain would teach us a lot about scaling up to a human brain, and a mouse brain would teach us even more. Artificial intelligence may not have clear milestones like that. It can feel like wandering through dense jungle with no sign of the finish line until a breakthrough suddenly reveals it.

Building the first human emulation is hard. It takes advanced scanning and image interpretation. But improving an existing emulation is a software problem, and that could be much easier. Progress might stall, then speed up again as inefficiencies get smoothed out and regulations appear. For AI, how hard improvement is depends on the architecture, and it can be extremely easy. A single long-sought insight could let a system jump from below human level to far above it.

An AI might combine a narrow problem-solver with a general reasoning system. While the general system stays below a certain threshold, its answers always lose out, so overall performance doesn't move. Once it crosses that threshold, performance improves as fast as the subsystem itself, and the difficulty of further improvement drops away.

We tend to see these jumps as sharper than they really are, because we judge minds by human standards. Eliezer Yudkowsky made this point with a scale running from mouse to village idiot to chimp to Einstein. A village idiot and an Einstein are almost the same as minds in general. Anything less intelligent just reads as dumb. So an AI could pass mice and chimps while still seeming dumb, then reach ultra-Einstein within a month.

Content is everything outside a system's core algorithms: stored perceptions, skill libraries, and collections of facts. A system gets better both by being smarter and by knowing more. TextRunner, a University of Washington project, works across over 500 million web pages, and IBM's Watson won Jeopardy!. A descendant system that read with a ten-year-old's comprehension at TextRunner's speed could absorb the Library of Congress in weeks and become at least weakly superintelligent.

At human-level software, more processors, faster processors, or more copies of a system all boost collective and speed intelligence. In the short term, computing power scales with money. A PC-based system could be scaled up a thousandfold for a million dollars. Custom microprocessors add one or two more orders of magnitude, and Moore's law, which doubles power per dollar about every 18 months, leaves room before physical limits are reached.

There are also overhangs to consider. A hardware overhang means enough computing power already exists to run many fast copies. A content overhang means pre-made material like the internet is already available at human parity. An algorithm overhang means pre-designed enhancements may already exist, though this is less likely.

A fast takeoff only needs the design effort to keep growing. It doesn't require improvement to get easier. Before the crossover point, programmers, engineers, and the wider world supply most of that effort, and their input grows as the approach proves out. After the crossover, the system supplies most of its own. Each gain then feeds proportionally into the next, so growth becomes exponential. Doubling times might be measured in seconds.

With steady design effort and improvement getting easier over time, capacity doubles every 18 months. At crossover, applied effort has doubled, so the next doubling comes 7.5 months later. Within 17.9 months, capacity grows a thousandfold, yielding speed superintelligence.

Key Takeaways

• The speed of an intelligence explosion depends on how much design effort gets applied and how hard it is to keep improving the system.
• A slow takeoff is unlikely. If a takeoff happens, it will probably be explosive, and a moderate takeoff of months or years is the most important scenario to plan for.
• Capability is best judged by the general reasoning system, not overall competence. A system below its threshold reads as merely stupid, and above it improves as fast as the subsystem itself.
• Latent capacity already sits in existing hardware, online content, and pre-designed algorithmic enhancements.
• Once a system supplies most of its own design effort, capacity can rise a thousandfold in under two years.