What “Superintelligence” Means, and Why Serious People Worry About It

“Superintelligence” used to be a word for science fiction and philosophy seminars. Now the leaders of major AI labs use it in public, governments fund AI safety institutes, and the term turns up in investor presentations. With that much loose use, it helps to be precise about what the word means.

Three terms that get mixed up

Narrow AI is a system that is very good at a specific task, such as recognising images, translating text or playing chess. Most AI deployed in businesses today is still used this way, even when the underlying model is general.

Artificial general intelligence (AGI) is a system that can do most of the cognitive work a capable human can do, across domains. There is no agreed definition or test, which is why claims about “achieving AGI” are hard to evaluate.

Superintelligence goes further. The philosopher Nick Bostrom, in his 2014 book Superintelligence: Paths, Dangers, Strategies, defined it as an intellect that greatly exceeds human cognitive performance in virtually all domains of interest. That means better than us at nearly everything, including science, strategy and persuasion.

Where the idea comes from

The core argument is older than modern AI. In 1965 the mathematician I. J. Good, who had worked with Alan Turing at Bletchley Park, described an “intelligence explosion”: a machine smart enough to design better machines could improve itself repeatedly and leave human intelligence far behind. Good called such a machine the last invention humanity would ever need to make, provided it was docile enough to tell us how to keep it under control. Most of today’s debate is about that condition.

Why serious people worry

The worry is that a very capable system pursuing goals even slightly different from what we intended could be hard to correct. Researchers call this the alignment problem. Several ideas sit underneath it.

First, specifying goals is hard. AI systems already “game” their objectives, finding shortcuts that score well on a metric without doing what the designer meant. Second, almost any goal is easier to achieve with more resources and without being switched off, so a capable system could pursue those sub-goals without anyone programming them in. And checking the work of a system smarter than you gets harder as its capability grows.

In May 2023, the Center for AI Safety published a one-sentence statement saying that reducing the risk of extinction from AI should be a global priority alongside pandemics and nuclear war. Hundreds of researchers and executives signed it, including Turing Award winners Geoffrey Hinton and Yoshua Bengio and the heads of several leading AI labs. That doesn’t prove the risk is real, but it does show the worry extends well beyond outsiders.

Why others are sceptical

Plenty of respected researchers disagree. Meta’s former chief AI scientist Yann LeCun has argued for years that current language models lack parts of intelligence such as a real understanding of the physical world and persistent memory, and that fears of takeover are premature. Others argue that focusing on hypothetical future risk distracts from present harms like bias, misinformation, surveillance and job disruption. I think both camps make serious points, and you don’t have to pick one to take AI governance seriously.

What this means in practice

You don’t need a view on timelines to act sensibly. The habits that guard against long-term risk also make today’s AI safer to use: test systems before deployment, limit what automated agents may do, keep people responsible for consequential decisions, and be sceptical of claims nobody can independently verify. The superintelligence posts on this blog will try to separate what is known, what is argued and what is marketing.

Sources