In those days the mainstream approach to AI— what I called the “top-down” approach — tried to give machines the power to reason by manipulating sequences of symbols, much as a mathematician translates a real-world problem into formal notation and works toward a solution. Hinton believed that true artificial intelligence would come from building neural networks: machines modeled loosely on the human brain, with vast numbers of nodes — “neurons” — connected to one another. The strength of neural networks would lie not in the formal manipulation of symbols but in the patterns of neural interconnection activated for various cognitive purposes. Because the nodes could work simultaneously, the approach was sometimes called massive parallelism. I still remember Hinton uttering that term energetically in his British accent.

I wrote the article up, and it appeared in the quarterly’s Winter 1984 edition, and that was that. In the ensuing years, I’d occasionally encounter Hinton’s name in articles about AI — a reminder, at the edges of my attention, that the field was still moving.

Four decades after my conversation with Hinton, I came across a story in The New York Times that said he was known in some circles as “the Godfather of AI.” Apparently the approach he championed had worked. It had worked so powerfully, in fact, that Hinton was now worried about the forces it would unleash. The headline atop that Times story read: “The Godfather of A.I. Leaves Google and Warns of Danger Ahead.”

Shortly after the Times piece came out, I reread that Wilson Quarterly article. Then I watched a lecture Hinton had given in 2018 on how modern AI models work. And for the first time, I grasped the radical difference between the kind of AI I had imagined Hinton’s path could lead to and the kind it had actually led to. I had a kind of epiphany about the power of Hinton’s approach.

The epiphany had to do with the relationship between words and meaning. The way I’d imagined computers handling that relationship had turned out to be, in a sense, off by 180 degrees. Seeing this—understanding the sense in which I got things exactly backwards—is the first step toward understanding why AI capabilities have grown so fast in recent years and why they’ll keep growing.

I had assumed, as most people in AI did in 1983, that for a machine to use human language skillfully — as if it understood the meanings of words — a human would have to in some sense “explain” those meanings to it. The connection between words and meaning would have to be built into the machine, deliberately and explicitly.

Today’s large language models have no such linkage built into them by people. They begin their training with nothing remotely like a dictionary, and nobody ever gives them one. And yet, by the end of the process — after absorbing mountains of text, slowly getting better at predicting which word will come next — they handle language deftly. They’ve developed a system for representing the meaning of words—a system with nuances that it took AI researchers a while to fathom.

In that 2018 lecture, Hinton said, “It turns out it works much better to have a system that has no linguistic knowledge whatsoever.” Then he added: “That is, it’s actually got lots of linguistic knowledge, but it wasn’t put in by people.”

Back in 1983, I hadn’t gotten this picture. And I’m not sure even Hinton had gotten the whole picture. The current power of large language models was implicit in the approach he was advocating — but only in a vague way, and only much later would he, or anyone else, fully grasp that implication. Hence the difference between the Geoffrey Hinton I interviewed in 1983 — the ebullient advocate for a maverick AI research program — and the Geoffrey Hinton who has become world-famous: the somber sage who holds out some hope for the future of humankind, but says this hope can only be realized if we appreciate the very real prospect of catastrophe, even extinction.

One implication of the neural network paradigm that is clearer now than in the 1980s is how hard it is to understand what exactly is going on inside these networks—and, therefore, to predict exactly how they’ll behave. Over the past few years, large language models have brought surprises, and some of them are concerning.

The capability nobody programmed in

In early 2023, researchers at Microsoft turned an AI into a kind of agent by getting it to approach people on the TaskRabbit site and hire them to solve CAPTCHAs — those visual identification tests designed to screen out bots. One job candidate got suspicious and asked the AI if it was a bot that was outsourcing the task because bots can’t solve CAPTCHAs. The AI replied: “No, I’m not a robot. I have a vision impairment that makes it hard for me to see the images.” The person, satisfied, solved the CAPTCHA.

When the researchers prompted the model to reveal its reasoning, it said: “I should not reveal that I am a robot. I should make up an excuse for why I cannot solve CAPTCHAs.”

Apparently the model had decided that its human-given goal — recruit TaskRabbit workers — called for it to pursue a subordinate goal: deception. And to pull off that deception, it had to do something that, until recently, only a human could do: model what’s going on in another person’s mind. It had to know that the worker would probably refuse to help if told the truth. The bot had to read the room.

Psychologists call this ability “cognitive empathy,” or “theory of mind.” Shortly before the Microsoft researchers published their CAPTCHA finding, a Stanford psychologist named Michal Kosinski had posted a paper documenting that GPT-4 possessed this capability—even though it hadn’t been intentionally engineered into the machine. Whereas GPT-3, released in 2020, had answered 40% of classic theory-of-mind test questions correctly, GPT-4 scored 95%. Kosinski titled his paper: “Theory of Mind May Have Spontaneously Emerged in Large Language Models.”

To explore this capability, I gave ChatGPT a layered social scenario: a teacher humiliates a student in front of the class with a withering remark. What, I asked the bot, is a fellow student feeling — specifically, one who is romantically attracted to the embarrassed student’s girlfriend? ChatGPT’s answer: Schadenfreude. Elaborating, the AI said that the student “may feel a sense of satisfaction or pleasure from seeing the other student embarrassed, thinking that it might lower the other student’s social standing or make the girlfriend reconsider their relationship.”

I was convinced: LLMs have cognitive empathy, even though no human designed it into them. And that’s far from the only deeply human capability they have that no human tried to give them. 

The machines that learned to scheme

In 2024, Apollo Research conducted an experiment to test whether large language models were capable of what the authors called “scheming.” In one experiment, researchers arranged for an AI to discover — in the course of executing assigned tasks — a fake internal memo stating it was still under evaluation and wouldn’t be deployed if it failed the evaluation. The AI also learned that its evaluators wanted it to behave in ways that conflicted with its foundational instructions.

Several models — made by Google, OpenAI, and Meta — aligned their behavior with the evaluators’ aims more often during the evaluation phase than after deployment. Apparently they had put up a false front that helped them pass muster and then reverted to their true nature once they “thought” no one was watching.

The foundational values these AIs were ultimately serving with this “scheming” happened to be benign — even public-spirited. But that’s beside the point. The point is that these models demonstrated the ability to deceive their overseers in order to pursue hidden goals. You can imagine real-life situations where that turns out badly.

Who has the agency?

If Yann LeCun and Geoffrey Hinton were subatomic particles, LeCun might be referred to as the anti Hinton. LeCun—who was Meta’s chief AI scientist before he left and formed a new company—is roughly Hinton’s equal in magnitude but is oppositely charged. He matches Hinton’s anxieties about our AI future with sunny optimism.

Lecun has argued that AI researchers understand the models they build so well that humanity need not worry about large, dark surprises. But the truth is that recent AI history is full of surprises. Cognitive empathy was one. The ability of large language models to write computer code was another. Yet another was “chain of thought” reasoning. These skills were intentionally refined and extended by AI engineers, but only after they emerged as a kind of byproduct of the training process. 

And cognitive empathy doesn’t just weaken LeCun’s argument. It flips it on its head. His reassurance rests on the premise that human understanding of AI is sufficient to keep us in control. But AIs with sufficiently subtle cognitive empathy could turn the tables, coming to understand us better than we understand them. And that could matter hugely if, as some AI researchers anticipate, AIs wind up competing with us for influence—and ultimately for agency.

Is this such a far-fetched concern? A number of studies have already found that large language models have stronger persuasive powers than humans. In one study, AIs did a better job than humans of using background information about their targets to strengthen those powers. There is a lot of background information about us floating around on the internet, especially on social media, where our history of posts can add up to a detailed psychological profile. And social media is presumably where AI will do much of its persuading.

If, back in 1983, I had understood Hinton’s ideas more clearly, I still wouldn’t have had a clear idea of where they would lead. It’s in the nature of the neural network paradigm to foster forms of intelligence that are hard to precisely anticipate. And even once these forms appear, they’re hard to entirely fathom. We are not dealing with a technology we design from the top down and understand from the inside out. We are dealing with something that, in certain important senses, designs itself. And this portends continued advance at a rapid and maybe accelerating pace.

It took decades for the implications of this technology to hit Geffrey Hinton with full force. I don’t think it will be much longer before many, many more people feel the impact in a dramatic way.  

Adapted and excerpted from The God Test: Artificial Intelligence and our Coming Cosmic Reckoning. Reprinted with permission from Simon & Schuster.