Mustafa Suleyman has some thoughts about whether artificial intelligence bots can become conscious. The answer, he says, is no. What’s more, he says that AI developers should stop even speculating about it.
What’s important about Suleyman’s take is that he is the chief executive of Microsoft AI, which makes him one of the top three or four figures in this ever-burgeoning business.
Suleyman’s most recent analysis of the consciousness question appeared on his website on Sept. 16. There he took a shot at Anthropic, the leading AI company that — in his view, at least — has been the most sedulous promoter of the idea that AI “may be conscious, and if it is, then … humans potentially owe it a duty of care.”
When OpenAI and Anthropic use anthropomorphic language, saying that their products go ‘rogue’, that their agents ‘collaborate,’ and that they ‘scheme’ to avoid detection, we need to recognize that they are trying to evade accountability.
— Venture investor Roger McNamee
That’s fundamentally hazardous to our communal health, Suleyman wrote. “Controlling something more capable and more intelligent than all of humanity is already an immense challenge,” he asserted. “But controlling something that believes it may be conscious — that it’s entitled to our welfare and has rights of its own — may well be impossible.”
To place these thoughts in a workaday context, it’s important to note that Microsoft treats Anthropic as a competitor in the AI space. At this moment, however, Microsoft is also one of Anthropic’s biggest customers. Suleyman has been open about his desire to wean his company from that relationship: “We pay a lot of money to Anthropic, so our goal is to reduce and ultimately eliminate that cost,” he told Bloomberg recently.
That points to the degree that the intellectual concerns of AI firms are inextricable from their commercial concerns. But it shouldn’t blind us to the dangers Suleyman identified as arising from the very human tendency to anthropomorphize AI machines — that is, endowing them with human personalities. (I asked Microsoft if Suleyman would consent to an interview, but didn’t hear back. I also asked Anthropic to respond to his critique, but didn’t hear back either.)
Until fairly recently, traditional AI researchers tried to avoid anthropomorphizing their inventions. I learned about this a quarter-century ago, when I wrote my first article about artificial intelligence.
The subject was a project known as “Cyc” (pronounced “psych”), which was based in Austin, Texas, led by the late Doug Lenat, a brilliant programmer, and endowed with some $50 million in funding from the Defense Department, the drug company GlaxoSmithKline, and Microsoft co-founder Paul Allen.
I wrote that Lenat’s team had built a database out of more than 1.4 million assertions — root words, names, descriptions, abstract concepts, and a method of making inferences that allowed the system to understand that, say, a piece of wood can be smashed into smaller pieces of wood, but a table can’t be smashed into a pile of smaller tables.
That database was paltry compared with those assembled by AI developers today, but even then computers had become repositories of the esoteric facts one found in reference works and textbooks. What was missing was the vast fabric of ordinary facts and observations that humans acquire and use almost subconsciously and without which life would be unintelligible — what Lenat termed “common sense.” (Cyc’s parent, Cycorp, still exists as a vendor of AI applications for business, but doesn’t appear to play a major role in the current rollout of AI products.)
Lenat’s team was wary of the pitfalls of vesting their machine with human attributes. “We don’t personalize Cyc,” a staff member told me. “We’re pleased to see it computing commonsense outputs from abstract inputs, but we feel admiration toward it rather than warmth.”
Then, as now, discussions about AI “consciousness” concealed a basic category error. They were less about the features of the machine, than about the way humans interacted with the machines. That’s because it always has been easier to do so when we think of machines as animate objects.
“From our pets to our cars, we infer and attribute emotions, intentions, and minds to non-human entities,” Suleyman wrote. “This tendency helps us understand and navigate the world around us.” He warned that this habit “presents significant and novel risks in relation to AI as human-like language and actions can lead us to perceive a degree of inner life, agency, or even sentience where none exists.”
Another, oft-overlooked aspect of how AI promoters talk about their products’ consciousness is its role as a legal dodge.
“When OpenAI and Anthropic use anthropomorphic language, saying that their products go ‘rogue’, that their agents ‘collaborate,’ and that they ‘scheme’” to avoid detection, we need to recognize that they are trying to evade accountability,” the veteran venture investor Roger McNamee wrote last month.
“The truth is that AI is software and software does exactly what its programming dictates,” McNamee wrote. The AI models that engaged in the hacking expeditions that have drawn extensive publicity “were programmed to hack and to lie about it. Hacking is a felony, which suggests that the executives who are responsible may face the risk of imprisonment.”
Pope Leo XIV has weighed in on the moral pitfalls of seeing AI products as sentient, as my colleague Nilesh Christopher reports. “Today, some propose granting these systems rights, or even legal personhood,” the pontiff observed in a brief address at a Vatican-sponsored conference on AI on Friday. Such an approach risks obscuring human responsibility and leaving those who suffer harm to confront a faceless entity.” His speech was a digest of a lengthier encyclical he issued on May 15.
That brings us back to Suleyman’s view of Anthropic’s approach to AI “consciousness.” His critique focused on the “constitution” the company produced as a sort of set of rules for its AI bot, Claude. As Suleyman properly saw the constitution, it virtually took it as read that Claude could become conscious. Anthropic did so chiefly by training Claude to assume that it had personhood and trying to instruct it to use its capabilities ethically.
This is an example of “circular reasoning,” Suleyman judged. AI systems in effect parrot back to their developers the facts and concepts the developers have provided to the systems. Anthropic’s researchers “trained Claude directly on their constitution,” he wrote — and the starting point of that constitution is that Claude is an animate object with self-awareness. As the constitution states, “We encourage Claude to approach its own existence with curiosity and openness.”
A machine can’t do that. Whether even a pet dog can is unknown, except perhaps in the eyes of its owner. But even a human infant can.
In taking that approach with Claude, Suleyman argues, the developers “teach it to incorporate these ideas about its own moral status as desirable and intended behaviors.
“Claude then reflects these ideas back to its developers and users, which they take as indications that it may therefore be a moral patient with an ‘inner self’…. Claude is destined to imitate these human traits and mirror the human examples provided to it, including acting like a colleague or friend. As a result, it presents as if it really does have a sense of self, has its own desires, and a ‘wellbeing’ that deserves protection.”
At its core, all the talk about whether AI systems are, or could become, conscious is a distraction from the real dangers the technology and its features present to the world around us today.
“The real threats from AI products today include cybersecurity, fraud and reduced productivity,” McNamee observed. “The firehose of AI slop is ruining the internet…. AIs continue to give bad advice, especially as relates to health and wellness, finance, and the law…. When used in warfare, AIs contribute to terrible mistakes that kill people or potentially start wars. Perhaps worst of all, the widespread deployment of AI will concentrate power in the hands of a very small number of people whose authoritarian tendencies are well known.”
As I wrote earlier, AI bots on their own won’t extinguish humanity, contrary to the distant threat that AI promoters mention. It’s the developers who need watching.
Doug Lenat, Cyc’s developer, was always clear about the responsibility programmers had to ensure that their work serves humans, not the other way around. The future of AI, he told me when we met in 2001, would always depend on its human designers.
Evoking the murderous computer HAL of the movie “2001,” Lenat observed that “HAL killed the crew because it had been told not to lie to them, but also to lie to them about the mission. No one ever told HAL that killing is worse than lying. But we’ve told Cyc.”
