Drink the Kool-Aid all you want, but don’t call AI an existential threat

By Jeff Caruso | April 29, 2024

Gif of HAL
A friendlier HAL from “2001: A Space Odyssey.” By Thomas Gaulkin / Cryteria (CC-BY)

In a 2012 essay for Aeon, physicist David Deutsch wrote of artificial intelligence: “I cannot think of any other significant field of knowledge where the prevailing wisdom, not only in society at large but among experts, is so beset with entrenched, overlapping, fundamental errors. Yet it has also been one of the most self-confident fields in prophesying that it will soon achieve the ultimate breakthrough.”

The self-confident prophesying Deutsch wrote of has since continued, except it is now accompanied with the warning that a rogue artificial intelligence could end life on Earth as we know it, and the technology should be categorized as an existential threat.

Artificial intelligence has come into the spotlight since 2022’s release of OpenAI’s ChatGPT, which was followed by several other text and image generation models. The applications have already been incorporated into businesses, healthcare settings, and even military operations. When given prompts, these systems respond by statistically predicting words or other forms of output based on the massive amounts of data they were trained on. In other words, they cannot think or rival human cognition. They are, at least for now, just mathematical models.

While generative AI can wreak havoc in many ways—such as amplifying existing biases or dispensing misinformation at scale—it’s not an existential threat any more than computer code is. It’s merely a tool that could, like many other tools, facilitate a bad outcome. It is, therefore, imperative that the public and policymakers understand this distinction to devise appropriate regulations that address the real risks of these applications and not fictional ones.

The thermonuclear bomb would never have been invented had it not been for an innovation in computing called the Von Neumann architecture whereby a computer could store data and run a program in memory. To successfully achieve that innovation, a single mathematical operation on a computer ironically named MANIAC at Los Alamos National Laboratory ran non-stop for 60 days. Which, then, was the weapon of mass destruction, or existential threat, the thermonuclear bomb, or the innovation in computing that brought it into existence? Obviously, it was the former.

Even in the case of lethal autonomous weapons or AI-controlled drone swarms, the designation belongs to the weapon, not to the tech that helped create, or control, it. But when the discussion centers on AI as an existential risk, the proponents are not speaking of an AI-controlled weapons platform. They’re speaking of a superintelligence—an entity whose cognitive abilities surpass human intelligence—that could wipe out humanity simply by doing what it was asked to do.

According to AI philosopher Nick Bostrom and computer scientist Geoffrey Hinton, the capabilities of a superintelligence would be exponentially greater than a human being in every way. If life were a game like the one in the television show Survivor, an artificial general intelligence (systems that could rival human intelligence) would outthink, outwit, outlast, or outright kill humans in short order because of two theoretical dilemmas: misalignment and instrumental convergence. Misalignment is what happens when an AI completes a task or operation in a way that is detrimental to humans or the environment. Instrumental convergence theorizes that an AI, when given a task, would maximize all available resources towards its completion.

One famous example of misaligned AI and instrumental convergence is the paperclip maximizer thought experiment that Bostrom came up with in 2003. “Suppose we have an AI whose only goal is to make as many paper clips as possible. The AI will realize quickly that it would be much better if there were no humans because humans might decide to switch it off. Because if humans do so, there would be fewer paper clips. Also, human bodies contain a lot of atoms that could be made into paper clips. The future that the AI would be trying to gear towards would be one in which there were a lot of paper clips but no humans.”

Wargames and AI: A dangerous mix that needs ethical oversight

Another theory, the Orthogonality Thesis (a term Bostrom took from mathematics and applied to artificial intelligence), says that distinct AIs with different goals will share similar values such as self-preservation, cognitive enhancement (software), technical enhancement (hardware), and resource acquisition (time, space, matter, and free energy).

These thought experiments and theories recall HAL, the antagonist and superintelligence featured in 2001: A Space Odyssey, a collaboration between iconic film director Stanley Kubrick and writer Arthur C. Clarke.

Kubrick, who used a familiar science fiction trope—that humans can no longer control the technology they created—provided some background to HAL in a 1969 interview, the year after the movie was released: “One of the things we were trying to convey in this part of the film is the reality of a world populated—as ours soon will be—by machine entities who have as much, or more, intelligence as human beings, and who have the same emotional potentialities in their personalities as human beings.”

In the movie, astronaut Dave Bowman, the protagonist, asks HAL to open the pod bay doors. To that, the supercomputer eerily responds, “I’m sorry, Dave. I’m afraid I can’t do that…this mission is too important for me to allow you to jeopardize it.”

That science fiction template of a runaway machine found its way into science as well. In 1964, RAND, a non-profit policy think tank, hired Hubert Dreyfus, a professor of philosophy at MIT, to review the work of AI researchers Allan Newell and Herbert A. Simon. Newell was one of AI’s earliest scientists who in 1957 predicted that in 10 years a digital computer would accomplish three remarkable achievements: become a world champion in chess, discover a new mathematical theorem, and write music that will be acclaimed by critics. That level of optimism energized his work at RAND and, since he and Simon’s research presumably had defense implications, the think tank was looking for an objective review by an outside expert. In 1965, Dreyfus shared his findings in Alchemy and Artificial Intelligence and refuted the thesis that human and computer intelligence operate in similar ways. Both Newell and Simon called the paper nonsense and fought against its release. They eventually lost that battle, and Dreyfuss’s analysis became one of RAND’s most popular papers.

Artificial intelligence research has come a long way since then. Today, the world is transfixed by what AI can do. Google DeepMind AlphaGo’s momentous victory against World Go champion Lee Sodel in 2015 was instrumental in Elon Musk and Sam Altman’s decision to launch OpenAI with the intention of achieving the benefits of artificial general intelligence safely while mitigating the existential risks. ChatGPT became the fastest growing app in history with 100 hundred million active users just two months after its release. Soon, AI startups popped up everywhere as venture capitalists dumped billions of dollars into new players including Anthropic, started by former OpenAI employees that raised $7.3 billion in its first year. Everyone was, and still seems to be, drinking the Kool-Aid and the AI innovation race is on.

Despite all that investment and progress, there is no evidence AI is closer to achieving superintelligence than it was in 1965 when Dreyfus wrote: “An alchemist would surely have considered it rather pessimistic and petty to insist that, since the creation of quicksilver, he had produced many beautifully colored solutions but not a speck of gold; he would probably have considered such a critique extremely unfair. But if the alchemist had stopped poring over his retorts and pentagrams and had spent his time looking for the true structure of the problem, things would have been set moving in a more promising direction.”

Could AI help bioterrorists unleash a new pandemic? A new study suggests not yet

Predictions of future catastrophe rely upon the ability of AI to apply knowledge it learned in one area to problem solving in a completely different area that it hasn’t been trained on. This broad generalization capability is the key ingredient to achieving human-level intelligence. That not only hasn’t happened, but there’s no clear path to achieving it.

The AI must also be strongly goal oriented to the point where it will take independent action to prevent any interference in the achievement of its objective. There’s no evidence that such a thing is even feasible.

Additionally, any suggestion of a rogue AI showing ill-intent requires that it be conscious and self-aware. That is the stuff of alchemy and other esoteric magical traditions, not science.

In Why general artificial intelligence will not be realized, Norwegian physicist and philosopher Ragnar Fjelland points out that Dreyfus’s critiques still hold today; that human intelligence is partly transmitted in ways that are not taught but are communicated tacitly; ways that cannot be duplicated using algorithms or code.

Turing Award winner Yann LeCun, the Chief AI Scientist at Meta and the Silver Professor of the Courant Institute of Mathematical Sciences at New York University explains the problem this way: “A 4-year-old child has seen 50x more information than the biggest LLMs [large language models] that we have.” If you do the math, that’s 20 megabytes per second through the optic nerve for 16,000 wake hours, and that doesn’t include how much data is received through the other senses.

Considering the lack of any substantive evidence supporting the existential risk theory, it’s puzzling why many in the fields of computer science and philosophy appear to believe AI is an existential threat. It turns out, there aren’t that many who have bought into the theory. A recent poll of more than 2,000 working artificial intelligence engineers and researchers by AI Impacts put the risk of human extinction by AI at only five percent.

Also, despite the fact that thousands of experts signed petitions on AI risk, it doesn’t necessarily mean they agree with those statements, as was discovered by two MIT students who took a closer look at the Future of Life Institute’s 2023 open letter to pause AI. The undergraduate students, Isabella Struckman and Sofie Kupiec, discovered that “very few claimed to sign because they agreed with all or even most of the letter.”

These concerns might have gained traction because AI safety is an extremely well-funded segment of the tech industry, and developers have an interest in securing a voice on AI policy in the United States and European Union, especially when it comes to future regulation of the industry. Fear mongering tends to be more successful and work faster than objective reasoning in convincing an underinformed group of people, which legislators can be, to adopt policy recommendations. In this case, the AI safety lobby hopes governments regulate the use, rather than the development, of artificial intelligence. In other words, the lobby wants to put the onus on the user, not on the developer. If that sounds familiar, it’s because the tech industry has been beating the same drum for the past 40 years: “If you regulate us, you hurt innovation. We’ll just have customers accept an end-user licensing agreement and all will be well.”

The specter of a rogue superintelligence is just a distraction by the tech industry to stave off the regulations that they know are coming their way.

Together, we make the world safer.

The Bulletin elevates expert voices above the noise. But as an independent nonprofit organization, our operations depend on the support of readers like you. Help us continue to deliver quality journalism that holds leaders accountable. Your support of our work at any level is important. In return, we promise our coverage will be understandable, influential, vigilant, solution-oriented, and fair-minded. Together we can make a difference.

Get alerts about this thread
Notify of
Newest Most Voted
Inline Feedbacks
View all comments
Grant Castillou
Grant Castillou
22 days ago

It’s becoming clear that with all the brain and consciousness theories out there, the proof will be in the pudding. By this I mean, can any particular theory be used to create a human adult level conscious machine. My bet is on the late Gerald Edelman’s Extended Theory of Neuronal Group Selection. The lead group in robotics based on this theory is the Neurorobotics Lab at UC at Irvine. Dr. Edelman distinguished between primary consciousness, which came first in evolution, and that humans share with other conscious animals, and higher order consciousness, which came to only humans with the acquisition… Read more »

Gary Robert Childress
Gary Robert Childress
13 days ago

Why should we want to create a conscious *machine*? Can we try it for manslaughter if it ends up killing someone? If we unplug it, are we guilty of manslaughter? AI research seems more concerned with immediate economic benefits than it is with long-term ethical questions and concerns. And when someone raises the issue of responsibility of scientists for what they create as I have done with the case of Oppenheimer’s responsibility for the destruction and terror of his creation, the comments disappear into ether space never to show up in discussion. Accountability doesn’t seem to be a popular topic… Read more »

William Baird
19 days ago

Factual statements. The statement “Additionally, any suggestion of a rogue AI showing ill-intent requires that it be conscious and self-aware.” is false. There existed a non-conscious autonomous AI agent ChaosGPT whose first goal was “Destroy Humanity” Link. Link. Link. Link. Link.   The statement “…there is no evidence AI is closer to achieving superintelligence than it was in 1965…” is false. Since 1965, there have been over 10,000 papers published on AI, many of which have advanced capabilities. Link. Since 1965, AI have improved on a range of benchmark tests. Link. Link. Link. Link. Link. Link. Link. Link. Link. Link.… Read more »

William Baird
19 days ago

My Opinions. I am deeply concerned by the title. Its central claim is refuted by thousands of scientists, and by the Bulletin publishing it, the article damages the Bulletins reputation as a source of reliable scientific information. Hundreds of the top AI scientists have signed the statement “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.” Link. Additionally, over one thousand AI scientists and engineers that were surveyed gave a 5% or higher chance of human extinction from AI. Link. 5% is non-zero.  In my opinion, people should… Read more »

19 days ago

I agree with this article’s claim that AI is not conscious and there is no clear path to AGI or AI consciousness, and the notion of a superintelligent mind through AI is a kind of myth. But that doesn’t mean that AI is not dangerous, as any new and increasingly powerful tool can be dangerous. Humans with bad intent could direct AI to do disastrous things.