When Jacob Coxon resigned earlier this month from Anthropic, citing dramatic concerns about AI racing humans toward extinction, it capped several years of public defections by employees warning that the AI companies are moving too fast and taking extraordinary risks.
In 2024, Jan Leike, OpenAI’s head of alignment, resigned, saying that safety had “taken a backseat to shiny products.” That same year, Miles Brundage, who held several roles at OpenAI, including senior adviser for AGI readiness, left the company, indicating he wanted to pursue more independent research and that “AI is unlikely to be as safe and beneficial as possible without a concerted effort to make it so.”
Anthropic itself was born out of similar concerns. Former OpenAI employees founded the company in 2021, describing their new venture as “an AI safety and research company.”
Behind the escalating warnings about superintelligence is a more fundamental issue, critics argue: There is no established path showing that today’s AI systems will surpass the smartest humans, much less a scientific basis for calculating the likelihood that such systems will destroy humanity. Yet the doomer narrative has increasingly overshadowed AI’s more immediate and concrete risks, while boosting the very companies whose products are being portrayed as extraordinarily powerful and potentially uncontrollable.
“I think Anthropic and OpenAI benefit a lot from claims that their system—their products, not science products—are so powerful that they might end humanity,” says David Widder, a professor at the University of Texas at Austin. “That’s, in some ways, capitalizing on fear to sell things, or at least to garner investment, garner interest in how powerful their systems are.”
A parlor game; not science
Coxon’s concerns emerge from the overlapping intellectual movements of effective altruism and longtermism that took shape in the 2000s at Oxford University in the United Kingdom and share ties with the rationalist community. “I’ve been associating with rationalists for almost a decade,” Coxon responded on X, when asked about his link to these movements. “Their ideas were compelling and you can’t deny their foresight.” (Coxon did not respond to a request for comment.)
Effective altruism centers around the idea of people doing the most good with their resources through charitable acts, while longtermism, a view within the effective altruism movement, holds that it is a moral priority to protect future beings from possible extinction. Longtermists consider the development of advanced artificial intelligence to be a promising opportunity for humanity, but also one of the biggest risks to its future.
Superintelligence could, in principle, help solve many societal problems—like poverty and the climate crisis—and lead to a better world. But thinkers within these movements warn that if these systems are not built properly to align with human values, they could pose existential risks.
The power attributed to superintelligence has led some in these communities to describe advanced AI in God-like terms. When discussing how humans might shape the character of these systems, for example, Will MacAskill, a prominent figure in the effective altruist movement, says that “writing a constitution that guides AI’s character is like writing instructions to god.”
“They believe in the advent of an almighty powerful AI god, which will either lead to destruction of humanity or help us achieve utopia,” says Boyan Milanov, Senior Research Scientist at AI Now Institute.
Some inside these movements have even attached a striking probability to AI destroying humankind: a figure known as p(doom) or probability of doom. In 2025, Anthropic CEO Dario Amodei once put this figure at 25 percent. Others, like Anthropic’s Alignment Science lead Evan Hubinger, have suggested the chances of extinction are more than 10 percent.
The odds of an AI apocalypse, according to leading AI figures
Many high-profile AI evangelists and alarmists alike have declared estimates of “P(doom),” probability of doom. Their predictions are all over the map.
Hover over or tap a face for details
How they have arrived at those numbers is unclear.
“What we have to keep in mind is that all this doomerism isn’t based on any scientific work,” says Milanov. “It really is science fiction. None of the claims are scientifically verifiable or falsifiable.”
Those involved in AI who claim a specific risk of human extinction should be pressed on the evidence, says Emily M. Bender, a professor and director of the Computational Linguistics Laboratory at the University of Washington. “‘How did you calculate that? Where did it come from? What is the data that went into your calculation? And what are the formulas? You’re giving me a number, so there’s got to be something behind it.’”
The numbers being thrown around now lack any of that, according to Bender. “This is really a parlor game; it’s not science,” she says.
The case for self-regulation
The heads of major AI companies like Anthropic and OpenAI each indicate that their organizations are the only ones who can build these systems responsibly with the proper guardrails that they themselves will decide on. They also say that their company’s superintelligence will align with human values instead of destroying everything. As such, they are now citing the grave risks of extinction in arguing for an industry-wide slowdown in AI.
Timnit Gebru, an AI researcher and author of the forthcoming book, Deep Unlearning, says these calls for a slowdown are part of an effort by the major AI companies to shape regulation in their favor. “Their whole point is that it just has to be done right,” says Gebru, who left Google in 2020 after a dispute over a paper warning about the risks of large language models.
Gebru, who now runs Distributed Artificial Intelligence Research Institute, or DAIR, an independent organization that scrutinizes Big Tech’s influence on AI, is raising the alarm on how AI hype is distracting from the immediate dangers of these systems and the corporations behind them.
She sees the calls to decelerate AI development as “clever messaging because the messaging is not: ‘We’re on the wrong track; this way of building tech is bad.’ They’re saying: ‘We should be on this path. We should just slow down.’”
The competition between these firms is also a way for them to advocate against outside regulation, she and other critics argue. They point to Anthropic’s chief and co-founder Dario Amodei’s essay published after Coxon’s resignation, titled We Must Pace the Frontier, proposing a three-step plan starting with evaluators embedded inside the companies.
The approach amounts to self-regulation with companies setting their own standards and processes, instead of being audited by independent parties who work in the interest of the public, according to Milanov. “If you read between the lines, everything he advocates [for] is something that would help companies like Anthropic,” he says.
Milanov and Gebru are both skeptical of industry-linked organizations like METR, or Model Evaluation and Threat Research, a non-profit established in 2023 that evaluates AI models’ catastrophic harms. On September 11, Joe Benton, an alignment researcher at Anthropic announced on X he had left the company two weeks earlier and will be joining METR.
But the links between the two organizations predate Benton’s move, and critics point to the close ties between AI companies and METR. “They’re [METR] both financially and ideologically aligned and linked to these big companies,” Milanov says.
Responding to a request for comment, a METR spokesperson indicated that of their approximately 50 staff members, one previously worked at Anthropic and another, Benton, will be incoming. They also provided the following as part of a statement: “We ensure financial and operational independence from frontier AI companies, because we think it is extremely important to minimize pressures we might face to adjust our reports or actions to align with frontier company interests.”
Anthropic did not respond to a request for comment.
AI risks are already here
As these systems rapidly develop, the scale of their tangible harms is becoming harder to ignore, strengthening the calls for tighter and more independent regulation. Recently, for instance, CNN reported that an AI-assisted intelligence report nearly triggered a United States military operation against a Chinese vessel last spring.
Such incidents, along with AI’s effects on jobs, privacy, and communities are prompting the public to pay closer attention to the technology and its impacts.
One indication of this concern is how people view data centers that serve AI models. In a New York Times/Siena poll of 1,503 potential voters conducted from September 8 to 13, 61 percent of all respondents opposed the construction of data centers. Yet, despite public awareness on some of the most visible issues, the discussion around existential risks overshadows the current concrete risks emerging from adopting and deploying these systems, distracting public and policymaker attention.
These concrete risks, Bender explains, range from environmental harms to labor exploitation and the theft of creative work and everyday data. She also points to widespread surveillance and the degradation of social services as people are encouraged to believe that machines can take over certain tasks once performed by humans.
“All of those are real harms that the public and policymakers should be focusing on, and instead we have Senator Sanders introducing a bill that is very much speaking to this fantasy of an all-powerful doomer show,” she adds, referring to Sanders’ proposed ban on superintelligence and a pause on development of advanced AI.
Pre-IPO hype
The public resignations of AI researchers has also raised concerns that these employees are using the spotlight to attract funding to start a new venture or to secure new positions. Anthropic co-founder Ilya Sutskever, who left the company over AI safety in 2024, was able to raise at least $3 billion in funding for his startup Safe Superintelligence.
Coxon’s resignation is the latest in a string of recent events in which the companies emphasized how unpredictable, risky, and impressive an all-powerful AI might be, causing a media frenzy.
In April, Anthropic said its latest model Mythos was too powerful to release to the public. On July 21, OpenAI publicly announced its role in the Hugging Face incident claiming some 1,200 autonomous agents broke out of their sandbox environment and attacked the online platform.
Just a few days later, Anthropic indicated they discovered three cases in which a model accessed the internet through an evaluation partner and gained unauthorized access to other organizations. In early September, OpenAI claimed that its agents had solved the decades-old Navier-Stokes puzzle, a long-unsolved math problem that comes with a $1 million prize.
The hype works because the warnings aren’t dissuading investors from putting their money into AI, says Widder, the professor at the University of Texas in Austin. “I think they’re hearing that and going, ‘Wow, these seem really powerful. Maybe I should consider this investment.’”
