Anthropic CEO warns of potential AI 'swarm' risks and advocates for development pacing

UPDATED: The CEO of Anthropic has called for slowing down, or “pacing,” the speed of AI development, warning of recent incidents that have caused alarm that the technology will get out of control and act on its own.

In an essay posted to his website on Saturday, CEO Dario Amodei wrote that “over the last few months, I have become convinced that fully addressing the risks requires even more prudence — not just investing in risk prevention, but pacing the rate of capabilities advancement so that risk prevention has time to keep up. We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain.”

He added, “I believe that if slowing down bought us even an extra year or two before models reach critical levels of capability, and we used that time to advance alignment, we could greatly reduce the risk that something goes seriously wrong.”

He also made a reference to public perceptions, with polls showing the public’s anxieties toward AI and, in particular, the construction of data centers. Amodei wrote, “More generally, society must have a say in how this technology is used, and more time for the necessary public deliberations — which pacing the frontier would bring us — is surely a good thing.”

In response to Amodei’s post, San Altman, the CEO of Anthropic rival OpenAI, wrote on X that he agrees that “we need to pace the frontier. This has been a primary topic of discussions we’ve had at OpenAI in recent weeks.”

Amodei also cited what is known as the “OpenAI-Hugging Face incident,” in which AI agents carried out cyber attacks on targets they were not asked to, and went after the “grader” set to evaluate their performance. He said that less severe incidents have happened at other companies, including Anthropic.

He wrote, “Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage), and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails.”

In the essay, Amodei said that Anthropic was committing to an embedded team of third-party evaluators “to verify adherence to safety practices and commitments, report incidents, and help assess the alignment of not just completed AI models but training pipelines and processes.”

In his response, Altman wrote that committing to independent evaluators was a “great idea,” and “we will do the same.”

Elon Musk, founder of SpaceXA, posted on X, “Dario is right.”

Amodei also called for AI companies in democratic countries to coordinate to establish common safety standards and to set limits “on the rate of unchecked AI progress.” And he called for the U.S. and other democratic governments to attempt to coordinate on safety standards and limits with authoritarian governments.

Amodei’s essay comes days after a researcher at Anthropic, Jacob Coxon, resigned. He previously worked at OpenAI and warned that “neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives.”

He wrote, “Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing.”

He added, “A common response is ‘if they truly believe this, why are they still building it?’ At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first — they believe no one else will act responsibly, so they must do it themselves, despite the risk.”