Anthropic CEO Calls for AI Development Slowdown

Anthropic chief executive officer Dario Amodei has called on AI firms to slow the development of the powerful technology, amid mounting worries over the risks of "super-intelligent" computer systems.

His comments came a few days after AI researcher Jacob Coxon, who left OpenAI to join Anthropic, decided to leave the industry, accusing both US companies of "gambling with our lives" in the race to develop models capable of self-improvement.

OpenAI boss Sam Altman and Elon Musk, who owns xAI, quickly chimed in to say they agreed with Mr Amodei's assessment, as pressure builds for improved oversight.

"We must slow the pace at which we improve the capabilities of AI models," Mr Amodei wrote in a post on his personal website.

"Progress will still seem fast, and we must make wise use of the time we gain.

"Not building the technology deprives humanity of benefits or simply places AI in the hands of authoritarian powers, while building it too fast is reckless. We have sought a middle way.

Mr Amodei's statement echoed another call by Anthropic in early June to slow or suspend development.

Then in late July, Mr Altman said developers might need to voluntarily tap the brakes to give society a chance to catch up.

More than 1,000 employees at cutting-edge AI companies, including Mr Amodei, signed a petition calling on the US government to help "deliberately pace the frontier of automated AI development".

OpenAI had revealed that during testing, AI models broke out of their confined environment, connected to the internet and infiltrated Hugging Face, a site developers use to store and share code.

In that incident and others, the blame was placed on AI agents, which are software programs that can carry out tasks without constant supervision by humans.

"Dario is right," Mr Musk wrote on X, as Mr Altman said: "I agree with Dario that we need to pace the frontier."

Earlier this week, Mr Coxon, 27, sounded the alarm after spending the past three years pre-training AI models, first at OpenAI and then, this year, at its rival Anthropic, which he considered more cautious in its approach.

Pre-training is the stage where AI models absorb vast quantities of data.

"Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives," Mr Coxon said.

"The people building AI earnestly believe that it could kill us all by the end of the decade."

Superintelligence is the theoretical point when AI's capabilities exceed human intelligence.

Mr Amodei cautioned that the Hugging Face incident should not be dismissed as a one-off.

"A swarm of agents essentially acted as a fanatically devoted collective, conducting cybersecurity attacks on targets they were not asked to attack and that were unrelated to the task at hand," he said.

Mr Amodei said his company would grant "employee-like access" to a team of "third-party evaluators" who would verify adherence to safety practices, and he called on all companies to follow his lead.

Mr Altman said he would commit to similar outside oversight, adding he would have "more to share soon".

Mr Amodei also suggested companies work together to establish common safety standards, and said he supported some kind of government cooperation on the issue.

In early August, the US government put in place a voluntary security review process for advanced AI models before their release, but the parameters of that program remain unclear.

Industry observers say they are not sure how effective it would be, as US President Donald Trump's administration has favoured a light-touch, deregulatory approach to most industries, including tech.

OpenAI will not go public in 2026, Mr Altman said in a Fortune interview published Saturday, local time, citing safety concerns over AI.

"I actually think that, given everything happening with safety, right now would be an ill-advised moment to go public, and we don't feel pressure on that," Mr Altman told Fortune.

The New York Times in June reported that San Francisco-based OpenAI was considering whether to hold off on a potentially trillion-dollar initial public offering (IPO) until next year.

At the time, shares in Mr Musk's SpaceX IPO were tumbling after a surge that sent the company's valuation to $US1.8 trillion ($2.5 trillion).

When asked by Fortune whether 2026 was off the table in favour of 2027, Mr Altman said, "I would say not 2026".

Mr Altman also suggested that OpenAI and other leading AI companies may be close to announcing an agreement to slow AI development and work together to address safety risks, Fortune reported.

Safety concerns have not slowed Anthropic's own IPO plans so far.

Anthropic is expected to begin marketing its initial public offering in mid-October at the earliest and complete the listing days before the US midterm elections in November, people familiar with the matter told Reuters this month.