Anthropic Warns AI Is Approaching Self-Improving Intelligence

AI development has been accelerated rapidly by companies in an effort to remain competitive in the market, while Favaro and Clark argue that a slower pace of advancement would provide additional time for the broader implications of the technology to be addressed.

AI development is being accelerated at a rapid pace, and US-based AI firm Anthropic has warned that systems could soon be capable of building, training, and improving themselves without human involvement, leading the firm to recommend a slowdown in development.

In a blog post published on Thursday, it was stated by Marina Favaro, lead at the Anthropic Institute, along with Anthropic co-founder Jack Clark, that AI agents are already capable of running code independently, delegating hours of tasks to other agents, and may soon reach a point where full control could be taken over by them.

“For most of AI’s history, humans drove every step in its development cycle. But at Anthropic, we are delegating a growing share of AI development to AI systems themselves, which is speeding up our work,” they said.

“Taken far enough, and given enough compute, that trend points to an AI system capable of fully autonomously designing and developing its own successor,” Favaro and Clark added.

Concerns have been raised about potential outcomes if AI systems become capable of self-directed intelligence growth. In December, OpenAI stated that research is being conducted into the safe development and deployment of increasingly advanced AI systems, including those capable of recursive self-improvement.

“We want these systems to consistently follow human intent in complex, real-world scenarios and adversarial conditions, avoid catastrophic behavior, and remain controllable, auditable, and aligned with human values,” it said.

A recruitment process is also underway by the company for a researcher focused on recursive self-improvement preparedness, a role that is part of its Safety Research team.

AI model improvement is reported by Favaro and Clark to have been roughly doubling every four months instead of every seven months. At each stage, the role of humans is being reduced, with Anthropic’s Claude model generating about 80% of the code that is merged into the company’s codebase.

“We are not there yet, and recursive self-improvement is not inevitable. But it could come sooner than most institutions are prepared for,” they said.

“Once human- and AI-authored code quality reach parity, humans will stop writing code entirely and shift to only reviewing it. But if they can’t review code as quickly as Claude can generate it, human review will become the bottleneck to AI development,” they added.

Favaro and Clark also stated that a slowdown in AI development, to provide additional time for addressing its “immense” implications, would be considered the ideal approach.

Anthropic Halts Claude Mythos Release Over Global Cybersecurity Risks

In April, the release of Anthropic’s AI model, Claude Mythos, was ruled out by Anthropic due to concerns regarding potential risks to global cybersecurity.

At the same time, an open letter was released on Thursday by a group of tech leaders, including members associated with Anthropic and OpenAI, calling on lawmakers to implement stronger regulatory safeguards around the technology amid concerns that it could be used to bypass long-standing “knowledge barriers” that have historically prevented malicious actors from developing biological weapons.

“We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology,” Favaro and Clark said.

AI agents are being adopted at a rapidly increasing rate, including within the crypto sector. It has been speculated by some crypto executives that transaction settlement handled by AI agents could boost adoption levels and increase overall transaction volumes. In January, Circle CEO Jeremy Allaire predicted that billions of AI agents would be operating on behalf of users within the next five years.

Last month, it was reported by crypto investment firm Keyrock that AI agents handling payment settlements have moved from concept to practical implementation over the past 12 months, with around $73 million being settled across 176 million transactions.