Anthropic Says Maybe We Should Pause AI

Posted by Adam Danyal in AI Risk on

Anthropic has published a report on recursive self-improvement, the point at which an AI system could autonomously design and develop its own successor. The company says this has not happened and may never happen. It argues that current progress is moving in that direction: AI systems are taking on a growing share of the work involved in building AI systems, and that is speeding up the development cycle.

The report describes a sequence inside Anthropic’s own work. In the early days, people wrote code and documents themselves. Chatbots then helped with short snippets that humans copied into editors. More capable coding agents began writing and editing files on their own. Today, Anthropic says agents can run code themselves and delegate hours of work to other agents. The next stage is “closing the loop,” where agents become capable enough to build and train models themselves. If that happens, future versions of Claude could be improved by Claude.

Public benchmarks in the report show AI systems working across longer tasks. Anthropic says the length of tasks AI systems can reliably complete on their own has been doubling roughly every four months. In March 2024, Claude Opus 3 could complete software tasks that took humans about four minutes. A year later, Claude Sonnet 3.7 managed tasks that took about an hour and a half. A year after that, Claude Opus 4.6 managed 12-hour tasks. The measure is about reliability on task baskets; it shows longer working horizons within that test.

Inside Anthropic, the central internal number is code. As of May 2026, the company says more than 80% of the code merged into its codebase was authored by Claude. Before Claude Code launched in research preview in February 2025, that number was in the low single digits. Anthropic also says that in the second quarter of 2026, the typical engineer was merging eight times as much code per day as in 2024. The report adds a caveat: lines of code measure quantity, and the eightfold figure almost certainly overstates the true productivity gain. Even so, Claude is now doing enough of the writing that human engineers are increasingly directing and reviewing work while Claude produces a large share of the code.

The boundary around today’s capability is judgment. Anthropic says Claude can be handed an underspecified engineering problem and find a way to solve it. In research, it can match or outperform skilled humans at executing a well-specified experiment. The gap is choosing goals, deciding which experiments deserve attention, and knowing when a direction should be abandoned. That gap separates today’s systems from one that could autonomously design its own successor.

Anthropic’s concern is that the gap may narrow while the development cycle is already accelerating. If AI systems handle more of the execution, each engineer or researcher can steer far more work than before. If the systems also improve at choosing useful research directions, the process could become increasingly self-reinforcing. Anthropic does not claim recursive self-improvement is inevitable. It says the possibility could arrive sooner than institutions are prepared for.

The proposed answer is coordination. Anthropic says it would likely be beneficial to have the option to slow or temporarily pause frontier AI development, so societal structures and alignment research can keep pace. The company also says a slowdown could make the world less safe if it merely allows less cautious actors to catch up. Its position is conditional: Anthropic says it would expect to slow or temporarily pause if other frontier developers did the same in a verifiable way.

Verification is the hard condition. A meaningful pause would require multiple well-resourced labs, in multiple countries, agreeing to the same terms and being able to confirm that others have actually stopped or slowed. Without that, a unilateral pause changes who is ahead and leaves out the wider deliberation Anthropic says is missing.

The report is restrained and serious. One of the major AI labs is saying that AI is already speeding up AI development, future systems may eventually help build their own successors, and a credible slowdown would be beneficial if it could be coordinated. It says the window to investigate these questions is open, and that policymakers, researchers, civil society, and other AI companies need to be involved.


Sources

Anthropic, When AI builds itself — anthropic.com