Anthropic Calls for Global Pause in AI Development, Warning of "Self-Improvement" Risk

Photo: AP Photo / Patrick Sison, File

As The Wall Street Journal writes, a few days ago, Anthropic co-founder Jack Clark and Marina Favaro, head of the company's internal research institute, published a post on the company blog calling on leading AI laboratories to consider slowing down the pace of development.

The authors argue that AI systems are progressing so rapidly that they may soon gain the ability for "recursive self-improvement" — a state where AI systems can improve themselves independently. Some experts consider this threshold a signal of potential danger and a source of large-scale risks to society.

The authors emphasize that such an ability has not yet emerged and its occurrence is not guaranteed. However, in their opinion, this could happen much sooner than most governmental and public institutions have time to prepare.

As noted in the post, it would be beneficial for humanity to be able to "slow down or temporarily pause" the development of advanced AI so that societal structures and AI safety research can keep pace with technological progress.

To this end, the company proposes developing a global agreement on a possible slowdown in development and creating a verification mechanism to ensure that competitors adhere to these agreements. This agreement is compared to nuclear arms control treaties.

However, as the WSJ writes, this will be much harder to implement: AI training centers are much easier to hide than missile silos, and anyone who refuses to pause could instantly seize global leadership.

How Sincere is the Statement?

Not everyone perceives these warnings as sincere. Venture investor and informal advisor to President Trump, David Sacks, sees Anthropic's actions as a manifestation of "regulatory capture." In his view, the policies advocated by the company could lead to a ban on open-source models — versions of AI systems that are significantly cheaper for organizations to use and develop internally.

Others suggest that Anthropic's warnings about the dangerous potential of its own tools might be a marketing ploy. As an example, they cite the company's decision to restrict access to the Mythos model, capable of finding bugs and vulnerabilities. In their opinion, Anthropic simultaneously demonstrates the capabilities of its products and creates additional attention around them.

At the same time, as the WSJ notes, there is no consensus within the industry itself regarding how close modern systems are to the level of AGI — general artificial intelligence comparable to human intelligence.

Professor Ethan Mollick of the Wharton School emphasizes that Anthropic is a complex blend of business interests and sincere "believers" in the need to take potential risks seriously. Meanwhile, AI pioneer Yann LeCun is skeptical of such fears, comparing the intelligence of modern models to that of a cat and stating that they will never equal humans.

Nevertheless, Anthropic's leadership, including CEO Dario Amodei, continues to paint alarming scenarios. As the WSJ writes, Amodei warns of the possible disappearance of up to 50% of entry-level office jobs and even the risk of dangerous behavior by future powerful AI systems. In one of his essays, he allowed for the possibility that training artificial intelligence on science fiction plots about machine uprisings could theoretically influence the behavior of real systems.

Separately, the WSJ mentions the words of Anthropic co-founder Jack Clark, who stated that technologies capable of recursive self-improvement could emerge within the next two years or even sooner.

In his opinion, today the world finds itself in a situation where powerful artificial intelligence systems are being developed at an unprecedented speed in various countries and companies, and commercial and geopolitical rivalry often overshadows broader questions about the consequences of these technologies for humanity.

As a result, Anthropic wants to develop practical mechanisms for implementing such agreements. Research into these issues will also be conducted by the Anthropic Institute, established by the company — an internal research center that plans to collaborate with other organizations to study the impact of artificial intelligence on society and possible control mechanisms.

In addition, the company plans to initiate dialogue with politicians, researchers, and other stakeholders in the coming months.

• 1
• 1
• 1
• 2
• 0
• 0
• 0
• 0