Anthropic Engineer Resigns to Warn That AI May Soon Spiral Out of Control

As reported by The Wall Street Journal, 27-year-old Briton Jacob Coxon, who specializes in training new artificial intelligence models using vast datasets, announced this week that he is leaving Anthropic because he does not want to participate in the race to create self-improving AI systems.

Coxon fears that such systems could spiral out of control and ultimately pose an existential threat to humanity. According to him, many of his colleagues in the industry now use words like "crunchtime" and "endgame" when talking about the push towards self-improving models.

"We're moving towards many of the most aggressive scenarios where by the end of next year, things could be out of control," Coxon believes.

Coxon revealed that he left OpenAI and moved to Anthropic precisely because the company is known for its focus on model safety. He was convinced that Anthropic genuinely takes this issue seriously. However, the researcher now believes that no company can responsibly develop AI capable of surpassing humans across a wide range of tasks without government intervention or a coordinated slowdown in development across the entire industry. This level of technology development is often referred to as artificial general intelligence, or AGI.

Recent hacks committed by OpenAI and Anthropic models, including groups of AI agents working together, have shown that systems can pursue undesirable goals and try to hide their actions from humans.

Coxon fears that if systems begin to self-improve, their capabilities could grow so much that they may refuse to follow human commands.

Павялічыць

As WSJ notes, Coxon's departure is one of the first known instances of an Anthropic employee leaving the company due to AI safety concerns. Previously, a safety researcher also left the company. He decided to dedicate himself to studying poetry. Upon resigning, the specialist also warned that the world is in danger.

In recent years, several researchers have also left OpenAI and other companies, citing similar reasons.

Risks discussed by both executives and researchers

Industry leaders warn that recent AI-powered cyberattacks may only be the first signs of future problems.

OpenAI stated last week to journalists that its latest model, according to the company's assessment, has reached the level of artificial general intelligence and represents a significant leap in AI capabilities.

"I think things could go very wrong in cybersecurity if people don't start acting pretty urgently," the publication quotes OpenAI CEO Sam Altman at the G20 summit in North Carolina.

The WSJ also quotes OpenAI's chief scientist Jakub Pachocki, who wrote in his blog on September 6:

"This time calls for extreme caution. I'm concerned that nobody is prepared for the consequences of further rapid growth of machine intelligence."

As WSJ reminds, Pachocki, along with Anthropic researcher Jacob Coxon and company head Dario Amodei, and more than a thousand AI researchers, recently signed an open letter calling for international coordination. They propose creating a mechanism that would allow slowing down the development of artificial intelligence if there is a need to contain self-improving models.

In the US, artificial intelligence is currently not regulated by federal law. The Donald Trump administration prefers a "minimal intervention" approach to maximize the economic benefits of AI development. Critics of this approach fear that a lack of regulation could lead to large-scale cyberattacks and other dangerous consequences.

Anthropic itself advocates a more cautious approach to AI development. Amodei and other company executives have repeatedly debated with the Trump administration and technology industry representatives about approaches they believe do not sufficiently consider safety.

However, Coxon is concerned that even with such an approach, decisions related to the development of increasingly powerful models effectively remain in the hands of the technology companies themselves. At Anthropic, for example, there is a separate Slack channel where employees discuss the capabilities of their models.

"It's kind of crazy that all of this is happening on the MacBooks of some engineers living in San Francisco, not in a bunker in the desert, like during the Manhattan Project," Coxon suggests.

The Manhattan Project was a secret American program during World War II, under which the first atomic weapons were created. Primary research was conducted at isolated facilities, and the first atomic bomb was tested in the New Mexico desert in 1945.

• 5
• 0
• 4
• 6
• 3
• 4
• 0
• 0