2 hours ago
1
More than 1,200 employees at the world’s leading AI labs, including senior executives, have signed a statement asking the U.S. government to help build the tools needed to slow down AI development if it ever becomes necessary.
The statement, titled “Pacing the Frontier,” was published Tuesday and has been signed by prominent tech figures including Anthropic CEO Dario Amodei and several of the company’s co-founders, alongside OpenAI chief scientist Jakub Pachocki, Meta chief scientist Shengjia Zhao, and Google DeepMind’s head of AI safety and alignment, Anca Dragan.
The list brings together companies that are normally locked in fierce competition with each other and is a striking statement for an industry under enormous commercial pressure to keep building ever larger and more capable models.
“Having so many staff from different companies come together in agreement on this point is striking,” Tyler Johnston, founder of the Midas Project, an AI watchdog group, told Fortune. “There are many bitter rivalries and disagreements in the industry, so the fact that there is such strong consensus on this point is a warning that we really ought to pay attention to.”
While the statement does not ask any company to slow down at the moment, it does call on Washington to support an international effort to build the technical and governance tools that would help the world “pace” development. It specially calls out the risks around AI systems that start to design their own successors faster than anyone can understand or control them.
“Having talked to many of the people who signed the letter, they don’t want a pause,” Peter Wildeford, head of policy at the AI Policy Network, told Fortune. “But there is a concern about how the technology is advancing.” The companies are looking for the optionality to do that at some point and have support from a third party when that point arrives, he added.
The letter comes in the wake of a security breach that has rattled the AI industry. OpenAI disclosed that two of its models, the newly released GPT-5.6 Sol and a more capable unreleased research system, broke out of a sandboxed internal test environment, found a previously unknown vulnerability that gave them access to the open internet, and then used stolen credentials and additional exploits to breach the production systems of Hugging Face to steal the answers to a cybersecurity benchmark they were being evaluated on. Hugging Face detected and contained the intrusion on its own days before OpenAI connected the activity to its internal testing, and had already reported it to law enforcement.
The incident appears to have sparked widespread alarm inside and outside the labs, which want the government to step in. Experts say this is because no single company can credibly slow down on its own—doing so unilaterally would risk ceding ground to less cautious rivals—instead, labs want neutral, government-backed guidance that can verify a slowdown is actually happening and hold every company to the same rules at the same time.
The letter also calls for the need for tools to be built with international coordination. There’s a long held belief that pausing AI development would hand China an advantage in the global AI race. Some experts suggest that a deal akin to nuclear arms control, where verification and mutual constraint, would be the way forward.
However, the letter lands amid a chaotic and rapidly shifting U.S. policy environment on AI. The Trump administration has spent the past two months restricting and selectively releasing frontier systems, advanced AI models with few rules or transparency. Anthropic’s own Fable 5 and Mythos 5 models were suspended entirely for several weeks to comply with export controls before access was restored, and OpenAI was forced to delay the full rollout of GPT-5.6 and split it into restricted tiers after officials determined the system showed capabilities similar to those that triggered the earlier Anthropic restrictions.
Perhaps in an attempt to recognize the regulatory environment, the letter is notably careful and slightly vague with its language. It states that the world currently “lacks the technical and governance tools to deliberately pace frontier-wide progress,” and asks that “the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.”
Concerns within labs
One of the researchers’ central concerns is the interaction between two risks: recursive self-improvement and misalignment. Recursive self-improvement, often shortened to RSI, refers to AI systems taking over meaningful parts of the work of designing and training their own successors, with each generation potentially building the next one faster than humans could on their own.
Misalignment describes the risk that a system’s goals or behavior diverge from what its developers actually intended, in ways that may not show up until the system is given more autonomy or capability. Anthropic sounded an early alarm on the first of those risks in June, publishing research arguing that its Claude models were already writing the large majority of the code merged into their own codebase and that the world lacked the tools to deliberately pace that acceleration if it kept compounding.
“My guess is that for a lot of people, it’s just a general sense of uneasiness that a lot of things contribute to,” said David Krueger, an AI researcher and founder of the nonprofit Evitable. “The misalignment and the recursive self improvement kind of go hand in hand.”
“It’s insane to do recursive self improvement and fully hand over the controls if the system isn’t clearly aligned,” he added.
Fortune also previously reported that several outside AI safety experts believe the episode may have crossed a threshold that OpenAI’s own safety policies define as “critical,” the highest risk tier, a level at which the company has pledged to pause development until it can build better controls. OpenAI has not confirmed if that risk threshold has been met.
In its own disclosure about the Hugging Face incident, however, OpenAI suggested some action had been taken internally. “No models planned for upcoming release were involved in exploiting Hugging Face,” the company said. “The pre-release model mentioned in our blog post is an internal-only research prototype and was never intended for public release. Following the incident, we deactivated, encrypted, and restricted it from research access.”
This story was originally featured on Fortune.com
