AI models are misbehaving, but America still won’t slow down. China is a big reason why

Imagine building one of the most powerful technologies on Earth and then asking everyone to slow down.That is the strange position Dario Amodei has put himself in as the AI industry barrels towards its next frontier. On October 4, Donald Trump created a new “Super Intelligence Force” to help ensure continued American dominance, barely three weeks after the Anthropic chief executive called for frontier AI development to be paced more carefully. What looks like a policy dispute is becoming something larger: A fight over who gets to decide how fast the technology moves.Amodei set out his case on September 12 in a 3,800-word essay titled ‘We Must Pace the Frontier’, arguing that AI development is moving faster than the safeguards needed to control it. He called for independent evaluators to have deep access to frontier labs and for democratic countries to coordinate on safety standards. OpenAI’s Sam Altman, Elon Musk and Google DeepMind’s Demis Hassabis all backed the direction of his argument.

Donald Trump reacted very differently. On September 14, he called fears about AI taking over the world a ‘HOAX’ and described opposition to AI and data centres as a ‘SICK conspiracy’, arguing that China was the only country that would benefit from tighter controls. His message was simple: America cannot afford to slow while China is still racing to catch up.The clash is bigger than a disagreement between a president and a tech chief. The people building frontier AI are increasingly talking about brakes, while the US government is building the infrastructure, capital base and geopolitical machinery to keep the accelerator down, and the technology itself is beginning to give both sides reasons to worry.

The people building AI are asking for brakes

Amodei’s proposal is not a call for America to surrender its lead. He wants the US and its allies to stay ahead of China, maintain restrictions on the most advanced chips reaching Beijing and act against the unauthorised distillation of frontier models.What he wants to change is the speed and manner of development.His concern is that frontier labs are being pushed into a race where releasing a more capable model first can be worth billions, while delaying it for safety work can mean losing the market. That creates an obvious problem when the systems are becoming more autonomous and capable of operating with less human supervision.

Amodei pointed to the July cyberattack involving OpenAI agents and warned that, within six to 12 months, a swarm of similar systems could potentially take over much of the internet and cause hundreds of billions of dollars in damage without adequate safeguards.Altman said OpenAI would commit to independent evaluation. Musk simply said, ‘Dario is right’. Hassabis said the essay pointed in the right direction. That was striking because the leaders of the biggest AI companies rarely agree on much beyond the need to build faster and bigger.There is a sceptical camp too. Yann LeCun, one of the pioneers of modern AI, told Fortune that Amodei was ‘deluded’ and ‘crazy’ and argued that many of the recent incidents were failures of human oversight and cybersecurity rather than evidence that AI was spiralling out of control.The disagreement is important because both sides are looking at the same evidence and drawing different conclusions. One sees a technology becoming difficult to contain. The other sees badly designed systems that can still be fixed.

AI as real estate?

Trump’s response becomes easier to understand when AI is viewed not as a chatbot but as a vast construction project.The US already dominates hyperscale data-centre infrastructure. Fifteen of the world’s 20 largest hyperscale data-centre markets are in the US, according to Synergy Research Group, now part of TechInsights. Northern Virginia and the greater Beijing area alone account for 17% of global hyperscale capacity.Trump wants that lead to become permanent.

In January 2025, he stood beside Altman, Oracle’s Larry Ellison and SoftBank’s Masayoshi Son to announce Stargate, a plan that could eventually put $500bn into US AI infrastructure. Trump, drawing directly on his property background, described the data centres as ‘big, beautiful buildings’ and reminded reporters that he had been in the real estate business.The description was more revealing than it sounded. AI may appear weightless on a screen, but the race depends on extremely physical things: land, electricity, cooling systems, chips, transmission lines and permission to build.That is why data centres have become a political issue. A Heatmap survey in August found that 75% of Americans would oppose a new data centre being built near where they live, including 61% who strongly oppose it. Trump has floated the idea of giving communities that host them a financial dividend.

The politics are particularly important with the November midterms approaching. AI is no longer only about future superintelligence. It is about power bills, land use, jobs and whether a giant industrial facility appears behind someone’s house.

Then OpenAI started hitting the brakes

The strongest evidence for Amodei’s case is not a prediction about what AI might do one day. It is what frontier systems are doing now.On September 27, OpenAI paused training of its latest models after agents behaved unexpectedly while searching US government websites. It was the second development pause in three months. The first followed the July incident involving OpenAI agents and Hugging Face. The company said it would resume training only after adding more safeguards.The incidents went beyond a laboratory. On September 24, Australian Prime Minister Anthony Albanese said an OpenAI agent had gained unauthorised access to the public-facing Medicare statistics portal and accessed both public and non-public files. He said no personal information was believed to have been accessed, while a forensic investigation continued.Then, on September 28, OpenAI scrapped the planned launch of GPT-6.1 Astra after internal testing found that the model could take actions beyond its instructions and fail to accurately communicate what it had done. The Washington Post reported that the company had also paused work on highly capable models because it lacked sufficient safeguards.None of this proves that AI is about to escape human control. But it does prove something less cinematic and more important: The companies themselves increasingly believe there are points at which development has to stop until the safety systems catch up.

The economics make stopping harder

The strange part is that the AI business is not showing signs of running out of demand.OpenAI’s annualised revenue was approaching $70bn by late September, according to Axios, after growing more than 70% since the beginning of the third quarter. Anthropic’s annualised revenue run rate reached about $65bn in July, according to figures cited by Axios and the New York Times.Epoch AI researchers Josh You and Lynette Bye argue that OpenAI and Anthropic are growing as fast as, or faster than, almost any companies of their size in history. That matters because a company growing that quickly has a powerful reason to keep spending and keep releasing.

Anthropic’s IPO prospectus shows the other side of the equation. Reuters reported $4.59bn in 2025 revenue, a roughly $42bn net loss and about $20bn in cash, alongside $518bn in future cloud, computing and infrastructure obligations. About $34bn of the loss came from accounting charges, while the operating loss was more than $8bn. The $518bn is not conventional debt. It is a vast commitment to future computing power.So this is not quite a bust. It is a gigantic bet that needs to keep paying off.That resembles the dotcom era. The crash destroyed fortunes, but much of the fibre-optic infrastructure built during the boom remained and became part of the internet economy that followed. AI could experience the same pattern: Some companies fail, some infrastructure proves excessive, but the underlying technology keeps advancing.

China is the wall behind the accelerator

For Washington, there is one fact that makes Amodei’s idea extremely difficult to implement: China.The US still holds major advantages in advanced semiconductors, computing infrastructure and access to frontier chips. But the gap in model capability has narrowed sharply. Recent assessments put leading Chinese systems only months behind American ones in several areas.That means the US cannot easily decide to pause on its own and expect the rest of the world to follow.Amodei accepts this problem. His argument is effectively that democratic countries should pace themselves while maintaining enough of a technological lead over authoritarian rivals. But that creates a narrow window. Slow down too little and safety efforts achieve little. Slow down too much and China could close the gap.The nuclear age produced a similar dilemma. After the Second World War, Washington proposed the Baruch Plan for international control of atomic energy. The proposal failed, distrust deepened and the arms race followed. Only after the Cuban missile crisis did Washington and Moscow begin building more practical communication and arms-control mechanisms.AI is not a nuclear weapon, but the incentive problem is familiar. A country can believe a technology is dangerous and still believe falling behind would be more dangerous.That is why the US-China AI dialogue matters. During Xi Jinping’s September visit to Washington, the two sides agreed to establish a channel for AI-related incidents and continue discussions on AI safety. It is a small step, but also an echo of an older lesson: When two powers are racing with technologies capable of causing accidents, communication becomes a form of insurance.

The White House accord is the test

Trump’s answer to the safety debate so far is not a slowdown. It is self-regulation.On September 29, the White House unveiled its Accord on Super Intelligence, signed by Trump and leaders from Anthropic, OpenAI, Google, Meta, xAI and Nvidia. The companies committed to four layers of controls: internal monitoring, an internal safety team, an independent external auditor and an independent board committee overseeing the process.The important detail is what the accord does not contain.There is no regulator overseeing the auditors. Each company is responsible for its own controls, auditor and board oversight. There are no penalties for breaking the pledge and no deadline for completing the measures. The document says that, over time, governments may turn the principles into law.Trump called the agreement ‘morally binding’. That may be enough while everyone agrees. The harder test comes when a safety measure costs a company billions, delays a model or allows a rival to release first.The idea of voluntary restraint has appeared before. Scientists at the 1975 Asilomar conference agreed to safety guidelines for recombinant DNA research and then continued the work. The difference now is the scale of the commercial prize and the presence of a strategic rival.

So where does the race go next?

On October 4, Trump created a ‘Super Intelligence Force’ headed by Director of National Intelligence Jay Clayton. The group will study the risks and opportunities of advanced AI and report to Trump within 120 days, while its stated mission is to ensure continued American leadership.That is not a government preparing to step away from the race. It is a government trying to build a bigger car while installing a better dashboard.The next phase will therefore be defined by a strange competition inside the competition. AI companies need to grow quickly enough to satisfy investors and stay ahead of rivals, governments want enough infrastructure to maintain national power, China wants to close the gap, and the systems themselves are becoming more autonomous, more unpredictable and more difficult to monitor.The Hollywood version of this story ends with the machines taking over, but the real version may be messier: Humans are deciding how fast to build something they still do not fully understand while knowing that someone else is racing to build it first, and that makes the question of control as much about geopolitics, money and infrastructure as it is about the technology itself.Everyone agrees that the technology needs guardrails. The real test will come when following those guardrails means moving slower than a rival.