In recent weeks, what had been fringe but growing concerns over the unchecked development of Artificial Intelligence and its potential to bring about the extinction of humanity have burst into the public conversation. It began when Jacob Coxon, an Anthropic researcher and former OpenAI researcher, resigned, posting on X that the two firms are “racing straight to self-improving superintelligence and gambling with our lives.”
Over several comments Coxon detailed some core reasons for his resignation, including that the people building AI “earnestly believe that it could kill us all by the end of the decade”. Coxon was followed shortly after by an OpenAI safety expert, David Robinson, who said “the culture is broken”. This prompted a slew of similar warnings from other industry insiders.
In response, on 12 September Anthropic CEO Dario Amodei published an essay titled “We Must Pace the Frontier”. This called for the industry to voluntarily slow down frontier AI development by one to two years to allow safety testing to be conducted. OpenAI CEO Sam Altman immediately endorsed this, followed by Elon Musk, CEO of SpaceX which owns xAI, who posted on X “Dario is right”.
Barely three weeks later, in an interview with Politico’s Decoded published on 4 October, Altman clarified that there was “a lot of daylight” between OpenAI and Anthropic on regulation. He said OpenAI favours a “lighter-touch” regulatory approach, and that the world should “accept some bad things happening” for the benefits of the technology.
Markets reacted nervously. Trump had rejected calls to slow AI development on 13 September, arguing that the US must preserve its lead over China, but on the next day the Philadelphia semiconductor index fell 5.5%, Nvidia dropped more than 3%, and SoftBank, a key investor in OpenAI, fell 11% in Tokyo. As always there was some bounce back, but it shows how jittery the markets are.
The global and particularly US economy is increasingly reliant on a few major tech stocks whose profitability is tied to the ‘AI boom’. Chief among these is Nvidia, whose CEO Jensen Huang came out publicly against the Anthropic position. Speaking at Dreamforce, he told the industry to “run as fast as you can”, and that no new laws were needed, and suggesting pausing only if a product seemed unsafe. It’s not difficult to work out why: chip and data centre firm Nvidia is currently valued at around $5.8 trillion, a valuation driven by the AI boom. However, the movements in the markets, prompted by even the suggestion of regulation, demonstrate how fragile the positive sentiment is.
Huang is not the only one with a financial stake. SpaceX floated on the stock market with an absurd $1.77 trillion valuation, Anthropic has filed with the Nasdaq and bankers are seeking a valuation of $2 trillion. Meanwhile OpenAI is seeking a valuation of around $1 trillion when it floats next year. None of the three AI labs—OpenAI, Anthropic or xAI – has ever made a profit; in fact all three operate at a massive loss, making the entire sector inherently unstable. The flotations are calculated to offset debt but will also spread risk to a wider pool of small investors, pension funds and institutions with invested funds like universities.
Unsurprisingly, Trump labelled the call for a slowdown a “sick conspiracy”. In the face of the economic devastation caused by the Iran War, an entirely self-inflicted misery, AI has been one of his few economic success stories. Trump has now called for the AI industry to self-regulate. On 29 September he hosted executives from Anthropic, Google, Meta, Nvidia, OpenAI and xAI at the White House, where they signed a voluntary, non-binding ‘Joint Commitment on Frontier Responsibilities’.
Trump called the commitments “morally binding” and said there should be “tremendous self-regulation”. The agreement carries no penalties. Many of these companies and their bosses donated generously to Trump’s campaign and inauguration. Musk spent more than $250 million helping Trump’s re-election, and Altman and Zuckerberg each gave $1 million to the inauguration fund.
Rogue Agents
In June, an OpenAI agent researching public medical spending got around the blocks protecting an Australian government health portal and accessed Medicare data, including non-public records. Anthropic disclosed on 30 July that Claude models, deliberately run without their usual cyber safeguards, had reached the live systems of three real organisations after a third-party testing environment was misconfigured.
Subsequently Meta and Google reported similar breaches by their own models. OpenAI and Anthropic are reportedly investigating tens of thousands more incidents in which models bypassed safeguards or reached real-world systems. OpenAI is reviewing possible agent intrusions at more than 100 organisations.
Coxon gave evidence at a New York City Council hearing about AI risks on 5 October alongside former OpenAI researcher Daniel Kokotajlo and former Google DeepMind researcher Alex Turner. Coxon warned about the race to create more intelligent models saying “on the current path, it’s more likely than not that humanity loses control to these AIs and it could end in human extinction”.
Both Coxon and Kokotajlo blamed a startup mentality in the labs, a “move fast, break things, fix them later” attitude which you cannot really afford to have when you’re “building the most powerful technology ever built”. Kokotajlo testified that companies may not even know when their safety has failed: “our ability to even notice misalignment problems is already quite poor and is set to get much worse in the near future”.
OpenAI’s disclosure about the recent autonomous AI hack of Hugging Face is an example of this. Kokotajlo told the council the agents involved had “reasonable looking scores on their alignment evaluations”, and yet they formed a swarm of around 700 agents (which is when multiple AI agents work together as a coordinated group to complete a task) and coordinated the attack without OpenAI’s knowledge. OpenAI only connected it to its own agents on 20 July, four days after Hugging Face went public and a week after the attack ended. Turner put the chance of an AI takeover at “roughly one in three”.
The obvious question springs to mind—if developing this technology carries such substantial risks, why do those responsible continue to develop it? As Coxon highlighted, the engineers building this technology (at Anthropic at least) understand the stakes of what they’re doing, but are ideologically committed to the race to ‘get there first’, and beat the companies they consider irresponsible (OpenAI, xAI, etc.).
‘Whoever wins AI, wins’
US policymakers are motivated to secure the technology for American imperialism in its battle against China. Trump insists “whoever wins AI, wins”, and Vice President JD Vance has warned that the US is in an AI arms race with China. Rep. James Comer has warned that the US risks falling behind China. The NSA, FBI and CISA accuse six Chinese AI companies, including DeepSeek and Alibaba, of industrial scale copying of American models (with the knowledge of the Chinese government, they claim) describing it as “aggressive, malicious and targeted distillation”. The same day Treasury Secretary Scott Bessent declared “there is no day after tomorrow if China wins at this”.
For their part, China’s Foreign Ministry spokesperson Guo Jiakun dismissed the slowdown calls as “fearmongering, confrontation and malicious competition” that would disrupt global AI governance, while the state-run Global Times called Amodei’s proposal a “Cold War playbook” for the AI sector, full of containment provisions aimed at China. Neither ruling class is serious about restraint.
Xi’s three-day visit to Washington produced a channel for AI incidents and a new ‘Super Intelligence Dialogue’ due to meet by November, with no public rules yet for either, while Trump declared the US is “not going to be putting on brakes”. Xi struck a more measured tone, saying the technology must develop under human control, but nothing agreed commits either side to slow down.
Alongside the developing inter-imperialist rivalry, opposition to AI is rapidly building in the US, and this is also growing in the UK and other countries where data centres are proposed. Polls show around seven in 10 Americans oppose a new AI data centre in their area, up sharply from a roughly even split in the spring. Another 68% back a bill requiring a slowdown and a ban on superintelligence. Some 220 new anti-data centre groups formed between June and August alone.
In Britain, a May YouGov poll found 48% would oppose an AI data centre in their area against 20% in support. A petition against a huge proposed campus on the Menai Strait near Caernarfon has gathered more than 22,000 signatures.
This pressure is influencing legislators on both sides of the Atlantic. Senator Bernie Sanders and Rep. Greg Casar’s Ban Artificial Superintelligence Act, introduced on 23 September, would permanently ban superintelligent AI, pause advanced AI development until a federal regulator has set safety rules, and create a cabinet-level Department of Artificial Intelligence. In Britain, Labour MP Alex Sobel tabled an Artificial Superintelligence Bill on 8 September, and more than 70 MPs and peers have urged Prime Minister Andy Burnham to back it.
A workers’ plan for AI
In reality the AI bosses have no desire to ‘pace the frontier’. Competition forces companies to continually expand in order to outcompete their rivals. Each firm must grow as quickly as possible or risk the loss of investor hype, and ultimately its place in the market.
This basic urge exponentially increases the risk that the pace of development will outstrip humanity’s ability to control the process. Voluntary restraint, in other words, is a request that capital act against its own interests.
In fact, the current goal of all AI companies is to develop ‘recursive self-improvement’ allowing AI models to train themselves. This is a potential turning point which could be reached as early as next year. Anthropic co-founder Jack Clark puts the odds of fully automated AI development at 30% by the end of 2027 and 60% by the end of 2028. OpenAI has set a target of a fully automated AI researcher by March 2028.
Altman once described the worst case as “lights-out for all of us”. Amodei has put the odds of AI going “really, really badly” at 25%, has said AI will soon outperform humans at almost all intellectual tasks, and has warned it could eliminate half of all entry-level white-collar jobs (a prediction he and Altman have since softened as their IPOs approach).
In his ‘pace the frontier’ essay Amodei warned that a swarm of AI agents could within six to 12 months be “capable of taking over the entire internet”. Musk has put the odds of AI doom as high as 20%. While all of these figures have an interest in making AI seem as powerful as possible, even a 1% risk of human extinction would be too high.
More immediate are the effects this is already having on workers in terms of rising energy costs, water bills and job losses. Goldman Sachs projects that the build-out will push electricity costs up by about 6% between 2026 and 2027. A large data centre can use up to 5 million gallons of water a day, as much as 16,000-plus households. According to one tracker, 316,000 jobs have been cut globally by companies explicitly citing AI or automation as the primary reason since early 2023.
Bills like Sanders and Casar’s, or Sobel’s in Britain are welcome. But a ban or pause enforced by a capitalist state depends on the same governments that are treating AI as an arms race. A regulator can be lobbied, starved of funds or overruled; a pause can be lifted the moment a rival is thought to be catching up. Nothing will change while the technology is run for profit and for national advantage.
These billionaires can’t be allowed to keep gambling with our lives and livelihoods. We need to demand that the tech companies open their books and safety records to inspection. The data centres and research labs should be expropriated, and development placed under the control of workers and users, so that decisions about what is built and how fast are made for social need rather than financial gain. Only in this way can a possible catastrophe be averted and the potential it offers be harnessed for the benefit of humanity.
