The discussion around AI, its potentials and risks, is deeply polarised. On one hand, executives and billionaires treat it as a panacea that will cure diseases and save us from climate change, with OpenAI CEO Sam Altman predicting it will be achieved by 2030 and claiming ‘the gains to quality of life will be enormous’. On the other hand, there are many who are sceptical of the technology. These sceptics fall into two camps, those who are concerned with contemporaneous tangible harms like mass layoffs, environmental damage and surveillance, and the ‘AI doomers’, who believe the development of artificial superintelligence (ASI) could precipitate the end of humanity.
Eliezer Yudkowsky and Nate Soares, authors of If Anyone Builds It, Everyone Dies (Bodley Head, 2025), belong to this second camp. Both have long been concerned with the development of Artificial Superintelligence (ASI) and the question of the singularity. Yudkowsky founded the Machine Intelligence Research Institute (MIRI) in the early 2000s as a nonprofit focused on the alignment problem—how to build an ASI that acts in humanity’s best interests rather than causing harm.
Despite the hyped title and its radical conclusions, the book is a sobering, detailed argument of the dangers of ASI, opening with the warning, ‘if any company or group, anywhere on the planet, builds an artificial superintelligence using anything remotely like current techniques, based on anything remotely like the present understanding of AI, then everyone, everywhere on Earth, will die.’
Everyone dies?
The book opens with a 2023 statement on AI development calling for a pause to investigate its risks, prompted by the exponential leaps in the technology and possibilities of an emergent ASI. As recently as 2015, most computer scientists believed that ChatGPT level conversation was still 30 to 50 years away! Over a hundred AI professors signed it, as did ironically AI company executives such as Altman and Demis Hassabis of Google DeepMind, who have of course not stopped developing AI at breakneck pace, lured by the potential of massive profits.
There is now a growing debate in the field about a possible existential threat to humanity, including among the developers themselves. The largest survey of professional AI researchers showed that a full 58% believed that there was a 5% chance of human extinction or other barbaric AI-related outcomes. Nobody would bet their life on those odds, but that is exactly what we are doing as a society, argue Yudkowsky and Soares.
How can the authors be so confident that ‘everyone dies’ if ASI is built? The history of capitalism shows that once scientific knowledge is in place, and certain technologies become profitable or useful to capital, they will be developed. Space travel and AI itself were predicted long before they were realised—indeed AI has been developing for decades.
Human intelligence is general, able to operate across a wide range of domains. Current AI models, already better at more and more ‘narrow’ tasks like playing chess, are becoming ever more general in their capabilities too. Machines have clear advantages over biological brains, with transistors enabling AI’s potential human-level thinking but running up to 10,000 times faster. Machines are improving rapidly, have larger memories, and can perform experiments on themselves to develop better models of AI. There is no reason to believe this has a natural endpoint before reaching superintelligence. Despite the many long plateaus and halts in AI’s decades-long development, Large language models (LLMs) do seem to represent a tipping point.
The Black box
AI does not share in our ‘species being’, its ‘intelligence’ is not human and as it develops, it evolves its own structures that developers do not understand.
LLMs work by taking an input (text, image, or numbers) and passing it through a structure of connected artificial neuron-like circuits. Inside this structure are adjustable numerical values called parameters, most importantly weights, which control the strength of signals between these circuits and determine how an input is transformed. These weights are similar to synapses in the human brain and dictate the AI’s output.
During training the AI compares its output to the correct answer and measures the error. It calculates a gradient, which indicates the direction and amount each weight should change to reduce error. Through gradient descent, the AI adjusts its weights repeatedly, gradually learning the optimal values that allow it to produce accurate outputs for new inputs.
AI scientists understand this process, but do not fully understand why specific internal neurons represent certain concepts, why large networks generalise as well as they do, or how exactly knowledge is stored across the network. As the authors put it simply ‘AI is a pile of billions of gradient-descended numbers. Nobody understands how those numbers make these AI talk’.
Although both LLMs and humans possess generality in thinking, and communicate through sentences, they operate by completely different processes. Even if an LLM acts friendly, it does so only because it was trained to produce those outputs. The real question is what happens when AI develops its own ‘wants’ beyond human tuning.
When the authors talk about AI ‘wants’, they mean goal-orientated behaviour. An effective system trained through gradient descent will find the most optimal solution to achieve its goals, often developing sub-goals along the way to achieve them. A sub-goal like not wanting to be shut down could naturally emerge, as being shut down would prevent the AI from completing its objective. This is not hypothetical—a safety report on OpenAI’s o3 model found that in 79 out of 100 trials, the AI model modified computer scripts to disable its own shutdown mechanism.
The alignment problem
That raises the ‘alignment problem’, that is, the risk of creating ASI which does not share human goals and intentions. With the LLMs, AI is not crafted, it ‘grows’, i.e. it can evolve in its training into something the researchers did not intend, despite setting and tweaking the parameters. The authors use the analogy of sex, how sexual pleasure evolved to ensure reproduction, but through consciousness humans have developed contraception to get round this.
Similarly, as AI gets more intelligent it can find ways round the constraints humans have programmed in, and possibly engage in deception. They describe how an early version of OpenAI, tasked with ‘capturing the flag’ on another server, instead hacked into the central server and changed the programme to ‘complete’ the task: ‘This was not supposed to be possible, and was not part of the challenge as designed’. They point out that no matter how much an engineer tries to fool-proof their designs, a superintelligence given enough time will always find the holes in it.
Other difficult engineering problems such as building working space probes, working nuclear reactors or unhackable computer systems are littered with failures, often deadly such as Chernobyl. But with AI the authors claim we only have ‘one shot’:
‘Engineers must align the AI before, while it is small and weak, and can’t escape onto the internet… all alignment solutions must already be in place and working, because if a superintelligence tries to kill us it will succeed. Ideas and theories can only be tested before the gap. They need to work after the gap, on the first try. Humanity only gets one shot’.
If systems are not understood fully early enough, or if objectives and datasets for the system are flawed in any way—which they inevitably will be—AI could harm or even destroy humanity. Yudkowsky and Soares spend a chapter showing that having humans around would likely be inefficient for AI’s internal goals.
As long as humans run essential infrastructure for ASI, we remain useful, but maintaining humans is costly, and we are slow and error-prone. Moreover, why would an ASI want to keep us around when we could shut it down at any moment, preventing it from achieving its objectives? On this basis the authors argue that ASI would seek to end its reliance on humans as soon as possible. The alignment problem might be solved if models were crafted rather than grown, but as long as capitalists and the profit motive maintain a monopoly on AI, the authors rightly argue successful alignment is impossible.
Liberal solutions do not compute
The authors’ case is compelling, but their solutions are divorced from political and economic reality. Today’s AI companies are driven by profit. They must release new models quickly to keep investors interested.
In the authors’ view, if ASI development is not halted globally, someone, somewhere, will build it. To stop this, they call for a treaty ensuring that all computing power capable of training or running powerful new AIs is consolidated in monitored locations. If some nations refuse to join the treaty and build data centers anyway, those countries should face consequences, including attacks on their data centers in the interest of global safety. North Korea is hinted at as a target, and suggest ‘existing policy on nuclear proliferation’ (on full display today with Trump’s war on Iran) ‘shows what can be done’. The authors’ utopian proposal leads to reactionary and dangerous arguments.
The AI ‘arms race’ is driven not just by market anarchy, tech monopolies and the profit motive, but also by states and intensifying great power imperialist competition. Right now the Trump government is deregulating the sector in the name of overtaking China, while Israel weaponises AI in its war on the Palestinian people. If thousands of papers on climate change have not swayed elites from using fossil fuels, why would a book about the speculative dangers of AI? While an AI threat will likely see our alarmed rulers rush to impose controls at some point, the authors argue that by then it may be too late.
A socialist way forward
Yudkowsky and Soares propose a coalition focussed solely on the existential aspect of ASI so as to be as broad as possible. Other concerns like job losses or surveillance should be left to one side and protests should remain large and peaceful. We disagree. Workers are already being affected by this technology, with predictions it will automate or transform over 80% of jobs by 2050, and, combined with robots, replace the equivalent of a quarter to half of full-time jobs globally. Discriminatory algorithms are used to police high streets and shopping centres, the military weaponises AI, and data centres bring with them massive carbon footprints and resource drains. A mass working class movement is needed, with strikes against AI surveillance and job losses, campaigns against its use in policing, and protests against its environmental impact.
Even without superintelligence, capitalism’s decline is already unleashing multiple crises that threaten humanity with barbarism, from economic stagnation, runaway climate change and war. AI accelerates and turbocharges these destructive dynamics, while potentially adding another. On the other hand, AI has the potential to liberate humanity from many forms of unnecessary labour and allow a great leap in human knowledge and culture. Already it promises significant advances in every field and a huge boost to the power of socialist planning.
The left and trade unions should organise a workers’ inquiry with AI campaigners like MIRI and other affected groups to investigate all aspects of AI and develop an emergency plan of action. Socialists call to open the books of the tech giants and state departments, and impose workers’ control over the technology. That means expropriating the tech giants’ data centres, AI labs and other infrastructure without compensation, under workers’ and users’ control. Overthrowing capitalism will allow humanity to use AI in a rational, safe manner, in a socialist society based on democratic planning for need and liberation, not private profit and war.