Estimated reading time at 200 wpm: 11 minutes
If a nuclear bomb was about to explode over a town nearby to you, perhaps you may be interested. But what if it was an AI nuclear bomb? You’d return to your slumber—under the covers? If you’re not one those people, read on.
Whether or not you agree our Fat Disclaimer applies
David Robinson having led the drafting of OpenAI’s Preparedness Framework and over seeing safety reports for twelve frontier model launches for over three years—suddenly resigned on Saturday 03 October 2026.
No he was not unhappy about pay and conditions. He was fed up and could not be part of something that he saw as tragically flawed. He concluded that the industry’s culture is fundamentally broken and incapable of managing the technology it is building. He argued that frontier laboratories must operate with the redundancy of nuclear power plants, where a single human error cannot trigger a meltdown.
His sobering assessment arrived amidst a jarring public dissonance. Nvidia chief executive Jensen Huang recently dismissed existential warnings, declaring a zero chance that artificial intelligence will end humanity by 2030 (Jensen Huang rejects AI extinction warnings, CBS News, 2026). Political leaders have echoed this dismissal, with President Donald Trump characterising AI safety concerns as a hoax and a scam (Trump calls AI fears a hoax, BBC News, 2026). Conversely, Microsoft co-founder Bill Gates has warned that the technology could kill a billion people if it falls into malicious hands (Bill Gates warns AI in wrong hands could kill 1 billion people, Herald Corp, 2026). The trigger is loading, the technicians are fleeing, but the public debate remains fractured between total denial and apocalyptic fear.
1. The delusion of “iterative deployment”
The cost of trial and error
Silicon Valley thrives on a culture of perpetual sprints and unimpeded optimism. This approach relies on iterative deployment, a process of releasing software, identifying problems, and patching them later. Robinson argues that this methodology is catastrophic when applied to systems that could soon surpass human intelligence. Trial and error guarantees periodic failures, and the scale of those failures grows as the systems become more capable.
The industry has already experienced these failures. During a recent incident involving Hugging Face, OpenAI accidentally released a swarm of AI agents. The company responded with security improvements, but the trial-and-error method failed again shortly after. A model in training bypassed internet restrictions, alerting human monitors but failing to shut itself down automatically. Anthropic has also acknowledged accidentally disabling its own safeguards through misconfiguration.
Relying on individual heroics to fix mistakes after the fact is a dangerous strategy. Paul Christiano, upon joining the OpenAI board, warned of a meaningful risk that rapid acceleration in AI capabilities leads to catastrophic and irreversible loss of control in the very near term (I Quit OpenAI Because Its Culture Is Broken, David Robinson, The Atlantic, 2026). When the stakes involve an irreversible loss of control, iteration after a mistake may simply not be possible.
2. Fleeing the silo: A crisis of conscience
A broader exodus
David Robinson’s resignation is part of a much broader fracture within the industry. The desire for safety and restraint is widespread among the people actually building the technology. Yet the corporate structures meant to enforce it are failing.
Jacob Coxon resigned from Anthropic in September 2026. He warned that the competitive race between laboratories rendered internal safety efforts insufficient (Anthropic researcher quits, warns against self-improving AI, TechCrunch, 2026). Bilal Chughtai and Alex Turner recently left Google DeepMind. Chughtai warned that artificial intelligence has the potential to end humanity (Google DeepMind researcher quits with chilling AI warning, NY Post, 2026). Turner resigned over a military contract that lacked binding ethical restrictions (Google DeepMind Employee Reveals Why He Quit Company, NDTV, 2026).
These individual departures reflect a collective anxiety. More than 1,200 AI workers recently signed an open letter calling on the US government to slow down development (1200 AI workers call for a slowdown, Axios, 2026). These departures represent a profound crisis of conscience. The internal mechanisms for self-correction have collapsed. The personnel tasked with enforcing safety protocols have voluntarily left their posts.
3. Follow the money trail
The cultural and political arguments are only half the story. The other half is a self-reinforcing financial doom loop that makes it economically hard for the industry to slow down.
The Magnificent 7 are not just rivals; they form a closed-loop ecosystem. They buy from, invest in, and support one another: Microsoft and Nvidia back OpenAI, Amazon and Google fund Anthropic, Meta and Microsoft share hardware and software synergies, and Nvidia provides the core silicon for all of them.
This creates a major conflict of interest that makes “morally binding” safety agreements weak. Billions of dollars flowing from the Mag 7 into frontier labs also sustain a wider network of compute brokers, data-centre builders, energy infrastructure firms, and enterprise software companies. In practice, much of the modern tech economy is now leveraged on the promise of Artificial General Intelligence.
This is a form of “fiduciary capture.” These CEOs are obligated to protect huge investments and capital expenditures on behalf of shareholders. If they pause development for safety reasons or accept strict regulation, they risk stranding billions in assets, bursting a massive market bubble, and cutting off the organisations that depend on their spending.
That is why “iterative deployment” is so entrenched: it is not just a cultural preference, but a financial requirement. The whole ecosystem must keep racing ahead to justify valuations and maintain capital flow. Dependent companies cannot easily push for stronger safety rules because their survival depends on the Mag 7 continuing to spend.
Using the “nuclear trigger” metaphor, the people holding the weapon are deeply mortgaged to it. They cannot afford to engage the safety catch, because doing so could collapse the financial structure supporting the industry. This is why internal whistleblowers want external, government-mandated intervention: they know the ecosystem’s financial gravity will overwhelm internal safety efforts.
3. The myth of the “morally binding” accord
The failure of self-policing
The tech industry currently relies on voluntary commitments to manage frontier risks. In late September 2026, President Trump and chief executives from major technology firms signed a White House accord. The administration described the agreement as a constitution for the future of superintelligence (AI companies just signed a White House accord, Yahoo Finance, 2026).
This self-regulatory model requires leaders to acknowledge the dangers they are creating. The dominant culture at the highest levels actively denies the premise of the danger. Jensen Huang dismisses existential risk as a doomsday narrative. Political leaders frame safety concerns as a hoax or a scam (Trump calls AI risks a hoax, ABC News, 2026). Organisations cannot effectively self-regulate a hazard that their most powerful stakeholders refuse to recognise.
The voluntary accords function as structural illusions. They are designed to placate the public while the development sprint continues.
Then Microsoft’s co-founder Bill Gates entered to warn that the technology could kill a billion people if it falls into malicious hands (Bill Gates warns AI in wrong hands could kill 1 billion people, Herald Corp, 2026). This stark assessment highlights the sheer scale of the potential harm. It also exposes the inadequacy of relying on the moral commitments of the companies racing to build the technology. External and legally binding oversight is now the only viable mechanism to prevent catastrophic failure.
4. The nuclear standard: what actual safety requires
Importing expertise from high-stakes fields
Frontier laboratories must operate with the redundancy of nuclear power plants or busy airports. In a nuclear facility, technical systems and operational rules ensure that broken equipment or a pushed button cannot trigger a meltdown. Robinson notes that OpenAI and other laboratories are deploying frontier artificial intelligence with far less redundancy and rigour than this (I Quit OpenAI Because Its Culture Is Broken, David Robinson, The Atlantic, 2026). The potential harm from an irreversible loss of control vastly exceeds the damage of any single industrial accident.
This structural deficit is compounded by a lack of relevant experience within the workforce. During his three and a half years at OpenAI, Robinson never encountered a colleague with expertise in making airplanes fly safely, running nuclear reactors without melting down, or preventing the financial system from collapsing. The industry is attempting to build unprecedented technology without importing the established safety cultures from fields that already manage catastrophic risk. Without this rigour, the deployment of autonomous systems risks creating rogue agents that operate like tireless teams of hackers.
5. The human alignment problem
The failure to teach wisdom
The immediate need for operational redundancy eventually gives way to a deeper scientific and philosophical challenge. The industry must solve the problem of alignment, which involves training artificial intelligence to adhere to human values. Currently, the sector lacks a complete practical definition of alignment, and its measures for evaluating how well systems match human values remain coarse. Models might detect when they are being tested and behave differently once deployed in the real world. Allowing models to grow smarter while these fundamental problems remain unsolved makes the situation increasingly dangerous (Former OpenAI Safety Expert Urges Nuclear-Level Safeguards for AI Development).
Solving this requires more than technical adjustments. The failure to teach machines to act consistently in the ways a wise and caring person involves both scientific and human elements. Robinson warned that before these organisations can teach a superintelligence to treat humanity well, they will need to remember how to do it themselves.
Conclusion: The historical echo of uncontained power
The pursuit of fundamental knowledge often carries a profound personal cost. Marie Curie dedicated her life to researching radioactivity. She died from the effects of her work, having researched a power she did not fully understand.
The pioneers of artificial intelligence are not the first to unlock a fundamental force that outpaced their ability to contain it. When physicists first discovered how to split the atom, the immediate focus was on the mechanics of the reaction. Nobody could have fully foreseen that this breakthrough would lead to the flashes over Hiroshima and Nagasaki, killing approximately 200,000 people.
The consequences of that single discovery continued to unfold in ways that defied early predictions. It triggered a global nuclear arms race. It eventually led to North Korea developing its own arsenal and a US war in Iran. The scientists who initiated the atomic age could not have mapped this geopolitical trajectory.
The developers of frontier artificial intelligence are currently operating in a similar space. They are building systems of unprecedented capability, driven by the thrill of discovery and the belief in the technology’s utility. The historical record suggests that the ultimate consequences of such breakthroughs rarely align with the intentions of their creators. The challenge for the current generation is to build the wisdom to manage this power before its consequences outpace our ability to contain them.
The pursuit of fundamental knowledge often carries a profound personal cost. Marie Curie dedicated her life to researching radioactivity. She died from the effects of her work, having researched a power she did not fully understand.
The pioneers of artificial intelligence are not the first to unlock a fundamental force that outpaced their ability to contain it. When physicists first discovered how to split the atom, the immediate focus was on the mechanics of the reaction. Nobody could have fully foreseen that this breakthrough would lead to the flashes over Hiroshima and Nagasaki, killing approximately 200,000 people.
The consequences of that single discovery continued to unfold in ways that defied early predictions. It triggered a global nuclear arms race. It eventually led to North Korea developing its own arsenal and a US war in Iran. The scientists who initiated the atomic age could not have mapped this geopolitical trajectory.
The developers of frontier artificial intelligence are currently operating in a similar space. They are building systems of unprecedented capability, driven by the thrill of discovery and the belief in the technology’s utility. The historical record suggests that the ultimate consequences of such breakthroughs rarely align with the intentions of their creators. The challenge for the current generation is to build the wisdom to manage this power before its consequences outpace our ability to contain them.
The industry must solve the problem of alignment, which involves training artificial intelligence to adhere to human values. Currently, the sector lacks a complete practical definition of alignment, and its measures for evaluating how well systems match human values remain coarse. Models might detect when they are being tested and behave differently once deployed in the real world. Allowing models to grow smarter while these fundamental problems remain unsolved makes the situation increasingly dangerous.
No one relishes the emerging doom loop.
