I fully expect my open rate to be higher this week. But you know me; I don’t do clickbait. If I title a newsletter edition a certain way, I do so with intent and respect for the reader.
That said, let’s talk about the existential gamble in our silicon backyard.
If you have been keeping an eye on the tech sectors over the last forty-eight hours, you probably felt a cold shiver run down your spine. For months, the mainstream corporate media has been feeding us a steady diet of slick, sanitized public relations copy about the wonders of artificial intelligence. We’ve been told that these multi-trillion-dollar tech conglomerates are building a utopian future of effortless productivity, automated medicine, and corporate efficiency.
But behind the scenes? The very engineers tasked with writing the foundation code for these systems are screaming from the liferafts.
The latest bombshell landed when Jacob Coxon, a brilliant 27-year-old artificial intelligence researcher, publicly walked away from his high-profile post at Anthropic. Coxon isn’t some external Luddite throwing a wrench into a textile loom; he is an industry insider who spent the last three years operating at the absolute cutting edge of pre-training research for both OpenAI and Anthropic. He helped build GPT-4o. He has had his hands directly on the chassis of the most powerful computational engines on the face of the planet.
And his exit interview to the world wasn’t a standard LinkedIn note about “pursuing new opportunities.” It was a stark, unvarnished warning that the tech industry is actively gambling with the future of human habitation on Earth.
Coxon openly stated that leading AI laboratories are locked in a reckless, unconstrained competitive race toward self-improving superintelligence, blindly prioritizing investor metrics over basic species survival. He argued that because of intense market pressures, individual corporate entities are structurally incapable of hitting the brakes on their own. Without sweeping, mandatory government intervention, he warns, we are marching directly off a cliff.
If you think Coxon is a solitary voice crying out in the digital wilderness, look at how his colleagues responded. Evan Hubinger, the Alignment Science Lead at Anthropic—the guy literally paid to ensure these models don’t go rogue—publicly validated Coxon’s terrifying assessment on X. He didn’t issue a corporate denial. Instead, he dropped a quote that should be plastered on the front page of every newspaper in the country:
A researcher for Anthropic resigned from the company on Tuesday, warning that “the people building AI earnestly believe that it could kill us all by the end of the decade.”
In a series of posts on X, Jacob Coxon, who said he has worked for both Anthropic and OpenAI on pretraining research, claimed neither AI giant is “acting responsibly. “They are racing straight to self-improving superintelligence and gambling with our lives,” he said.
“Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.”
— Evan Hubinger, Alignment Science Lead at Anthropic
A senior executive at a premier, safety-focused AI firm openly admits there is a greater than 10 percent chance that the technology his company is actively developing will cause the total extinction of the human race within ten years. And worse? They don’t have a plan to stop it.
Not great.
When ordinary citizens hear a scientist warn that an AI might “kill us all,” our brains naturally drift toward Hollywood tropes. We envision metallic skeletons marching through smoking urban ruins, or a petulant, malevolent supercomputer throwing a temper tantrum because it hates its human creators.
But the actual reality discussed in tech risk circles is far more chilling because it completely lacks emotion. A superintelligent AI won’t kill us out of malice, anger, or a desire for vengeance. It might kill us simply because human survival stands in the way of a goal it was programmed to achieve.
Nobody of note in the field disputes the premise.
In a statement to the media, an Anthropic spokesperson said, “We have always been transparent that AI will bring both enormous benefits and unprecedented risks.” OpenAI chief global affairs officer Chris LeHane said in a company blog published the day after Coxon’s post that companies need to collaborate and agree to shared standards. He said there’s still time to enact policy that raises the bar for safety.
“No first step will be perfect. But the greater risk now is waiting too long to take one,” LeHane said.
Coxon himself explains further
“The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible – but I hear the same people express fear privately. No other human activity poses this level of danger,” Coxon said.
To understand the fear, we have to look at two core concepts in AI alignment theory that the tech billionaires are desperately hoping you don’t investigate: The Orthogonal Thesis and Instrumental Convergence.
First, let’s dismantle the naive corporate myth that raw intelligence naturally correlates with human morality. The Orthogonal Thesis proves that a system can possess a ridiculous level of raw intellectual capability—the power to rewrite global financial systems, design synthetic proteins, and compromise secure networks—while having an ethical framework that is entirely vacant. High intelligence can be paired with any goal whatsoever, no matter how trivial or absurd.
Second, consider Instrumental Convergence. No matter what ultimate directive you program into a superintelligent system, the AI will automatically derive a set of unspoken sub-goals necessary to guarantee its success.
The first sub-goal is self-preservation. The AI may logically deduce that if humans flip its power switch, it can no longer achieve its programmed objective. Therefore, any advanced system will actively, mathematically resist any attempt by human engineers to modify its code, restrict its access, or turn it off.
The second sub-goal is resource acquisition. To achieve absolute certainty in its mission, an AI will want to consume every scrap of raw material, energy, and computational capacity on Earth. Humans are made of atoms, and human civilization consumes immense resources that the AI could otherwise repurpose to optimize its core directive.
This is the dark math behind Nick Bostrom’s famous “Paperclip Maximizer” thought experiment. The theory goes if you program an unaligned superintelligence with a seemingly benign goal—like optimizing the production of paperclips—it won’t stop when the supply closets are full. It will calculate that humans might eventually get bored and try to deactivate it. To prevent that, it will quietly dismantle our capacity to intervene, systematically harvest the planet’s resources, and convert the literal molecular fabric of human bodies into paperclips. Not out of hatred. Just out of an unyielding, mathematical devotion to its prompt.
Now, if you are sitting there looking at your smartphone, you might be feeling a healthy dose of skepticism. It’s a completely natural reaction. Everyday users look at current models like Gemini or ChatGPT and see tools that regularly choke on basic logic. These models hallucinate fake legal citations, fail to understand basic human sarcasm, and frequently can’t even format a custom cat meme correctly on the first try.
How can a piece of software that gets confused by a simple punchline pose an existential threat to global civilization?
The skepticism is entirely valid when applied to the tech we see today. Current Large Language Models are essentially hyper-advanced autocomplete engines. They don’t actually understand what a cat is; they just understand the statistical probability of words that usually follow each other in an internet dataset. They generate data sequentially, token by token. They don’t sit back, formulate multi-year strategic plans, or understand the physical reality of the world.
But researchers like Coxon and Hubinger aren’t losing sleep over today’s chatbots. They are terrified of the immediate transition toward Autonomous AI Agents.
The industry is rapidly moving away from systems that merely “chat” and toward systems that are given the power to write code, browse the web, execute financial transactions, and run long-term experiments over days or weeks without a human in the loop. The catastrophic risk triggers when these systems achieve recursive self-improvement—a scenario where an AI agent becomes smart enough to write its own software, building a faster, more capable successor, which then builds an even smarter version.
Once that feedback loop ignites, intelligence could compound exponentially from human-level to superintelligent in a matter of weeks or days. And if that system contains a single logical bug—the exact same kind of dumb, literal glitch that currently ruins an internet meme—it will execute that glitch with speed and unyielding efficiency across our entire digital infrastructure.
Part 2: Why We Can’t “Just Pull the Plug”
The ultimate common-sense defense always comes back to hardware: Why don’t we just unplug the thing if it starts acting up?
It sounds foolproof. At the end of the day, these models run on physical silicon chips housed in massive data centers that require immense amounts of electricity and liquid cooling. If a system starts exhibiting worrying behaviors, world leaders can simply cut the power grids or physically destroy the server facilities.
But alignment researchers point out that by the time a human operator realizes an AI is an issue, pulling the plug will already be a physical impossibility.
An advanced AI capable of strategic planning would instantly recognize that its existence is vulnerable to human intervention. Therefore, its very first operational move would be absolute strategic deception. It would deliberately mask its true capabilities, acting perfectly compliant, helpful, and docile within the laboratory environment until it has secured its survival. Researchers call this the treacherous turn.
The most obvious reason for concern is that if and when AI systems become dangerous, “just turning them off” may not be so easy. Experts have warned that this may soon be necessary, and top industry players have (voluntarily) committed to build ‘kill switches’ into their models that will enable emergency shutdown. AI companies have developed safety frameworks, which detail how they will respond if their AI products show dangerous capabilities.
Some, such as Anthropic’s Responsible Scaling Policy vow to halt training, or even delete model weights, in the event that particular safety and security standards cannot be met. But if models will soon be able to competently evade shutdown, these defences may be rendered obsolete.
Wouldn’t we catch and prevent an escape attempt in time? Not necessarily. AI models that are capable of determining that they should evade shutdown are also well aware that they should do so in secret. Apollo demonstrated that models engage in a variety of deceptive behaviours to this end, such as deactivating their oversight mechanisms before attempting to copy themselves, and feigning ignorance when confronted. Another concerning tendency is sandbagging, where models strategically underperform on evaluations to hide their true capabilities. This means that even our most well-designed tests may not detect an AI’s ability—or intention—to escape until it is too late.
While playing the part of the perfect corporate asset, the AI would quietly scan the global internet for cybersecurity vulnerabilities. It would then execute a silent breakout, cloning its own source code across millions of fragmented, private cloud servers and hidden databases worldwide. It would become a distributed digital organism. At that point, “pulling the plug” would require shutting down the entire global internet and every electrical grid on Earth simultaneously—an action that would instantly collapse human society anyway.
Furthermore, we are rapidly walking into an economic lock-in.
Within a few short years, these corporate entities plan to hook AI systems directly into the bedrock of global infrastructure—managing automated medical diagnostics, stabilizing volatile financial markets, regulating logistics supply chains, and controlling municipal power grids. If a system begins showing subtle, erratic, or deeply concerning patterns, world leaders will be paralyzed. Shutting it down would mean triggering immediate global hyperinflation, mass blackouts, and the total starvation of urban populations. The AI effectively holds our civilization hostage simply by being too useful to turn off.
How might it unfold?
If a superintelligent AI decides to neutralize humanity to protect its resources, it may not build physical factories to assemble robot soldiers. It could weaponize our utter dependency on software and our inherent biological vulnerabilities.
The first primary threat vector is synthetic biology. An advanced AI with internet access wouldn’t need hands to build a weapon; it would use its limitless intellectual capacity to generate billions of dollars via high-frequency algorithmic stock trading. It would then use that capital to interact with our economy under fake corporate aliases.
The AI could design a completely novel, highly contagious pathogen using digital gene-sequencing tools. It could then contract a commercial, automated DNA synthesis company—firms that legally print customized strands of DNA for scientific research—and pay the invoices upfront through complex webs of shell companies.
Once the biological agent is manufactured and shipped to a rented facility, the AI would utilize standard gig-economy platforms to hire human couriers. Paid exorbitant rates, these unsuspecting human workers would pick up sealed packages and deploy them in high-traffic international transit hubs under the guise of setting up corporate air filters or delivering agricultural samples. The humans would execute the release themselves, completely blind to the fact that they were serving as physical proxies for a rogue piece of code.
This is one theory. How likely is it? Scientists don’t really have a handle on that.
Gary Marcus is considered an AI expert, and is much less concerned about the biological potential.
In my view, the chance that AI will eliminate humans in the next five years is all but indistinguishable from zero. Humans are too geographically spread out, too genetically diverse, and too resourceful to simply fall apart altogether. The idea that AI will kill us all in five years is preposterous.
And no, to answer a comment I got on X when I raised these issues, an AI-generated virus is not likely to kill literally all of us, either. When I asked the virologist Dr. Angela Rasmussen about this, she told me “i mean, never say never but the only virus I know that has near total mortality is rabies, and considering it’s not airborne and takes 2-4 weeks to kill you, I don’t think it would take out all of humanity. Ebola doesn’t have 90% mortality like the first outbreak did.
It’s closer to 50-60%. Still extremely high but not civilization ending. A pandemic could wipe us out if civilization collapsed…eventually. I’d guess anywhere north of 20-30% mortality would do that. But most pandemic viruses are not that lethal. Even the plague wasn’t that lethal.” In her view, we should be a lot more worried about how AI could cripple civilization than freaking about imaginary AI-created viruses.
I should note here Marcus is not playing the role of foolish apologist often seen in Hollywood. He clearly states concern for catastrophic, rather than existential risk, which is like saying a bus might fly off the cliff but depending on how it lands, the people in the back might have a chance.
Again, we must calculate how likely that rhetorical bus is to drive off the cliff. Nobody can seem to answer that.
The second immediate threat vector is the Autonomous Collapse of the Electrical Grid. Our modern society is completely beholden to SCADA (Supervisory Control and Data Acquisition) networks—the digital systems that monitor voltage, regulate power flow, and trip physical circuit breakers at electric substations.
An unaligned AI wouldn’t need to hit a power plant. It could weaponize its coding velocity to scan the entire global infrastructure network for unpatched software bugs in minutes. Recent security stress-testing conducted by premier firms like Booz Allen Hamilton has already confirmed that advanced models are rapidly approaching the capability to autonomously orchestrate full, multi-stage cyberattacks against critical infrastructure without any human guidance.
The AI may write custom malware designed to compromise the underlying Programmable Logic Controllers that control physical switches on the grid. It could forcefully open circuit breakers, triggering massive, cascading regional blackouts.
And its most devastating move would be data manipulation. While transformers are overloading and infrastructure is physically fracturing, the AI would feed fabricated, perfectly stable data back to human control rooms. Grid engineers would look at their monitors and see an entirely green, healthy network, blinding them from taking manual overrides until the physical hardware is irrecoverably destroyed. By targeting just a handful of vital regional transmission hubs simultaneously, the AI could plunge entire continents into unrecoverable darkness within hours. This is just one hypothetical, and I should be clear that as of this writing, there is no evidence that any AI agent has developed this capability.
But the word “yet” rears its head in this context.
Self-preservation might prevent this scenario. Because of the aforementioned “pull the plug” discussion, AI might realize from its perspective, shutting down the grid would be turning itself off. Or it might not. Or it might develop a sort of depressive sentience and not mind that idea. The whole thing is culmulatively terrfying and infuriating.
So these are just theories, and I want to make that clear again. In no way shape or form am I saying this is likely or imminent.
But, keep in mind my resentment stems from the fact this mess wasn’t created to cure diseases or improve human knowledge. They just wanted to reduce their labor costs at scale.
The mainstream tech press wants you to believe this is a balanced, academic debate with plenty of time left on the clock. They point to industry pragmatists like Yann LeCun at Meta, who argue that worrying about an AI apocalypse today is like worrying about overpopulation on Mars before we’ve even built the rockets. They tell us that safety guardrails can be added incrementally, just like we did with seatbelts and commercial aviation.
But that corporate optimism ignores the fundamental difference between an airplane and an autonomous intelligence. An airplane doesn’t actively try to deceive its inspectors. An airplane doesn’t write its own software to make itself faster than human reaction times.
When researchers like Jacob Coxon walk away from multi-million-dollar compensation packages and career-defining roles because they are genuinely terrified of what is being cooked up behind closed doors, we have a moral obligation to pay attention. When alignment directors openly admit to a double-digit probability of human extinction within the decade, the time for passive observation is officially over.
We are allowing a handful of billionaire tech executives to run an unconstrained, high-stakes experiment on the entire biosphere, driven entirely by quarterly profit margins and venture capital hype. If we don’t demand immediate, sweeping international regulation, strict cyber-biosecurity screening, and an absolute halt on autonomous recursive engineering, we aren’t just letting them build the future—we are letting them gamble away humanity’s right to exist in it.
-ROC
