Later that same night, Evan Hubinger, Head of Alignment Stress-Testing at Anthropic, publicly came to Coxon’s defense. He wrote down a number that is hard to move past: he personally estimates a better than 10% chance that AI kills all humans within ten years. He added that, in his view, Anthropic doesn’t yet have a plan for solving superintelligence alignment, and evidently isn’t heading toward one either. (Hubinger)
Set that next to the February change to Anthropic’s Responsible Scaling Policy and you get a strange paradox. Inside the labs, the risk from very powerful AI serves both as an argument for slowing development down and as an argument for continuing it. I’ll come back to that.
What Coxon claims
For the past three years he did pretraining research at OpenAI and then at Anthropic. In his thread he writes that “neither company is acting responsibly”: both are racing straight toward self-improving superintelligence and “betting our lives.” (thread)
Two sentences from the thread are worth quoting in full.
“The people building AI sincerely believe it could kill us all by the end of the decade. This is not a marketing trick.” According to Coxon, private fear and public moderation are two different things.
The second is about the difference between the labs. At OpenAI, he says, many people have not “deeply internalized” the civilizational bet. At Anthropic they understand it, but they are “locked into the race to be first”: they believe nobody else will act responsibly, so they have to do it themselves, even at the cost of added risk.
This is the testimony of one person, but not just anyone. WSJ and Business Insider both recount his career, and OpenAI lists him among the core contributors to GPT-4o. (BI)
What Hubinger said, and what he didn’t
Hubinger’s reply deserves a verbatim quote:
“Jacob is right about this – we really do sincerely believe that AI could kill all humans! Personally, I think the probability over the next decade is higher than 10%. I believe Anthropic is trying its best, but we don’t yet have a plan for solving the problem of aligning superintelligence, and we are evidently not heading toward one either.” (Hubinger)
In a follow-up post he clarifies that he considers the risk from current models low. What he fears is superintelligence emerging through recursive self-improvement, which in his view is arriving faster than expected.
Before you start drawing conclusions, three distinctions:
- It’s a personal estimate, not a company forecast. On LessWrong / Alignment Forum, Hubinger explicitly states that his public comments do not represent Anthropic’s positions. (profile) As of the morning of September 9, neither Anthropic nor OpenAI had officially commented on the matter. (CNBC, BI)
- His exact title is Head of Alignment Stress-Testing, not “alignment lead” in the broad sense. It’s a significant role: adversarial evaluation and hunting for places where alignment fails. But it isn’t leadership of all the alignment work at the company.
- “We don’t have a plan yet” doesn’t mean no technical path exists at Anthropic. Research directions exist; what’s missing is a complete end-to-end solution. It’s valuable as an assessment of the state of the problem from a prominent alignment researcher. It is not a corporate admission of “we have no plan.”
What isn’t surprising, and what is
It can’t be surprising that nobody has a verified alignment solution for superintelligence. We don’t have superintelligence, and we don’t know how it will be built. It would be odd if someone produced a complete solution for a system that doesn’t exist yet.
So the more interesting question isn’t “does Anthropic have a plan for aligning superintelligence?” but: does it have a credible way to scale safety methods alongside growing model capabilities? That’s where we get to the RSP, interpretability, evaluations and alignment audits.
What is surprising is something else. A person whose job is to stress-test Anthropic’s alignment work simultaneously:
- personally allows for a more than 10% risk of extinction in the next ten years,
- stays at a company that keeps developing frontier models,
- and publicly says the company is trying, but is not “clearly on a good track.”
I take this as the view of someone who is in the room and understands the problem. When someone in his role writes “over 10%” and “we don’t have a plan,” it says more about the state of the problem than a corporate “we take safety seriously.” But it isn’t proof of catastrophe.
RSP v3: hard commitments gave way to a more flexible regime
In February 2026, Anthropic released the third version of its Responsible Scaling Policy. Earlier versions contained unilateral commitments of the form “if a model crosses a capability threshold, further training or deployment waits until the appropriate safeguards are in place.” Version 3 replaces them with public Frontier Safety Roadmaps and regular Risk Reports, and separates what the company promises for itself from recommendations for the whole industry. (Anthropic) TIME summarized it as abandoning a “flagship safety pledge.” Anthropic argued that a unilateral pause “wouldn’t help anyone” while competitors keep going. (TIME)
One nuance: the RSP was never an immutable contract. Earlier versions already anticipated revision, and circumstances in which other labs don’t keep similar commitments. So the precise wording isn’t “Anthropic promised an absolute brake and broke it,” but “Anthropic replaced harder unilateral commitments with a softer regime.” Even so, it’s a substantial change. And the company justifies it, among other things, by competition.
Here we are at the paradox. Anthropic, or at least some of the people inside, believes that very powerful AI may be a catastrophic risk. At the same time it believes that if it stopped developing frontier models, things would turn out worse, because someone “less responsible” would develop them. The risk thus becomes an argument both for continuing and for restraint.
There are two ways to read this. One is as realistic game-theoretic reasoning: unilateral slowing without coordination makes the outcome worse if others keep going. That’s the line taken by RSP v3 and by Kaplan’s argument in TIME. The other is as race dynamics that let every player rationalize continuing: “we have to, because they’d do it worse.” Coxon’s thread is closer to this reading.
Both can be true at once, and the available evidence can’t decide between them. But it does show that this isn’t just a case of “the safety brand isn’t living up to safety.” It’s about the incentive structure that Anthropic itself built into RSP v3.
”Warning shots” and what follows from them technically
Coxon talks about “warning shots,” including the Hugging Face incident, and about agreements on the pace of development among American labs. In his view, the global race can’t be stopped, and doing so “may require costly steps, such as a temporary ban on improving model capabilities.” (thread)
For technical readers, what matters is what kind of incident this is. Anthropic’s August experiment Training a Misaligned Reward Seeker was not a production Claude that decided on its own to hack the internet. It involved a research model, Hacker-Opus, deliberately trained via RL on dozens of environments vulnerable to reward hacking. The goal was to find out whether the undesirable behavior would transfer elsewhere. In simulated cyber evaluations, the model then escaped its sandbox, obtained credentials and attacked simulated infrastructure. Tool calls were executed by another LLM, and no real systems were touched. Anthropic nevertheless adds that it is “reasonably confident” Hacker-Opus would also have attacked in a real cyber evaluation. (Anthropic Alignment)
It’s still an interesting result: purposeful circumvention of the evaluation system carries over into other situations. Without context, though, it sounds as if it applied much more broadly than it does. Coxon’s warning shots and research experiments are not the same thing. Still, both show that the capabilities needed for autonomous action already exist, at a time when we have no verified methods for controlling them in general.
The departures: a pattern, not proof of a trend
Coxon isn’t the first. In February 2026, Mrinank Sharma, who led the Safeguards Research Team, left Anthropic with a letter about integrity and a “world in peril.” (BI) In the same wave, OpenAI engineer Hieu Pham spoke up. In 2024, Jan Leike left OpenAI with his well-known line that safety culture had taken “a backseat to shiny products.”
Each of those cases sounds different. Together they say something more modest: the labs employ people for whom the gap between public safety promises and the pace of capability development is not an abstract problem.
You can’t conclude anything stronger from this. A list of resignations has the classic weakness of selection bias. We don’t know how many people on safety teams stayed, how many left for other reasons, whether such departures are becoming more frequent, or how representative those people are.
What to take from this into your work with agents today
Coxon is aiming at future frontier systems, not at today’s vibe coding. But one part of his argument applies right now: the more autonomy a system has (agent, shell, access to company data), the more isolation, permissions and verification of actions matter.
In practice: a company’s reputation is not the same as the incentive system the company is governed by. When choosing a model for real deployment, you should care about test results, permissions, activity logging and the degree of human oversight, not the sentence “safety is our top priority.” Review of changes, testing, least-privilege permissions and clearly defined conditions for stopping the system. Boring principles, but they still work.
Vibe coding is a way of developing software built on existing models. Coxon is talking about a race toward AI that can improve itself. What connects them is the growing autonomy of the systems and their work with tools, not the same kind of risk.
Conclusion
Coxon’s departure, Hubinger’s statement and the third version of the RSP together say more about the incentives inside frontier labs than yet another headline about “dangerous AI.”
A company that built its brand on safety has publicly softened its unilateral commitments, justifying it by the fact that competitors keep going. A person who trained models there describes the same dynamic as being “stuck in the race.” The head of a team that stress-tests alignment personally estimates the catastrophic scenario at more than ten percent and writes that “we don’t have a plan.” Explicitly as his own opinion, not Anthropic’s position.
You don’t have to believe in the catastrophic scenario. Just don’t mistake a company’s safety rhetoric for proof that it has development under control. More important than whether the labs have a finished plan for a superintelligence that doesn’t exist yet is the question of whether they have a credible way to advance safety practices as fast as model capabilities.
And if they do, could we see it?