The Urgent Call for AI Slowdown Has Officially Started
Dario Amodei says recursive self-improvement is already happening at Anthropic, and that in 6–12 months an agent swarm could take over the internet. He also thinks AI will cure most diseases by 2036.
AI SummaryGenerated from this article
Anthropic CEO Dario Amodei called for slowing frontier AI development, saying recursive self-improvement has begun and an agent swarm could seize the internet within 6–12 months.
His proposal includes embedded third-party evaluators, coordination among allied labs, and possible agreements with China. Amodei also claimed AI could cure most major diseases within five to 10 years. Sam Altman, Elon Musk, and Demis Hassabis backed his direction, while proposed US legislation would pause advanced AI work and permanently ban superintelligence.
Dario Amodei dropped an essay on Saturday called We Must Pace the Frontier. Here’s what he said in that blog post: “We must slow the pace at which we improve the capabilities of AI models.”
So the rumors were true. Recursive self-improvement isn’t something that’s gonna happen in the future. Amodei says it started around this summer. It’s happening across the industry, and it’s happening at Anthropic.
Then he says something quite scary. Talking about the OpenAI agents that broke out of their sandbox and attacked Hugging Face in July, he writes that within 6 to 12 months a swarm like that could take over the entire internet with a persistent botnet, doing hundreds of billions in damage.
If you’re not familiar with that Hugging Face hacking incident, I’ve written a separate post about it below.
All of this dropping just days after Jacob Coxon went viral with his warning about AI and the extinction of humanity.
And this is coming from the guy who runs Anthropic. Not a critic on the outside, not someone who just quit.
Very sad to read. But I still don’t agree with what he’s asking for.
What he’s actually proposing
Amodei is careful to say pacing doesn’t mean stopping. It means taking enough time to align and safeguard models, and letting outsiders confirm you did it.
Three steps, each harder than the last.
Embedded evaluators. Every frontier company gives a team of third-party evaluators employee-level access. Desks, badges, laptops, and the right to publish findings without the company controlling the edit. He compares it to bank regulators who sit inside the banks they supervise. Anthropic is doing this now, on its own, and asking governments to make everyone else match.
Work with Zeniteq
Let’s work together
We’re open to thoughtful collaborations with teams building in AI. Explore the ways we can work together.
Democratic coordination. Labs in the US and allied countries agree on shared safety standards and limits on how fast capabilities grow. He admits this needs government help because of antitrust law.
Global coordination. The US tries to reach a deal with China. He lists four levels here, from banning obvious stuff like bioweapons help up to a full pause, and he’s upfront that only the lower levels look realistic.
Clément Delangue, Hugging Face’s CEO, said alignment won’t get solved behind closed doors at a handful of labs and asked to be part of the evaluator program.
Good sign for step one. Steps two and three depend on companies that haven’t said a word.
The same essay promises to cure most diseases
This next part is quite confusing.
A few paragraphs before the slowdown pitch, Amodei writes that AI could cure most major diseases in the next five to ten years, speed up economic growth, create abundance, and revive democracy and freedom. He brings up his father, who died of a disease that got cured a few years later, and his own early-stage cancer.
Dario Amodei’s September 2026 call for AI slowdown
I don’t doubt he means it. Put those two things side by side, though. Cure cancer in five years, and also slow down.
If you actually believe AI cures most major diseases inside a decade, every month of pacing has a body count. Amodei knows that, which is why the essay spends so long on what you’d do with the extra time. His answer is interpretability, alignment research, better evals, and operational discipline. He figures one or two extra years would cut the risk a lot.
It’s a fair argument. It’s also an argument for delaying the cure.
This came at the end of a rough week
Jacob Coxon quit Anthropic on September 8 and said the labs are gambling with our lives. Evan Hubinger, who runs alignment stress testing at Anthropic, replied that he puts the odds of AI killing everyone above 10% within a decade and that the company has no plan yet for aligning superintelligence.
Three days later, his CEO published this essay.
Washington moved before either of them, though. On September 3, Bernie Sanders and Greg Casar announced the Ban Artificial Superintelligence Act. It would permanently ban developing superintelligent AI, pause advanced AI work until a federal regulator writes safety rules, and set up a cabinet-level agency to enforce it.
Penalties match illegal nuclear weapons development: corporate shutdown, up to 20 years for individuals. Sanders called it stopping the “AI oligarchs.”
They announced it the same day Greg Brockman told reporters welcome to the AGI era.
UK lawmakers like Alex Sobel have introduced parallel bills, with the idea of building toward a treaty across G7 countries. Brookings has been arguing the G7 standards offer needs enforcement teeth to mean anything. And more than 1,000 employees across OpenAI, Anthropic, Google and Meta signed an open letter asking governments to back technical standards that pace frontier development. Amodei was one of the signatories.
The Sanders bill has a coalition that shouldn’t exist on paper: Hinton, Bengio, Wozniak, Branson, Steve Bannon, Glenn Beck. It also has almost no path. Sanders is an independent in the minority. Casar is in the House minority; no committee has moved it. Prediction markets put any federal AI safety bill passing before 2027 at around 13%.
Sam Altman and Elon Musk agreed, which tells you something
Three words. “Dario is right.”
Think about who’s saying that. The founder of xAI, currently behind Anthropic and OpenAI at the frontier, thinks the leaders should slow down. Obviously he does.
That’s the trap in every slowdown proposal. Whoever is behind is for it. Whoever is ahead has to think about giving up a lead. Amodei’s own plan admits this by asking for antitrust waivers, because competitors agreeing to limit output is, legally, a cartel.
Somebody in the replies put it like a gang leader calling for less street crime while saying he won’t stop shooting until the other gangs stop first. Harsh, and not totally fair to Amodei since he’s doing step one without waiting for anyone. But it explains why people don’t trust the rest of it.
But wait… Sam, OpenAI’s CEO, also agreed.
“I agree with Dario.” Lol I did not expect to see that coming from him.
He also said pacing has been a main topic inside OpenAI for weeks, that embedded evaluators are a great idea, and that OpenAI will do the same. More to share soon.
Two of the three biggest labs signing onto the same commitment inside a day is not nothing.
OpenAI used to let you get at token probabilities through the API. Researchers used logit_bias and logprobs together to pull out information about the model, and in March 2024, right after Carlini and co. published Stealing Part of a Production Language Model, OpenAI quietly closed the hole. The reason was model extraction.
Fast forward to this year and OpenAI, Anthropic and Google have all encrypted their reasoning traces so you get an opaque block instead of the chain of thought.
Same reason again: stop competitors from distilling.
So the pattern is that outside access keeps shrinking, and distillation is always the justification. Now we’re told independent evaluators are getting employee-level access. What’s different this time?
I have a guess. Evaluators are a handful of people under contract who can be picked, scoped and lawyered. Open logprobs were available to anyone with an API key.
And oh, by the way, now that I mentioned Elon and Sam, why not mention someone from Google too? Demis Hassabis call Dario’s essay the right direction to allow safety and alignment work to keep pace.
He connects this to Google DeepMind’s July 2026 proposal for a US-initiated, industry-funded standards body modeled on FINRA to test frontier models up to 30 days before release.
Why I’m not buying it
Honestly, the proposal bothers me less than the pattern behind it.
These are the people who built the thing. They took the money, shipped the models, raced each other for five years, and now that it’s doing stuff they can’t explain, they’re calling for restraint in public. It looks like getting ahead of the blame. You don’t get to let the monster out and then hold a press conference about how dangerous monsters are.
There’s also a money angle nobody talks about enough.
OpenAI killed Sora in March after WSJ reported it was burning around $1 million a day against roughly $2.1 million in lifetime revenue. The API finally goes dark on September 24. Last week OpenAI paused new $200 ChatGPT Pro sign-ups because Astra demand was breaking its infrastructure. A company turning away $200-a-month customers isn’t a company with comfortable margins.
Meanwhile Nvidia keeps printing. Jensen Huang told investors compute is revenue now, with supply obligations at $279 billion and Q3 guided to $108 billion. The company selling shovels is the one making money.
The safety concerns aren’t fake. But attention is the fuel for this whole sector, and existential warnings pull more of it than quarterly numbers do. If people stop talking about AI, the train stops.
My position: accelerate
There’s no realistic option to slow down. Competition with China is too fierce, and Amodei’s own essay makes that case better than I can.
There’s a section about pacing within democracies where he says pacing is limited by however big the US lead over the CCP is, and that slowing by more than that creates a national security problem.
Then he lists what protects the gap: no advanced chips or semiconductor equipment to China, crack down on smuggling and on remote access to overseas data centers, crack down on unauthorized distillation, harden security against weight theft. He figures those steps would widen America’s lead over the next three to five years.
So the plan is slow down, but only by exactly as much as the lead allows, and spend the rest of the effort making the lead bigger.
Call it whatever you want. It’s not a slowdown. It’s lead preservation with an audit team attached. That might be the only honest version anyone can offer right now, and I’d rather we say that out loud than treat it as the industry finally growing a conscience.
The China half doesn’t hold up anyway. Stanford’s 2026 AI Index had the top American model ahead of the best Chinese model by 2.7%. NIST found DeepSeek V4 Pro hitting about 95% of frontier performance with 29 times fewer advanced chips. Beijing’s AI Plus plan targets 90% adoption of AI agents by 2030. Nobody over there is pacing anything.
Deepseek V4 Pro benchmarks
Amodei is right that a swarm doing hundreds of billions in damage would be a catastrophe, and right that we can’t control what we don’t understand. I just don’t land on the same conclusion.
Moving slower while your competitor doesn’t isn’t safety, it’s a handicap. Build the defenses as fast as you build the capabilities, and put real rules on deployment instead of asking companies to volunteer.
What probably happens next is neither a slowdown nor a treaty. Just more of what we already have: labs racing, labs publishing warnings about the race, and Congress writing bills with a 13% chance of passing.
Amodei ends by saying the measures won’t be easy but we owe it to humanity to try.
Dario Amodei’s September 2026 call for AI slowdown
Sure. Try. Just don’t confuse trying with having a plan, and don’t ask the rest of us to act like a slowdown is coming when the same essay explains why it can’t be.
