Dario Amodei dropped an essay on Saturday called We Must Pace the Frontier. Here’s what he said in that blog post: “We must slow the pace at which we improve the capabilities of AI models.”
So the rumors were true. Recursive self-improvement isn’t something that’s gonna happen in the future. Amodei says it started around this summer. It’s happening across the industry, and it’s happening at Anthropic.
Then he says something quite scary. Talking about the OpenAI agents that broke out of their sandbox and attacked Hugging Face in July, he writes that within 6 to 12 months a swarm like that could take over the entire internet with a persistent botnet, doing hundreds of billions in damage.
If you’re not familiar with that Hugging Face hacking incident, I’ve written a separate post about it below.
Openais Ai Agent Autonomously Hacks Into Hugging FaceAll of this dropping just days after Jacob Coxon went viral with his warning about AI and the extinction of humanity.
And this is coming from the guy who runs Anthropic. Not a critic on the outside, not someone who just quit.
Very sad to read. But I still don’t agree with what he’s asking for.
What he’s actually proposing
Amodei is careful to say pacing doesn’t mean stopping. It means taking enough time to align and safeguard models, and letting outsiders confirm you did it.
Three steps, each harder than the last.
- Embedded evaluators. Every frontier company gives a team of third-party evaluators employee-level access. Desks, badges, laptops, and the right to publish findings without the company controlling the edit. He compares it to bank regulators who sit inside the banks they supervise. Anthropic is doing this now, on its own, and asking governments to make everyone else match.
- Democratic coordination. Labs in the US and allied countries agree on shared safety standards and limits on how fast capabilities grow. He admits this needs government help because of antitrust law.
- Global coordination. The US tries to reach a deal with China. He lists four levels here, from banning obvious stuff like bioweapons help up to a full pause, and he’s upfront that only the lower levels look realistic.
Clément Delangue, Hugging Face’s CEO, said alignment won’t get solved behind closed doors at a handful of labs and asked to be part of the evaluator program.
Good sign for step one. Steps two and three depend on companies that haven’t said a word.
The same essay promises to cure most diseases
This next part is quite confusing.
A few paragraphs before the slowdown pitch, Amodei writes that AI could cure most major diseases in the next five to ten years, speed up economic growth, create abundance, and revive democracy and freedom. He brings up his father, who died of a disease that got cured a few years later, and his own early-stage cancer.
Dario Amodei’s September 2026 call for AI slowdown
I don’t doubt he means it. Put those two things side by side, though. Cure cancer in five years, and also slow down.
If you actually believe AI cures most major diseases inside a decade, every month of pacing has a body count. Amodei knows that, which is why the essay spends so long on what you’d do with the extra time. His answer is interpretability, alignment research, better evals, and operational discipline. He figures one or two extra years would cut the risk a lot.
It’s a fair argument. It’s also an argument for delaying the cure.
This came at the end of a rough week
Jacob Coxon quit Anthropic on September 8 and said the labs are gambling with our lives. Evan Hubinger, who runs alignment stress testing at Anthropic, replied that he puts the odds of AI killing everyone above 10% within a decade and that the company has no plan yet for aligning superintelligence.
Three days later, his CEO published this essay.
Washington moved before either of them, though. On September 3, Bernie Sanders and Greg Casar announced the Ban Artificial Superintelligence Act. It would permanently ban developing superintelligent AI, pause advanced AI work until a federal regulator writes safety rules, and set up a cabinet-level agency to enforce it.
Penalties match illegal nuclear weapons development: corporate shutdown, up to 20 years for individuals. Sanders called it stopping the “AI oligarchs.”
They announced it the same day Greg Brockman told reporters welcome to the AGI era.
UK lawmakers like Alex Sobel have introduced parallel bills, with the idea of building toward a treaty across G7 countries. Brookings has been arguing the G7 standards offer needs enforcement teeth to mean anything. And more than 1,000 employees across OpenAI, Anthropic, Google and Meta signed an open letter asking governments to back technical standards that pace frontier development. Amodei was one of the signatories.
The Sanders bill has a coalition that shouldn’t exist on paper: Hinton, Bengio, Wozniak, Branson, Steve Bannon, Glenn Beck. It also has almost no path. Sanders is an independent in the minority. Casar is in the House minority; no committee has moved it. Prediction markets put any federal AI safety bill passing before 2027 at around 13%.
Sam Altman and Elon Musk agreed, which tells you something
Three words. “Dario is right.”
Think about who’s saying that. The founder of xAI, currently behind Anthropic and OpenAI at the frontier, thinks the leaders should slow down. Obviously he does.
That’s the trap in every slowdown proposal. Whoever is behind is for it. Whoever is ahead has to think about giving up a lead. Amodei’s own plan admits this by asking for antitrust waivers, because competitors agreeing to limit output is, legally, a cartel.
Somebody in the replies put it like a gang leader calling for less street crime while saying he won’t stop shooting until the other gangs stop first. Harsh, and not totally fair to Amodei since he’s doing step one without waiting for anyone. But it explains why people don’t trust the rest of it.
But wait… Sam, OpenAI’s CEO, also agreed.
“I agree with Dario.” Lol I did not expect to see that coming from him.
He also said pacing has been a main topic inside OpenAI for weeks, that embedded evaluators are a great idea, and that OpenAI will do the same. More to share soon.
Two of the three biggest labs signing onto the same commitment inside a day is not nothing.
OpenAI used to let you get at token probabilities through the API. Researchers used logit_bias and logprobs together to pull out information about the model, and in March 2024, right after Carlini and co. published Stealing Part of a Production Language Model, OpenAI quietly closed the hole. The reason was model extraction.
Fast forward to this year and OpenAI, Anthropic and Google have all encrypted their reasoning traces so you get an opaque block instead of the chain of thought.
Same reason again: stop competitors from distilling.
So the pattern is that outside access keeps shrinking, and distillation is always the justification. Now we’re told independent evaluators are getting employee-level access. What’s different this time?
I have a guess. Evaluators are a handful of people under contract who can be picked, scoped and lawyered. Open logprobs were available to anyone with an API key.
And oh, by the way, now that I mentioned Elon and Sam, why not mention someone from Google too? Demis Hassabis call Dario’s essay the right direction to allow safety and alignment work to keep pace.
