The Godfather of AI Thinks Something Huge Is About to Happen

Back in 2023, I remember seeing the term “intelligence explosion” come up quite a lot in AI discussions. People were already talking about AI improving itself and what could happen if that process started speeding up, but it never seemed like an immediate concern.

Most of the attention was still on things like bigger models, better chatbots, and the race toward AGI. The intelligence explosion was there in the background, but it still sounded like something we might have to worry about much later.

Geoffrey Hinton apparently doesn’t see it that way anymore.

Hinton, the Nobel Prize-winning researcher widely known as the “Godfather of AI”, recently wrote that recursive self-improvement has been discussed for a long time, but “until very recently it did not seem imminent.” Now, he says, many leading researchers think it could happen “quite soon.”

That’s a pretty big change in tone. In fact, some users have criticized him in one of his X posts about this paper:

More important than the post itself is the paper behind it.

Hinton is one of 22 authors of What if automating AI R&D triggers an intelligence explosion?

Paper: What if automating AI R&D triggers an intelligence explosion?

The list also includes Yoshua Bengio, OpenAI chief scientist Jakub Pachocki, Anthropic co-founder Jack Clark, Microsoft chief scientific officer Eric Horvitz, Dawn Song, Andrew Barto, and researchers from Cambridge, Oxford, and several AI policy organizations.

Their argument is simple: AI is getting good at AI research. If AI eventually becomes good enough to improve the systems that improve AI, you get a loop. If every trip around that loop makes the next trip faster, things could get weird very quickly.

The mechanism for a software-driven intelligence explosion

The paper defines an intelligence explosion as a dramatic acceleration in AI development where progress that would normally take years happens in months or less. The authors argue that there is now enough evidence to take that possibility seriously, although they repeatedly stress that it remains uncertain.

Keep repeating that process and you have the basic idea behind an intelligence explosion.

Intelligence explosion in detail

An intelligence explosion doesn’t simply mean that AI gets really smart.

We have already been watching AI get better and better every year. Models improve, costs go down, context windows get larger, coding ability gets better, and tasks seemed impossible a year ago become normal.

The important part of an intelligence explosion is that the speed of improvement starts increasing too.

Imagine AI capability improving by roughly the same amount every year. Even if that improvement is fast, it is still a relatively normal development curve. Researchers build a stronger model, spend another year doing research, then build another one.

Now imagine that the stronger model can itself contribute meaningful work toward building its successor.

Researchers give it experiments to design. It writes the code, analyzes failed runs, proposes changes to the training process, builds evaluations, and searches through hundreds or thousands of possible improvements. The next model arrives sooner partly because the previous model helped build it.

Then the new model joins the same research process, except it is better at the work.

The cycle starts again.

The paper describes an intelligence explosion as a qualitative break from the progress we have seen so far, potentially compressing advances that would otherwise take years into months or less.

The idea itself is old. Computer scientist I. J. Good was writing about an “intelligence explosion” in the 1960s. The interesting part in 2026 is not that someone rediscovered the theory. It is that we now have AI systems participating in the research process that creates future AI systems.

We Can Already See the Early Signs

OpenAI recently published a detailed look inside how AI agents are being used by its research teams. The company says it has now reached its goal of building what it calls an “automated research intern,” a system that can handle well-defined assignments that might take a skilled researcher a few days.

OpenAI says it is working toward an automated AI researcher by March 2028. By mid-August, its research organization was using around 3.1 agent-workdays for every human workday.

Agentic workdays now far exceed those of human researchers

Researchers were also running more experiments, writing more code, and handing increasingly complicated work to agents.

But there is a useful reality check in the same report.

Humans are still choosing the big research priorities, deciding which ideas deserve more work, and making decisions about scaling and deployment. More than half of successful four-to-eight-hour agent tasks still required at least one human intervention.

That is almost the perfect snapshot of where we are.

AI is not independently running OpenAI’s research organization. At the same time, calling these systems nothing more than coding assistants no longer describes what is happening either.

Anthropic’s numbers are even more interesting.

The company recently published its own measurements of how much AI is involved in building future AI systems. It uses a scale that runs from no AI involvement all the way up to full autonomy.

Claude has not reached full autonomy for any measured part of Anthropic’s R&D.

But as of August, Anthropic says Claude leads 26% of its measured AI R&D work. That means it can complete most of those tasks from a high-level prompt while a human supervises. More than 90% of the measured work involves AI at least collaboratively.

Claude now leads 26% of model R&D work

Anthropic says around 30,000 agents were doing research and engineering work at the company at any one time on its most-used internal platform. In August alone, its monitoring systems processed more than a billion decisions from those agents.

Those 30,000 agents are obviously not the equivalent of 30,000 Geoffrey Hintons sitting around inventing new architectures. A lot of the work is engineering, and humans are still heavily involved.

But think about how strange that phrase would have sounded three years ago.

A major AI lab already has tens of thousands of AI agents doing internal research and engineering work, while another is publicly targeting an automated AI researcher.

The feedback loop has not taken off.

Pieces of it are already here.

AI 2040 suddenly looks more interesting

This story made me think about the AI 2040: Plan A released a couple of weeks ago.

The people behind AI 2040 are also behind AI 2027, and I want to make one thing clear before comparing it with what is happening now. AI 2040 is a scenario and a policy proposal, not a prediction of exactly what will happen.

AI 2040 Report

Still, parts of its setup are becoming strangely familiar.

In the scenario’s 2027 section, the United States effectively has two workforces. One is human. The other consists of millions of AI agents running constantly. Most of the AI output is mediocre, but enough of it is useful that the systems become economically important.

AI 2040 Report

More importantly, the scenario says there is one job AI companies want to automate more than anything else: their own AI research.

At that point in AI 2040, full recursive self-improvement has not happened. Humans are still needed. But AI is already helping create future AI, and the labs appear to be getting closer to automating more of the process.

We obviously are not living in the AI 2040 scenario. Anthropic’s 30,000 internal agents are not the millions of economically useful agents described there, and neither Anthropic nor OpenAI claims to have fully automated AI research.

But some of the early pieces rhyme with it.

The scenario starts with AI agents becoming useful at digital work. Those agents then spread across companies. AI labs become particularly interested in automating AI R&D because improving that one job could help them improve every future model. Eventually humans become less important to the development loop.

In the AI 2040 timeline, the authors use 2030 as the point where AI R&D would become fully automated under their default technical scenario, potentially leading rapidly toward superintelligence if the feedback loop were allowed to run at full speed. Their proposed Plan A instead imagines intentionally delaying that process. Again, this is scenario planning, not an established forecast.

What interests me is not whether they got the year exactly right.

It is the sequence.

Agents become really useful. Companies deploy huge numbers of them. AI starts doing larger pieces of AI R&D. Humans gradually move from doing the work to supervising it. Eventually, AI can perform enough of the research loop that the limiting factor changes.