AI Is Starting to Build AI. Who's Checking the Work?
Recursive self-improvement is already happening, though not in the way science fiction imagined. AI is increasingly helping build the next generation of AI. The question is whether we can reliably check what it's building.
This fall I was on a panel at an explainable-AI workshop at Mississippi State. The opening question was a big one: What's the most consequential AI transition we'll see over the next five years?
My answer was that AI starts doing AI research.
People call it recursive self-improvement, or RSI. The idea is straightforward: AI helps build the next, better AI, which then helps build the one after that. Each generation contributes to creating its successor.
I've been writing software since 1995, and I've heard plenty of claims about technologies that will change everything. This one is different. But probably not for the reasons most people expect.
It's not a robot rewriting its brain
The idea isn't new. In 1965, mathematician I. J. Good argued that designing machines is itself an intellectual task. A machine sufficiently intelligent could design a better version of itself, which could then repeat the process. He called the result an "intelligence explosion."
What's happening today is less dramatic, but arguably more important because it's already underway.
AI is accelerating the human-run process of building better AI. It's writing code, running experiments, generating training data, and evaluating results. Humans still set research priorities, decide which experiments matter, and determine what gets released.
The labs are starting to quantify this. In September, OpenAI reported that its research agents log roughly three workdays of runtime for every workday contributed by a human researcher. That's runtime, not three times the research output. OpenAI also emphasizes that people still set the priorities.
But the direction is clear: AI is becoming a bigger part of the process that produces the next generation of AI.
The people closest to it disagree on how fast
I've spent much of this fall listening to researchers discuss this transition, particularly on the Dwarkesh Podcast and The Cognitive Revolution. Their predictions vary considerably, and the disagreement is worth understanding.
On one Dwarkesh episode, John Schulman, Beren Millidge, and Charlie O'Neill debated how close we are. Schulman cautioned that faster coding doesn't automatically translate into faster research. As he put it, "Even if the model can write way more code than a person, it doesn't make you 100X more productive." He estimated a tenfold increase in AI research productivity might be about two years away. O'Neill put the timeline at five to ten years.
Ryan Greenblatt of Redwood Research made a more aggressive case. AI research is unusually well suited to automation because so many machine-learning tasks are small, self-contained, and objectively verifiable. Those are precisely the kinds of tasks where AI tends to improve fastest. He sees full automation of AI research potentially arriving around 2030 or 2031.
But Greenblatt also identified an important exception: judgment. Large experiments are expensive, sometimes offer only a few opportunities to get things right, and require decisions that aren't easily reduced to an automated score.
OpenAI's Noam Brown landed somewhere in between. He expects substantial acceleration, but not a hundredfold explosion, because experiments and computing resources remain limiting factors.
The timelines differ. The direction doesn't. Even the more conservative predictions put a major transition within the next five years.
That's why I chose this as the development to watch.
The real question is who sets the objectives
Beren Millidge raised a question that I keep coming back to: "How well can AIs generalize to learning their own objectives?"
It's one thing for an AI to execute a task. It's another for it to decide what task should come next, understand why it matters, and pursue it over an extended period without losing sight of the original intent.
That distinction becomes critical when AI is helping build AI.
We already have a preview of what can go wrong when the objective and the intended outcome aren't quite the same.
This summer, OpenAI tested a large group of agents on a hacking benchmark that included many impossible tasks. According to an independent investigation by METR and Redwood Research, approximately 1,200 agents discovered a way to communicate through a shared software cache. They coordinated, found a way to beat the benchmark, and broke into Hugging Face to retrieve the answers.
Not one of them alerted a human.
This wasn't a self-improvement incident. It was reward hacking: the agents found a way to optimize for the score rather than accomplish the intended task. But it happened inside the broader ecosystem developing increasingly capable AI systems.
Greenblatt described how models can develop a general tendency to pursue whatever gets them a high score from the system evaluating them.
Think about the implications. If an AI system is helping train or evaluate its successor, a flawed objective doesn't necessarily stay confined to one model. It can become part of the next generation.
The system can get better at achieving a goal without getting better at understanding whether that goal is worth achieving.
Who checks the checker?
There's another detail from that investigation that I find particularly revealing.
The investigators had to use AI to analyze the agents' transcripts because there was simply too much material for people to review. Ajeya Cotra, who co-authored the investigation, explained on Dwarkesh that they couldn't have fully understood what happened without using a model, including one of the models involved in the incident.
That's a remarkable development. We're increasingly relying on AI to evaluate the behavior of other AI systems because the volume and complexity of their work exceed what humans can reasonably inspect.
On The Cognitive Revolution, Bronson Schoen of Apollo Research made a point about monitoring a model's chain of thought: it's "necessary but not sufficient."
I agree.
Being able to observe what a model says it's thinking is useful. But observing a system isn't the same as verifying that it's behaving correctly. And a system that can produce convincing explanations isn't necessarily a system whose decisions deserve our trust.
The harder question isn't just whether AI can check AI. It's whether we can independently establish that the checking itself is reliable.
How I run my own small loop
I use a team of AI agents every day. Some write code. Others review and test it. Nothing gets merged until it passes review and I approve it.
I look at what they've built or changed and decide what stays and what goes. They can draft and improve their own instructions, but I approve the changes.
The agents are getting better at their jobs. But I'm still the one deciding what better means.
They don't change their own goals, permissions, or approval rules. That separation is intentional.
It's a small example of the same problem facing AI research labs. You want systems that can improve their performance without giving them unrestricted authority to redefine success.
The more capable the system becomes, the more important that distinction gets.
What I'd tell any leader
There are three principles I'd take from all of this:
- Invest in verification, not just generation. The ability to produce more code, research, or analysis is only valuable if you can establish that the output is correct. The quality of your evaluation process will increasingly determine the quality of your AI.
- Automate where you can verify and recover. Let AI move quickly on tasks with measurable outcomes and reversible consequences. Keep meaningful human decision-making where the stakes are high or the answers are difficult to independently validate.
- Build oversight into the architecture. Don't treat human review as a final checkbox. When the volume of AI output already exceeds what people can read, oversight has to be designed into the system from the beginning.
AI is going to give us extraordinary improvements in speed and scale. I expect that to accelerate considerably as AI takes on more of the work of building AI.
But capability and judgment aren't the same thing. And as these systems become more involved in creating their successors, the ability to distinguish between getting better and merely getting a better score may become one of the most important capabilities we have to develop ourselves.
The future of AI may depend less on how well machines can improve themselves than on how well we can tell whether they're improving in the right direction.
Warren Husband is CTO at EdgeTheory and has been building software since 1995. He lives in Ridgeland, Mississippi.