LLM Agent Tackles CIFAR-10 With $0.0085 in Tokens

Lex Toumbourou gave an LLM-driven agent a training job without supplying a dataset URL, script, or model architecture. The agent found CIFAR-10, trained an image classifier, and reported 81.41% accuracy on 10,000 held-out images. His reported model-token bill for the final attempt was about $0.0085.

The result wasn’t entirely hands-off. Toumbourou explicitly allowed the dataset hosts through the sandbox’s network restrictions and gave the agent one nudge toward a faster download.

His agent-harness walkthrough, published October 11, 2026, presents the experiment as an attempt to satisfy a challenge Ian Goodfellow discussed in a 2019 interview. It also offers a step-by-step explanation of the software surrounding a coding model, which is more useful than a declaration about intelligence.

That software determines what an agent can execute, which information it retains, where its credentials live, and when a human must intervene. The CIFAR-10 run makes those responsibilities concrete. It remains one author-documented experiment, however, rather than an independently reproduced benchmark.

The Challenge Was to Complete a Training Job

Toumbourou’s test asked the finished agent to find CIFAR-10, train a classifier, report held-out accuracy, and leave both the code and the trained model behind. He did not supply a dataset URL, training script, or model architecture.

A plausible explanation of convolutional networks wouldn’t satisfy that request. The agent had to obtain data, produce runnable software, execute training, and return a measurable result with artifacts.

After 12 CPU training epochs, the reported outcome was 8,141 correct predictions out of 10,000 held-out images. Two models had different roles: the LLM organized and wrote the work; the resulting image classifier performed the CIFAR-10 predictions.

Toumbourou’s Goodfellow framing supplies historical motivation, not a standardized pass/fail protocol. Completing this particular workflow provides evidence of useful coding-agent behavior. It does not, by itself, establish general intelligence.

Four Tools Make the LLM’s Decisions Executable

The walkthrough defines a harness as everything in an agent that isn’t the model. Its smallest version builds context, calls the LLM, executes a requested tool, and returns the tool’s output to the conversation.

Conceptually, the loop looks like this:

while True:
    reply = model(context)
    if not reply.tool_call:
        break
    result = run_tool(reply.tool_call)
    context.append(result)

Those simple-looking operations contain much of the engineering work.

Toumbourou puts the model behind an adapter, allowing the rest of the harness to use a common interface instead of a vendor client directly. His example retains provider-specific response items for subsequent turns and tracks token cost. Retaining that state preserves the conversation structure the API expects in the next request.

For local actions, he implements four tools: read_file, write_file, search_replace, and bash. Together, they let the model inspect files, create programs, modify existing code, and run commands.

The shell tool delegates to Python’s subprocess.run, with captured output and a timeout. It returns an exit code, standard output, and standard error, giving the LLM evidence to correct a failed command instead of guessing what happened.

Tool output is also truncated. An unrestricted training log could consume the model’s available context without helping it decide what to do next. Details like these distinguish an executable agent from a chat interface with a terminal attached.

Memory, Skills, and Sub-Agents Add Structure Around the Loop

The guide covers context management, safety controls, skills, orchestration, and extension surfaces alongside tool execution. These components solve different problems and shouldn’t be treated as interchangeable additions.

As a task grows, context management determines what the model can still see. Tool results provide feedback, but preserving every byte indefinitely is neither necessary nor practical. A harness needs to retain information relevant to the next decision while controlling the volume of accumulated output.

Skills supply reusable task guidance and procedures without requiring the harness itself to encode every workflow. Sub-agent orchestration introduces a delegation boundary: each assignment must be explicit and return something the coordinating agent can use.

The model adapter also exposes optional web search. External information can help an agent locate resources, though access to that information does not grant permission to download or execute something.

The implementation guide covers these capabilities; it does not establish that every component contributed to the CIFAR-10 success. A single end-to-end result cannot isolate the contribution of memory, skills, search, or sub-agents. For developers, the value is in making those responsibilities visible and separable.

The Permission Gate Is Not the Sandbox

The walkthrough’s shell tool can execute commands on the machine where it runs. The permission gate and execution environment are therefore central design decisions.

A permission gate is the harness’s decision point for whether a requested action may proceed. The LLM proposes an action; software outside the model must apply the relevant policy. Asking the model to behave safely does not replace an enforceable check.

Isolation addresses a different question: what resources can an approved or mistakenly permitted action reach?

For the reported CIFAR-10 run, Toumbourou used a Docker sandbox and mounted the harness read-only. He kept the OpenAI API key on the host through Docker’s proxy instead of placing it inside the agent’s execution environment. He also explicitly allowed only the dataset hosts.

Each boundary has a distinct purpose:

• The read-only harness mount restricts changes to the mounted implementation.
• Host-side credential handling avoids handing the API key directly to generated training code.
• The network allow-list limits external destinations available to the workload.
• The permission gate controls whether requested actions proceed.

The allow-list also affects task completion. Finding a dataset address does not make that destination reachable, so a network restriction can produce an apparent agent failure even when the proposed download is reasonable.

None of this proves arbitrary generated code is safe. A coding agent’s security cannot be judged solely by its system prompt: the model’s instructions, action permissions, filesystem access, credentials, and networking are separate controls.

Reproducing the Run Means Recording the Human Help

Toumbourou’s published code directory includes harness.py, test_harness.py, trace_run.py, and a cifar10-first-run directory. Developers have an implementation to inspect, not only a screenshot of a successful answer.

A meaningful reproduction should preserve the experiment’s boundaries as well as rerun the training:

1. Record the code revision and model configuration. The repository links point to main, so a later checkout may differ from the published experiment.
2. Recreate the isolation setup. Keep the harness read-only, retain host-side credential handling, and use the documented network restrictions.
3. Start with the same level of task information. Ask the agent to locate CIFAR-10, train, evaluate, and save artifacts without supplying the missing URL, architecture, or script.
4. Log every intervention. Record network permissions separately from any advice that helps the agent choose a better action.
5. Verify the deliverables. Check the generated program, evaluation output, and saved model rather than accepting the final conversational claim alone.

Running the saved training code checks whether that code reproduces the classifier result. Starting a fresh agent session checks whether the harness can complete the workflow again. These are two different experiments, and success in the first does not establish reliability in the second.

The download nudge belongs in that record. Reproducing the documented assisted run and attempting a fully unattended run are different tests. Neither should quietly be presented as the other.

The Accuracy and Token Bill Answer Different Questions

CIFAR-10 contains 60,000 small color images across ten classes, with 50,000 training images and 10,000 held-out images. It is an established image-classification task, not a new benchmark created for this agent.

For perspective, Keller Jordan’s 2024 CIFAR-10 research reports 94% accuracy in 3.29 seconds on a single NVIDIA A100 GPU. Its timing protocol permits GPU warm-up before access to the training data. The hardware and training setup differ from Toumbourou’s CPU run, so this is not a direct speed comparison.

It does establish that 81.41% is not a frontier classification result. The useful finding is that the agent assembled and executed a working training pipeline under the reported constraints.

The $0.0085 figure measures only the author-reported model-token cost of the final attempt. It excludes CPU compute, development time, earlier attempts, web-search charges, and the human intervention. The walkthrough’s configurable token-cost tracker is an accounting mechanism, not an independent all-in billing audit.

Inexpensive model calls can sit inside a much larger engineering system. Tools supply feedback; permissions control actions; isolation protects resources; humans resolve some operational obstacles. Toumbourou’s disclosure of the nudge makes the result more interpretable by showing where that system still needed help.

For developers building an LLM agent, the reproducible target is an auditable workflow with explicit execution boundaries and counted interventions. Another fresh run would help establish reliability. A lower token bill alone would not.