Mahal Systems found that upgrading AI models often degrades agent performance, even when using recommended successors. A study published on October 28, 2026, revealed that replacement models lowered scores in three of 12 upgrade tests, while 11 of 12 upgrades caused regressions on specific cases.
The research tested six micro-agents across paths like Claude Haiku 4.5 to 5.5 and GPT-5.4 nano to GPT-6 Luna. Mahal suggests that teams should treat model swaps as tests rather than automatic deployments, re-tuning prompts and enforcing critical requirements in code.
Mahal Systems found that recommended replacement models lowered the scores of existing agent configurations in three of 12 upgrade tests. Eleven of the 12 upgrades also regressed on individual cases, including upgrades that improved the overall score.
The study, published on October 28, 2026, tested six small agents across two upgrade paths: Claude Haiku 4.5 to Haiku 5.5, and GPT-5.4 nano to GPT-6 Luna. Mahal kept each agent’s surrounding software fixed while changing the model underneath it.
The practical finding goes beyond occasional regressions. A competing lab’s replacement model matched or beat the recommended same-lab successor in 10 of 12 comparisons under the study’s two-point tie rule. Staying with the original provider also offered little advantage in preserving which configurations worked best.
For teams facing a retirement deadline, that makes the recommended successor a candidate to evaluate, not an automatic deployment choice. The useful migration routine is straightforward: freeze an evaluation set, compare replacements, re-tune model-facing instructions, and enforce durable requirements in code.
What the Benchmark Tested
Mahal’s six micro-agents covered several common patterns in agent software:
Agent
Task
Recommender
Select a gift from a set of ten
Tool router
Turn a support message into one tool call
Text-to-SQL agent
Convert a business question into a SQL query
Retrieval agent
Answer a product-handbook question using retrieved passages
Extraction agent
Extract six invoice fields into JSON
Error-recovery agent
Handle failures in a clinic booking workflow
Each task had verifiable ground truth, with answers graded by code. Mahal explored 20 configurations per agent, varying a particular component while holding the rest fixed. Those variations included prompt structure, tool descriptions, schema context and retrieval settings.
The fixed harness matters. A harness is the software surrounding the model: it prepares inputs, exposes tools, handles responses and errors, and applies checks. Holding it constant helps isolate what happens when a team changes the model without rebuilding the application.
Across the suite, Mahal covered six areas of its agent-optimization framework: tools, prompts, context, orchestration, retrieval and verification. Memory, model selection, agent topologies and safety were outside that coverage. The study describes the task and configuration setup here.
The evidence is narrow: six small agents, 20 cases per task and reasoning disabled. The authors acknowledge that some two-point differences may be noise. They also say detailed per-call data and some methodology links are still to be published, limiting independent scrutiny. These are Mahal’s published measurements, with no independent replication supplied in the evidence.
That scope supports a migration warning. It cannot predict how every larger, reasoning-enabled agent will behave.
Average Gains Can Hide Broken Workflows
The two upgrade paths across six agents produced 12 same-lab comparisons. Mahal reports that:
Three upgrades lowered the existing configuration’s score.
One upgrade left it flat.
Eleven upgrades regressed on at least some cases.
Every agent improved under at least one lab’s successor.
Those findings can coexist. A replacement may solve several previously failed cases while breaking a smaller group that the old model handled correctly. Its average rises, but a previously reliable workflow becomes less dependable.
That distinction is important for shipped software. An overall score treats the benchmark according to its scoring rules; a business may place much greater weight on one failure, such as an incorrect tool action or an invalid invoice total.
My view is that the case-level result is the more useful operational warning. Three declining averages are easy to notice on a dashboard. Regressions hidden inside a higher average require a comparison of individual outcomes.
A migration evaluation should therefore record both the total score and the cases that changed from pass to fail. Teams can then decide whether a new failure is acceptable, repairable or a release blocker. Those decisions should be set by the application’s requirements, not by whether the replacement wins on average.
A Same-Lab Successor Offers Limited Continuity
Mahal’s cross-provider comparison weakens another common migration assumption: that staying with the same lab will preserve more of an agent’s existing tuning.
Under its two-point tie rule, the other lab’s new model matched or beat the recommended successor in 10 of 12 comparisons. That is a combined count of ties and wins, not evidence of ten outright victories.
The configuration rankings tell a related story. On five of the six agents, the same-lab successor preserved the ordering of configurations no better than the rival model did. On the sixth, it did only slightly better, according to Mahal.
A ranking measures something different from the final score. Suppose one prompt format consistently beats another on the retiring model. Preserving that ranking would mean the same format remains the better choice after migration. Losing it means the team’s previous tuning work no longer reliably identifies the strongest setup.
I would use that finding to justify adding another provider to the migration evaluation. It does not imply that switching providers is always worthwhile. Authentication, tool interfaces, data requirements, latency and operating cost still affect the decision. But provider continuity alone is a weak reason to skip the quality comparison.
What Survives Depends on Who Enforces It
Mahal found that prompts, schema descriptions and tool descriptions needed re-tuning after the swap. Retrieval settings, error handling and rules enforced in code were more stable.
The crucial distinction is between describing a requirement to the model and enforcing it in the application.
A tool description can tell the model when an action is permitted. Backend code can reject an action that violates the rule. Both are useful, but only the second remains an enforcement mechanism if the replacement model interprets the description differently.
The study includes an important boundary: code that depends on the model reading its inputs held up no better than prompts. Storing instructions in a source file does not make them durable if success still depends on the model following those instructions. Mahal’s analysis separates these components explicitly.
Validated schemas are a practical application of this distinction, though validation has its own limits. A schema can check required fields, types and allowed values. Separate application rules can check permissions or business constraints.
A valid structure does not guarantee a correct answer. An invoice extraction can contain all required fields and still report the wrong amount. Durable controls should enforce what they can verify, while task evaluations continue to test meaning and accuracy.
A Practical Model-Migration Checklist
Mahal summarizes its advice as:
“Treat the swap as a test, not an edit.”
The following workflow turns that principle into a repeatable release process.
Freeze the Evaluation Set and Current Baseline
Save the inputs, expected outcomes, grading logic and application configuration before changing models.
Include ordinary requests, known failures and cases with costly consequences. For a tool-using agent, checking the final response alone may miss a wrong intermediate action. For a retrieval agent, preserve the document versions and retrieval setup so a content change does not get mistaken for a model effect.
Record the current model identifier and relevant settings, alongside prompt versions, tool definitions and output schemas.
Set acceptance criteria before comparing candidates. A team might allow a modest average-score tradeoff but prohibit new failures involving authorization or irreversible actions. Those thresholds are application decisions, not findings from Mahal’s benchmark.
The goal is a stable comparison. Changing the model, prompt and test set at once makes an improvement or regression harder to explain.
Test the Named Successor and Another Provider
Run the same baseline cases through the recommended replacement and at least one plausible alternative provider.
Keep application behavior aligned where possible, and document necessary API differences. Record reasoning or effort settings explicitly. Mahal’s reasoning-disabled results should not be treated as predictions for a deployment using different inference settings.
Compare more than headline accuracy:
Which previously passing cases now fail?
Which failures disappear?
Does the agent choose the correct tool and arguments?
Are outputs valid and usable by downstream code?
Do latency and operating cost fit the application?
Repeated runs can help distinguish an intermittent failure from a consistent one. Preserve traces so a changed result can be connected to the model’s response, a tool call, retrieved context or error handling.
Re-Tune the Model-Facing Configuration
Use the existing prompt as a baseline, then test changes deliberately.
Mahal’s results suggest that prompt wording, tool descriptions and schema context are all migration-sensitive. A tool description that worked well on the retiring model may no longer produce the same routing behavior. A schema explanation may need clearer relationships or different examples.
Change one component at a time where practical. That makes the result interpretable and reduces the chance of keeping an unnecessary prompt rewrite because another change produced the gain.
Keep tuning cases separate from a held-out release set. Repeatedly editing instructions against the same small collection can improve those examples without producing a more reliable agent.
This distinction also improves the comparison between providers. The first pass measures compatibility with the current setup; the tuned pass measures what each candidate can deliver after adaptation.
Move Hard Requirements Into Enforced Controls
Identify requirements the application can check without trusting the model’s interpretation.
Depending on the workflow, these could include allowed tool names, required fields, parameter types, permitted destinations, authorization checks or business limits. Enforce them at the point where the application accepts an output or performs an action.
Keep explanations in prompts and tool descriptions too. They help the model choose a valid path before a validator rejects its output. The division of labor is useful: instructions guide behavior; code decides whether an action is permitted.
A validator failure also needs a defined response. Rejecting an invalid request is useful only if the surrounding workflow handles that rejection safely.
Save the Migration Result
Version the chosen configuration with its evaluation results, known regressions and release decision. Record why the team accepted the replacement, including tradeoffs that an aggregate score would hide.
Where the retiring model remains available, a limited rollout and rollback plan can reduce migration risk. After shutdown, switching back may no longer be possible, so the plan needs a fallback the application can actually use.
The lasting asset is the migration record: repeatable cases, meaningful traces and executable checks. Those give the next retirement a starting point that is stronger than a remembered impression of how the old model behaved.
Frequently Asked Questions
3 questions
1
Does a Higher Average Score Mean a Safe Upgrade?
No. Mahal found case-level regressions in 11 of 12 upgrades. Review newly failing cases and apply acceptance criteria based on their consequences.
2
Should Teams Replace All Prompt Rules With Code?
No. Prompts still guide behavior. Put requirements that software can reliably verify into executable checks, while retaining instructions that help the model produce acceptable actions.
3
Can the Old Prompt Still Be Useful?
Yes. Keep it as the baseline for measuring compatibility. Then compare tuned configurations separately so improvements can be traced to specific changes.
