Mahal Systems found that recommended replacement models lowered the scores of existing agent configurations in three of 12 upgrade tests. Eleven of the 12 upgrades also regressed on individual cases, including upgrades that improved the overall score.
The study, published on October 28, 2026, tested six small agents across two upgrade paths: Claude Haiku 4.5 to Haiku 5.5, and GPT-5.4 nano to GPT-6 Luna. Mahal kept each agent’s surrounding software fixed while changing the model underneath it.
The practical finding goes beyond occasional regressions. A competing lab’s replacement model matched or beat the recommended same-lab successor in 10 of 12 comparisons under the study’s two-point tie rule. Staying with the original provider also offered little advantage in preserving which configurations worked best.
For teams facing a retirement deadline, that makes the recommended successor a candidate to evaluate, not an automatic deployment choice. The useful migration routine is straightforward: freeze an evaluation set, compare replacements, re-tune model-facing instructions, and enforce durable requirements in code.
What the Benchmark Tested
Mahal’s six micro-agents covered several common patterns in agent software:
| Agent | Task |
|---|---|
| Recommender | Select a gift from a set of ten |
| Tool router | Turn a support message into one tool call |
| Text-to-SQL agent | Convert a business question into a SQL query |
| Retrieval agent | Answer a product-handbook question using retrieved passages |
| Extraction agent | Extract six invoice fields into JSON |
| Error-recovery agent | Handle failures in a clinic booking workflow |
Each task had verifiable ground truth, with answers graded by code. Mahal explored 20 configurations per agent, varying a particular component while holding the rest fixed. Those variations included prompt structure, tool descriptions, schema context and retrieval settings.
The fixed harness matters. A harness is the software surrounding the model: it prepares inputs, exposes tools, handles responses and errors, and applies checks. Holding it constant helps isolate what happens when a team changes the model without rebuilding the application.
Across the suite, Mahal covered six areas of its agent-optimization framework: tools, prompts, context, orchestration, retrieval and verification. Memory, model selection, agent topologies and safety were outside that coverage. The study describes the task and configuration setup here.
The evidence is narrow: six small agents, 20 cases per task and reasoning disabled. The authors acknowledge that some two-point differences may be noise. They also say detailed per-call data and some methodology links are still to be published, limiting independent scrutiny. These are Mahal’s published measurements, with no independent replication supplied in the evidence.
That scope supports a migration warning. It cannot predict how every larger, reasoning-enabled agent will behave.
Average Gains Can Hide Broken Workflows
The two upgrade paths across six agents produced 12 same-lab comparisons. Mahal reports that:
- Three upgrades lowered the existing configuration’s score.
- One upgrade left it flat.
- Eleven upgrades regressed on at least some cases.
- Every agent improved under at least one lab’s successor.
Those findings can coexist. A replacement may solve several previously failed cases while breaking a smaller group that the old model handled correctly. Its average rises, but a previously reliable workflow becomes less dependable.
That distinction is important for shipped software. An overall score treats the benchmark according to its scoring rules; a business may place much greater weight on one failure, such as an incorrect tool action or an invalid invoice total.
My view is that the case-level result is the more useful operational warning. Three declining averages are easy to notice on a dashboard. Regressions hidden inside a higher average require a comparison of individual outcomes.
A migration evaluation should therefore record both the total score and the cases that changed from pass to fail. Teams can then decide whether a new failure is acceptable, repairable or a release blocker. Those decisions should be set by the application’s requirements, not by whether the replacement wins on average.
A Same-Lab Successor Offers Limited Continuity
Mahal’s cross-provider comparison weakens another common migration assumption: that staying with the same lab will preserve more of an agent’s existing tuning.
Under its two-point tie rule, the other lab’s new model matched or beat the recommended successor in 10 of 12 comparisons. That is a combined count of ties and wins, not evidence of ten outright victories.
The configuration rankings tell a related story. On five of the six agents, the same-lab successor preserved the ordering of configurations no better than the rival model did. On the sixth, it did only slightly better, according to Mahal.
A ranking measures something different from the final score. Suppose one prompt format consistently beats another on the retiring model. Preserving that ranking would mean the same format remains the better choice after migration. Losing it means the team’s previous tuning work no longer reliably identifies the strongest setup.
I would use that finding to justify adding another provider to the migration evaluation. It does not imply that switching providers is always worthwhile. Authentication, tool interfaces, data requirements, latency and operating cost still affect the decision. But provider continuity alone is a weak reason to skip the quality comparison.
What Survives Depends on Who Enforces It
Mahal found that prompts, schema descriptions and tool descriptions needed re-tuning after the swap. Retrieval settings, error handling and rules enforced in code were more stable.
The crucial distinction is between describing a requirement to the model and enforcing it in the application.
A tool description can tell the model when an action is permitted. Backend code can reject an action that violates the rule. Both are useful, but only the second remains an enforcement mechanism if the replacement model interprets the description differently.
The study includes an important boundary: code that depends on the model reading its inputs held up no better than prompts. Storing instructions in a source file does not make them durable if success still depends on the model following those instructions. Mahal’s analysis separates these components explicitly.
Validated schemas are a practical application of this distinction, though validation has its own limits. A schema can check required fields, types and allowed values. Separate application rules can check permissions or business constraints.
A valid structure does not guarantee a correct answer. An invoice extraction can contain all required fields and still report the wrong amount. Durable controls should enforce what they can verify, while task evaluations continue to test meaning and accuracy.
A Practical Model-Migration Checklist
Mahal summarizes its advice as:
“Treat the swap as a test, not an edit.”
The following workflow turns that principle into a repeatable release process.
Freeze the Evaluation Set and Current Baseline
Save the inputs, expected outcomes, grading logic and application configuration before changing models.
Include ordinary requests, known failures and cases with costly consequences. For a tool-using agent, checking the final response alone may miss a wrong intermediate action. For a retrieval agent, preserve the document versions and retrieval setup so a content change does not get mistaken for a model effect.
Record the current model identifier and relevant settings, alongside prompt versions, tool definitions and output schemas.
Set acceptance criteria before comparing candidates. A team might allow a modest average-score tradeoff but prohibit new failures involving authorization or irreversible actions. Those thresholds are application decisions, not findings from Mahal’s benchmark.
Frequently Asked Questions
3 questions
1Does a Higher Average Score Mean a Safe Upgrade?
No. Mahal found case-level regressions in 11 of 12 upgrades. Review newly failing cases and apply acceptance criteria based on their consequences.
