Opus 5.5 Study Finds No Significant Accuracy Decline | Zeniteq
Loading
Loading article
Opus 5.5 Study Finds No Significant Accuracy Decline
LiveNerf’s first 30 days show no statistically significant accuracy degradation, while controlled tests suggest token counts could flag reasoning-effort changes more clearly.
LiveNerf’s 30-day analysis of Opus 5.5 found no statistically significant accuracy decline across its frozen panel of 78 challenging questions. While the model’s performance remained stable, the study’s sensitivity was limited, with a minimum detectable effect of 6.6 percentage points. The project notably identified output-token counts as a potential early warning system, as controlled tests showed that reduced reasoning effort and model substitutions significantly shortened response lengths. These findings suggest that while accuracy metrics provide a baseline, monitoring token distributions offers a more sensitive, albeit ambiguous, indicator of behavioral shifts in AI services.
Opus 5.5 scored 58.4%, 59.2% and 57.1% across three consecutive ten-day windows in LiveNerf’s public monitoring experiment. Neither comparison with the initial baseline crossed the project’s preregistered threshold for declaring a change.
The finding comes from LiveNerf’s October 27, 2026 report, published under the name NinjaHawk. By that date, the project had collected 34 consecutive days of measurements, with the first 30 available as three complete windows.
The important qualification is statistical power: the experiment’s estimated minimum detectable effect was about 6.6 percentage points per ten-day window. Smaller regressions could escape detection.
A separate validation exercise produced a useful lead for future monitoring. Deliberately lowering reasoning effort reduced output-token counts much more clearly than it changed accuracy. That makes token distributions worth watching, although a shorter response alone cannot identify what changed behind the service.
How LiveNerf Measures Opus 5.5
LiveNerf tracks whether a model’s served behavior changes after release. Its public repository describes an effort to replace arguments about silent degradation with a fixed test panel, repeated measurements and public logs.
The project began collecting daily data on September 24, roughly two and a half days after Opus 5.5’s launch, according to the author. That timing matters: this is a launch-period baseline, not a measurement of the model’s very first hours.
To build the panel, the project screened 2,336 questions, sampling each four times. The questions came from:
GPQA Diamond
MMLU-Pro
Competition mathematics
AIME 2025–26
Opus 5.5 answered about 93% correctly on the first attempt during screening. Roughly 97% of the questions were either always right or always wrong across the four samples.
Those consistently solved or consistently missed questions offered little room to observe a marginal change. LiveNerf instead selected 78 questions the model sometimes answered correctly, then froze that panel. Each daily run also includes a 12-question control arm.
This selection explains why the monitoring accuracy sits near 58%, despite the much higher screening score. The panel deliberately concentrates on unstable questions. Its score is a measurement instrument, not an estimate of how often Opus 5.5 succeeds across ordinary work.
The repository documents a pinned Claude Code CLI, version 2.1.280, alongside fixed prompts, grading and harness settings. The measurement plan calls for explicitly fixing effort, freezing the system prompt and isolating runs from tools, hooks and project configuration.
The basic principle is straightforward: hold the controllable inputs steady, then measure variation in the outputs. Pinning the CLI removes one possible moving part; it cannot freeze the service behind it.
The main route uses a Claude Max subscription through headless Claude Code. There was a documented exception: day 19 used Anthropic API credits after the subscription’s weekly allowance was exhausted. The repository also records a crashed run that was excluded and rerun, plus other deviations.
Those records are valuable because operational exceptions can affect interpretation. Public logging makes them inspectable; it does not make every run identical.
Each complete ten-day window contains 780 measured-panel responses: 78 questions run once daily for ten days. Neither later window met the preregistered threshold for a change.
The author summarizes the result narrowly:
“we haven't detected a sustained change in accuracy on this particular panel.”
That wording carries more information than a claim that Opus 5.5 stayed unchanged. A null result describes what the experiment detected under its design. It leaves open changes that are too small, too brief or outside the selected tasks.
The estimated 6.6-percentage-point minimum detectable effect gives readers a sense of that sensitivity. For scale, a drop of that size from the 58.4% baseline would put accuracy around 51.8%.
That is a meaningful decline. It also leaves substantial space for smaller changes that could matter to users.
Minimum detectable effect is not a sharp boundary where every larger change will be found and every smaller one will be missed. It summarizes the experiment’s detection capability under its statistical assumptions. The practical point is that this setup has limited power to rule out modest regressions.
I’d read these results as a bounded answer to the degradation claim: LiveNerf found no statistically significant sustained accuracy decline on its frozen panel over the completed windows. The evidence supports that statement. A blanket assurance about every Opus 5.5 capability would go beyond the measurement.
Token Counts Gave a Clearer Effort Signal
LiveNerf also performed positive-control experiments: deliberately change a setting, then check whether the instrument notices.
The accuracy estimates were noisy. Output tokens fell substantially under both reduced-effort settings.
A token count provides information that pass/fail grading throws away. Two responses can both be correct while using very different amounts of output. A model can also produce a much shorter unsuccessful attempt without changing its already-incorrect grade. Tracking tokens therefore gives the monitor another dimension of behavior to examine.
The project’s observation is useful: a change in effort may become visible in output-token distributions before there is enough evidence to declare an accuracy decline.
There is an important timing qualification, though. These were controlled comparisons between settings, not an observed sequence in which a provider changed effort and tokens moved first. Earlier warning is a plausible monitoring implication, not a demonstrated incident.
The ordinary daily measurements showed relatively stable median output tokens: 588, 606 and 594 across the three windows. They did not show reductions resembling the validation’s 26% or 62% cuts.
A Token Alert Is Not a Diagnosis
Token counts are useful precisely because they capture behavior beyond whether an answer passes. That breadth also makes them ambiguous.
LiveNerf’s model-substitution test produced fewer tokens too. When the project substituted Opus 5 for Opus 5.5, output tokens fell by about 23%. That is close in magnitude to the medium-effort experiment’s 26% reduction.
The comparison yields a practical lesson: a token shift can flag a change without identifying its cause. These validation results provide two different interventions that reduced output length.
My view is that token monitoring earns a place beside accuracy monitoring because it can trigger investigation sooner. It should not be treated as a fingerprint proving lower effort or a particular model substitution. A useful alert says that behavior has moved; determining why requires further evidence.
The Model-Swap Test Exposes a Detection Limit
The Opus 5 substitution test also challenges one of the strongest claims a monitoring project might be tempted to make.
LiveNerf reported an accuracy difference of −3.8 ± 6.3 percentage points when substituting Opus 5 for Opus 5.5. At the project’s preregistered 99% confidence threshold, that validation could not reliably distinguish the models on accuracy.
The author explicitly acknowledges the consequence: an instrument that cannot reliably detect that particular substitution should not be used to claim it has not happened.
That limit is specific to the tested sample size and setup. The repository notes that a ten-day window contains about 2.5 times as many samples as the validation exercise. More observations can improve sensitivity, but the project has not demonstrated that those windows are sufficient to detect a substitution of that size.
This is one of the report’s most useful disclosures. It tests the monitor’s capabilities against a known intervention and reports where the accuracy measurement falls short. Without that check, a stable score could invite much stronger conclusions than the instrument can support.
The Frozen Questions Have Problems Too
An audit of 80 reviewed questions, including two subsequently excluded items, found eight answer keys that appeared incorrect and about 30 ambiguous questions, according to the study report.
The selection method helps explain the concentration of troublesome items. Questions chosen because a strong model sometimes passes and sometimes fails can include genuinely difficult problems, but they can also include unclear wording or unreliable grading targets.
An incorrect key can mark a correct response as a failure. Ambiguity can make a change in interpretation look like a change in capability. Those are measurement problems even when every run uses exactly the same grader.
The project documented the issues and retained the frozen panel, with a sensitivity analysis to examine their effect. Quietly editing the panel after seeing results would create a different problem: the measuring instrument itself would change during the series.
Keeping the panel fixed preserves the comparison, while the audit makes clear why that comparison should not be confused with a clean measure of general intelligence or real-world usefulness.
What Claude Code Users Can Take From the Study
LiveNerf measures a deliberately constrained slice of Opus 5.5 behavior. It does not directly test coding agents, long conversations or every capability people rely on in Claude Code.
The author also identifies product factors that could affect someone’s experience without causing an obvious shift on this panel:
System prompts
Context management
Tool behavior
Inference settings
That distinction matters for interpreting complaints. A user’s coding workflow may depend heavily on tool selection and accumulated context, while this experiment prioritizes stable prompts and repeatable grading. Either could change without the other showing a clear signal.
Ordinary sampling variation is another possible explanation for uneven experiences. Repeated measurements help separate that noise from sustained movement, within the experiment’s sensitivity.
LiveNerf extended its planned run from 30 to 45 days. Longer observation adds further comparison windows, though extending the calendar alone does not establish sensitivity to every type of change.
The most actionable contribution is the combination of accuracy and token measurements. Accuracy tracks success on the selected tasks; tokens can expose behavioral movement that a binary grade misses. The validation results also show why any alert needs follow-up: reduced effort and a different model both shortened outputs.
Frequently Asked Questions
3 questions
1
Did LiveNerf Find That Opus 5.5 Degraded?
No statistically significant accuracy degradation was detected in the first 30 days. The experiment could still miss smaller regressions or changes outside its selected tasks.
2
Why Monitor Output Tokens Alongside Accuracy?
Controlled effort reductions cut output tokens by 26% at medium effort and 62% at low effort. Those changes were clearer than the noisy accuracy shifts, making tokens a useful additional monitoring signal.
3
Can the Study Rule Out a Model Swap?
No. Its validation could not reliably distinguish Opus 5 from Opus 5.5 on accuracy at the preregistered threshold. The project has not shown that its larger daily comparison windows can detect a substitution of that size.
