The highest-scoring LLM in Artificial Analysis’s new FrontierCode leaderboard clears every blocking merge-quality requirement in just 58% of attempts. In the remaining 42%, it fails at least one requirement.
FrontierCode asks whether an agent’s patch meets standards defined by maintainers of real open-source repositories, including correctness, test quality, scope discipline and codebase conventions. Passing functional tests alone is not enough. The result is more specific than “AI coding agents write bad code.”
Artificial Analysis launched its independent FrontierCode 1.1 evaluation on October 12, 2026. Claude Opus 5.5 leads on mean score, but GPT-6.1 Sol comes within roughly three percentage points at less than one-twelfth of the reported cost per task.
Teams choosing coding models face tightly grouped leaders, large cost differences and limited reliability under demanding merge criteria. Before using the rankings to make a purchasing decision, they need to understand what the scores measure.
A 52.2% Score Is Not a 52.2% Pass Rate
The FrontierCode leaderboard reports Claude Opus 5.5 at 52.2%, followed by GPT-6.1 Sol at 49.1% and Claude Fable 5.1 at 49.0%. These are mean scores, not the percentage of attempts that pass.
FrontierCode reports both metrics because they answer different questions. An attempt passes when it satisfies every blocking criterion: requirements that the benchmark’s maintainers consider hard stops during review. Failing any blocker earns a zero, as does being flagged for looking up a task’s solution.
Attempts that clear all blockers receive a score between 70% and 100%, depending on how well they satisfy additional, non-blocking criteria. The headline score combines the frequency of acceptable attempts with the quality of those that pass. Opus can therefore have a 58% pass rate and a 52.2% mean score because its passing attempts do not necessarily receive full marks.
The 42% failure rate does not establish that 42% of Opus patches passed their functional tests and then failed solely on style or maintainability. Failures can arise from correctness checks, quality requirements or other blocking rules. The benchmark measures their combined effect.
Maintainers Define More Than Functional Correctness
Cognition developed FrontierCode with maintainers of 36 open-source repositories. According to its benchmark introduction, each task received more than 40 hours of maintainer effort, with repository-specific expectations translated into evaluation criteria.
The verifier uses six types of checks:
- Classical tests inject tests into the repository to check behavior.
- Command checks run commands for requirements such as builds, linting and regression safety.
- Reverse-classical tests run the agent’s submitted tests against the base commit, checking whether they fail before the fix.
- Adaptive classical tests use an LLM to adapt reference tests or application code to accommodate the implementation.
- Scope checks assess boundaries such as permitted files and diff size.
- Prompt review uses an LLM to evaluate the patch against natural-language criteria.
The reverse-classical check addresses a practical weakness: added tests provide little useful evidence if they also pass without the proposed fix. Scope checks examine concerns such as changes to unrelated files or unnecessarily broad patches.
Together, these checks provide more information than a functional pass/fail result alone. A maintainer must consider whether a contribution solves the problem without creating avoidable review and maintenance work.
Still, “mergeable” here means satisfying an encoded evaluation. Maintainers helped define the standards, but the independent leaderboard is not a record of humans reviewing and accepting every generated patch.
Three Runs Measure Repeated Performance, Not a Lucky Success
Artificial Analysis evaluates FrontierCode 1.1 Main, a 100-task subset of the benchmark’s full 150-task collection. Each model runs every task three times in the same Stirrup agent harness, with a maximum of 250 turns per attempt.
The resulting 300 attempts per model are scored by averaging the three attempts for each task, then averaging across tasks with equal weight.
This is not a “best of three” score. If a model succeeds once and fails twice, the two failures remain in the average; the existence of one acceptable patch does not earn full task credit. That makes the result more relevant to teams that care about repeatability.
Using the same harness holds tool orchestration constant, making the comparison more controlled than one in which each provider submits a different coding-agent product. The tradeoff is narrower applicability: these results describe models operating within Stirrup, under this turn budget and evaluation configuration. They do not establish the same ordering for every commercial coding agent, custom scaffold or repository workflow.
Opus and Fable are listed at maximum effort with default fallback, while GPT-6.1 Sol is listed at maximum effort. Those configuration labels belong in comparisons; the results are not configuration-free properties of the models.
At launch, nine tasks were passed by no model on any attempt. No task was passed by every model on every attempt, showing both the benchmark’s difficulty and the variability behind its aggregate rankings.
Private Tasks Reduce One Risk but Limit Inspection
The tasks are private to reduce contamination. This makes it harder to train directly against the evaluation, while limiting outside inspection of task content and its fit with a team’s workload.
The underlying solutions may still be public. Cognition explains in its FrontierCode 1.1 update that tasks originate from real open-source changes, so answers may be available in upstream pull requests, repository mirrors or later package releases.
Version 1.1 distinguishes legitimate internet use, such as reading documentation, from retrieving a task’s solution. Agents receive fair-use instructions, and a trajectory scanner flags prohibited solution lookups. Artificial Analysis assigns flagged attempts zero.
This rule measures whether an agent can solve a problem without retrieving an existing answer. It also introduces a reason for failure beyond patch quality, so a zero should not automatically be read as a maintainability defect.
The grading standard has changed, too. Cognition says its 1.1 audit demoted 75 overly strict blocking criteria to non-blocking status. A score reflects the rubric in force, not an immutable definition of good code, which makes the benchmark version essential context.
GPT-6.1 Sol Narrows the Gap at Far Lower Cost
The leading results show a much wider spread in cost than in mean score:
| Model configuration | Mean score | Reported cost per task |
|---|---|---|
| Claude Opus 5.5, max with default fallback | 52.2% |
Sources
- independent FrontierCode 1.1 evaluationartificialanalysis.ai
- FrontierCode leaderboardartificialanalysis.ai
- benchmark introductioncognition.com
- FrontierCode 1.1 updatecognition.com
