Snowflake introduced SnowPR on October 27, 2026, describing a coding-agent benchmark built from 65 merged pull requests across 11 product areas. Agents receive the code from before each change and a task description, then face hidden tests drawn from the original pull request. Most tasks come from Snowflake’s 43-million-line Java and C++ monorepo. Snowflake’s announcement presents the benchmark as both an evaluation of outside agents and a development tool for its own Cortex Code harness.
The headline result needs two different readings. Snowflake’s internal REM mode, paired with opus-5, solved 62 of 65 tasks at least once across three attempts. But the four tested Opus harnesses had closely grouped single-attempt pass rates. Their clearer difference was cost.
For engineering teams, SnowPR’s most useful contribution is its evaluation design: real repository work, hardened hidden tests and explicit accounting for the harness around the model. The published evidence supports several practical choices about building such a benchmark, without supplying a universal agent ranking.
Why Monorepo Tasks Change the Evaluation
Snowflake’s repository differs substantially from the projects underlying SWE-bench. Its announcement compares 63,000 non-test source files and 17 million lines of non-test code with the average SWE-bench repository. The 43-million-line headline describes the larger repository; the 17-million-line figure is the narrower, non-test comparison.
| Measure | SnowPR or Snowflake monorepo | SWE-bench comparison reported by Snowflake |
|---|---|---|
| Non-test source files | 63,000 | Snowflake has 21 times more |
| Non-test source lines | 17 million | Snowflake has 39 times more |
| Average files changed by the reference fix | 4.0 | 1.7 |
| Average lines changed by the reference fix | 262 | 33 |
These are Snowflake’s published comparisons, not independent measurements. They describe a workload with more navigation, larger patches and more opportunities to miss a dependency.
Of the 65 tasks, 62 come from the main monorepo and three from a separate Go developer-tools repository. The workload includes 52 Java control-plane tasks, eight C++ execution-engine tasks and two spanning Java and C++.
The surrounding infrastructure matters too:
- One Bazel build graph connects Java and C++. Snowflake says a C++ task requires about 21,700 build actions before the agent starts and 32 GB of memory.
- Tests can live far from implementation code. In 33 tasks, graded tests sit in a shared test tree while the changed code belongs to another module.
- Grading can cross build targets. Thirty tasks require two or more Bazel targets.
- Tests can depend on stateful services. In 39 tasks, graded tests start their own FoundationDB instance.
My reading is that repository navigation and build knowledge belong inside the evaluation. An agent that produces a plausible method but cannot locate the relevant test target has failed a consequential part of enterprise development.
Snowflake also reports a 55% trial pass rate for changes spanning Java and C++, compared with 80% for Java-only work. That is a useful signal for choosing evaluation tasks, although it does not isolate language boundaries as the cause: the task groups also differ in other ways.
Hidden Tests Need Their Own Quality Checks
SnowPR grades two things: new behavior must work, and existing behavior must remain intact. A task needs at least one new test case covering the main change and at least 10 regression cases. Where a pull request lacks enough regression coverage, the benchmark borrows existing tests that pass before and after the reference change, or drops the task.
Across the benchmark, grading covers 147 Bazel targets, 676 new-behavior cases and 2,890 regression cases. After the agent stops, the grader injects the hidden tests and runs them on Snowflake’s remote build farm. Every graded test must pass for the trial to pass, according to the opened reprint of Snowflake’s methodology.
The hardening process adds several safeguards:
- Check the reference patch. The real fix must pass, while an empty change must fail.
- Validate the task description. Eleven automatic checks include checking that the description names new classes or methods called by hidden tests.
- Use human and adversarial review. Snowflake reports 84 confirmed findings across 49 tasks, including eight blocking issues.
- Keep the LLM judge out of scoring. It filters candidate pull requests; executable tests determine outcomes.
That last distinction matters. Snowflake reports that an LLM judge’s predicted pass rates correlated weakly with measured results: Spearman correlation was +0.18 across 51 tasks. A useful task-selection assistant was a poor substitute for running the code.
Task Wording Is Part of the Measurement
Snowflake found that 10 agents passed 83% of trials using one engineer’s description of a pull request, versus 57% using another description, with the same hidden tests and reference fix.
That 26-percentage-point difference is more consequential than the small gaps among the leading Opus harnesses. It shows why benchmark authors need to audit specifications alongside tests.
I’d freeze task descriptions before comparing agents and review whether hidden tests demand interfaces the description never identifies. Otherwise, a benchmark can reward lucky interpretation of an underspecified request. The finding supports that precaution; it does not establish which description was better.
Hidden tests also have a scope limit. Snowflake grades tests associated with the original pull request because broad dependency queries can become impractical. In one example, Bazel identified 23,946 of 24,028 control-plane test targets as affected. Passing SnowPR therefore means passing the selected grading suite, not proving the entire repository remains correct. Methodology details
Opus Results Point More Clearly to Cost Than Ranking
Snowflake defines an agent as a model plus its harness, operating with repository context that includes skills. Cortex Code’s two modes expose tools differently:
- Code mode gives the model separate file, shell and code-search tools.
- REM mode, short for REPL Execution Model, exposes operations as Python functions inside one REPL tool, allowing several operations to be chained in one step.
Each configuration attempted every task three times. Pass@1 is the share of individual trials that passed. Pass@3 is the share of tasks solved at least once across those three attempts.
With opus-5, all four harnesses achieved pass@1 between 84.1% and 85.1%. Snowflake says its 65-task sample cannot resolve that spread. Costs were more separated:
Harness with opus-5 | Reported cost per trial |
|---|---|
| Cortex Code REM mode | $6.34 |
| Claude Code | $6.51 |
| Cursor | $6.93 |
| Cortex Code code mode | $8.19 |
Snowflake reports that REM cost 23% less than code mode and used 14% fewer new tokens, defined as uncached input plus output. The reported cost accounting includes subagents and uses billed or token-priced charges. Reported results and pricing conditions
REM’s 62 solved tasks translate to approximately 95.4% pass@3. That is not a 95.4% first-attempt success rate. The distinction matters when estimating how often a developer can expect a usable patch from one attempt.
These are vendor-reported measurements on private code, so the public article does not enable independent reproduction. REM is also internal-only and unavailable to customers. I would read this as evidence that tool orchestration can improve efficiency, not as a directly actionable recommendation to switch to REM.
Frequently Asked Questions
3 questions
1Is SnowPR a public, reproducible benchmark?
SnowPR uses private Snowflake repositories. The published article explains the methodology and results, but does not provide enough access to reproduce the benchmark independently.
