Parsimony takes patches already published for software-engineering benchmarks and asks a question that pass rates don’t answer: how much executable code did the successful fixes add?
The project ranks models by their mean net syntax units added on solved tasks. It counts elements such as statements, function calls, comparisons, names and literals, then subtracts deletions. Coding-agent evaluators can examine that measure alongside correctness and cost.
Passing a benchmark’s tests does not establish that a patch is economical, readable or easy to maintain. Parsimony offers a concrete, reproducible proxy for code growth, but it does not directly measure those broader qualities.
Several limits shape how to read the results. The project reuses published patches and outcomes without rerunning evaluations. The Python analysis excludes tests, documentation and generated files. Results also reflect the agent operating the model, and the project’s own data cautions against treating the smallest patch as the best one.
Parsimony Counts Syntax, Not Formatting
According to the project’s counting methodology, Parsimony retrieves each affected implementation file at the benchmark task’s base commit, applies the patch and parses the before-and-after versions. Nothing in the patch is executed.
For Python, the analyzer uses the language’s abstract syntax tree, a structured representation of the code. Its units include:
- Statements, including assignments, returns, conditionals and function definitions.
- Expressions, including calls, attribute access, arithmetic and Boolean operations.
- Individual comparisons, names and literals.
- An
EndBlockunit marking the end of each statement block, so nesting changes also count.
Comments, formatting, punctuation, brackets and identifier length do not count. Docstrings are removed before analysis.
This makes the measure less vulnerable than line counts to superficial compression. Putting several operations on one line does not make their syntax disappear, and shortening a descriptive variable name does not reduce its unit count.
The repository’s worked example replaces an explicit accumulation loop with sum(items). Parsimony counts 13 units removed and three added, producing a net change of minus 10.
The calculation is straightforward:
Net units = added units − deleted units
A negative result means the measured implementation became smaller. It does not mean that the agent produced no code.
The analyzer aligns the before-and-after unit sequences using a minimum-edit diff. For rewrites requiring more than 500 edits, the methodology specifies a line-anchored approximation and flags the affected files. Anyone reproducing the results needs to account for this: a large rewrite can receive a less exact alignment than a small patch.
Net Growth and Editing Volume Are Different Questions
Subtracting deletions helps identify fixes that replace existing machinery instead of continually adding more. Net growth, however, can conceal extensive rewriting.
As an illustrative example, a patch that adds 100 units and removes 90 has the same net growth as one that adds 10 units and removes none. Their editing footprints are very different.
Parsimony also defines churn, calculated as added units plus deleted units. Net units describe how much the implementation grows or shrinks; churn describes how much syntax changes. A large replacement can look restrained on the first measure while still requiring substantial review.
The website ranks models by mean net units on the tasks they solved. It separately displays a mean covering all measured attempts, including failures, along with the benchmark’s solve rate and the number of solved attempts that could be measured. These columns serve different purposes; not every displayed average is the ranking score.
Comparisons also depend on which tasks each model solves. A model handling harder issues could legitimately need larger fixes than one solving a smaller, easier subset. An average over each model’s successes does not hold the task population constant.
The methodology includes a shared-task comparison using tasks that every selected agent both solved and could be analyzed on. That helps control task mix, although the documentation warns that it can favor easier tasks. Neither view removes every source of selection bias.
Correctness Is Inherited From the Original Benchmark
Parsimony analyzes existing evaluation artifacts. Its repository describes importing patches and outcomes from sources including DeepSWE, SWE-bench Verified and SWE-bench Live.
For a task to receive credit as solved, Parsimony trusts the original benchmark’s published result. It does not rerun the tests to confirm that outcome.
Researchers can therefore inspect another property of existing patches without paying to generate and execute every attempt again. A Parsimony score cannot establish that a fix remains correct under a different environment, additional tests or a broader interpretation of the issue.
Some attempts cannot be analyzed because of missing files, parsing problems or other errors. The project says these attempts are excluded rather than scored as zero, and its coverage columns show how many results were measurable.
If the missing patches systematically differ from those successfully analyzed, an average over the remaining patches may give an incomplete picture. A low mean deserves less confidence when substantial portions of the relevant results are unavailable.
For reproducibility, the project records artifact hashes, base commits and analyzer details. Its documentation also advises pinning the Python version for comparisons. Recomputable numbers still depend on using the same source material and measurement configuration.
Python Scope and Agent Configuration Need Careful Reading
The landing page describes Python-only measurement. Within that analysis, Parsimony excludes tests, documentation and generated files, along with other categories specified in its implementation-file rules.
A small score therefore does not mean a small overall contribution. An agent could produce extensive tests and explanatory documentation without those additions increasing the measured implementation size. The metric also cannot tell readers whether a concise fix came with adequate regression coverage.
The repository lists separate JavaScript, TypeScript, Go and PHP boards. “Python-only” accurately describes the landing-page analysis discussed here, not every language track in the repository. The project explicitly keeps benchmark and language boards separate because their unit counts are not interchangeable.
Agent configuration needs similar care. The landing page says the compared runs use mini-SWE-agent, which helps reduce variation caused by the surrounding software. The repository gives a more qualified account: DeepSWE and Verified generally use mini-SWE-agent, with a named exception, while Live configurations use their own agents.
Sources
- Parsimonyparsimony.lasu.dev
- counting methodologygithub.com
- repository’s worked examplegithub.com
- Hacker News discussionnews.ycombinator.com
