Ruben, writing at A Naive Bidder on October 26, 2026, published a guide to building custom LLM benchmarks around deterministic checks, reusable test infrastructure and accuracy-cost tradeoffs. Its central argument is straightforward: general leaderboards measure useful capabilities, but they cannot tell a team which model handles its particular product requirements most economically.
The guide treats evaluation as maintained software. Test cases need consistent structure, expected answers, metadata and a way to investigate failures. Results should survive between runs, so adding a model or changing one case does not trigger an expensive replay of everything.
These are recommendations drawn from the author’s benchmark-building experience, not a formal evaluation standard. A promised follow-up worked example was not available for this analysis.
The useful contribution is how the pieces fit together. Deterministic scoring makes results easier to interpret; versioned cases make saved results reusable; cost records then make model selection defensible. Here is a reusable implementation recipe drawn from that advice.
Start With Answers You Can Check in Code
The guide’s first recommendation is to validate outputs programmatically wherever possible. Examples include checking JSON structure, confirming that a model found a particular record, and enforcing a maximum number of steps.
For evidence-finding tasks, Ruben recommends structured answers that cite specific records, files, lines or cells. The scorer can compare those references against an expected list, requiring every expected item and no extras.
For example, a team could use this illustrative Python check after parsing and schema-validating the model’s JSON response:
expected = {("sales.csv", "row_17")}
observed = [
(item["file"], item["record_id"])
for item in response["findings"]
]
passed = (
len(observed) == len(expected)
and set(observed) == expected
)
This checks that the response contains exactly the expected evidence reference. The length check also prevents duplicate findings from passing. It has not been executed here, and it would need to sit alongside the application’s schema validation.
The scope matters: matching a citation proves that the model selected the expected reference. It does not automatically prove that every sentence in an accompanying explanation is correct.
Deterministic scoring also differs from deterministic generation. A fixed scorer can return the same verdict for the same output even when repeated model calls produce different outputs. Teams should preserve that distinction when reporting repeatability.
The guide reserves LLM-as-judge evaluation for qualities that genuinely need it, such as whether a summary aligns with its evidence. I’d keep objectively checkable requirements separate from those judgments. Otherwise, a fluent explanation can obscure a concrete failure to find the required record. The guide explains this scoring approach here.
Separate the Harness From the Test Content
Ruben summarizes the architecture neatly: “the harness is code and the tests are content.”
The proposed structure has three layers:
- Harness: The reusable runner, agent, tools, aggregation logic and dashboard.
- Test families: Related tasks with their own instructions, scorers and cases.
- Test cases: Individual inputs, permitted evidence and expected outcomes.
That separation lets a team add cases without editing the central runner. A new task family may need a new scorer, but its task-specific logic should stay outside the harness.
Each case also gets a metadata file, such as meta.json, which the evaluated model cannot see. The guide recommends difficulty labels, tags and instructions for manually validating the checks.
A practical case card should make four questions easy to answer:
| Case-card content | What it helps explain |
|---|---|
| Stable case ID and revision | Which exact task produced this result? |
| Tags and difficulty | Where does the model succeed or struggle? |
| Expected findings and scoring references | What counts as a pass? |
| Step-by-step validation notes | Can a person reconstruct the expected answer? |
Revision tracking is an implementation extension of the guide’s reproducibility advice. If a source file or expected answer changes, an old result should not silently become evidence about the revised case.
Manual validation notes are especially valuable during failure analysis. They give the team a way to distinguish a model error from an incorrect answer key or an ambiguous task. The guide describes both the layered structure and case metadata.
Make Reruns Selective Without Making Comparisons Stale
The guide recommends independently runnable tests and reusable results. Adding a new model, case or scenario should not require rerunning unrelated evaluations.
The engineering consequence is that a saved result needs more identity than a case name and a model label. A practical result record should capture:
- Case and dataset revision.
- Instructions and scorer version.
- Model identifier and relevant generation settings.
- Tool configuration, where applicable.
- Run status, output, cost and timing.
This is an analysis of what safe reuse requires. If the instructions change but the harness reuses an old score, the dashboard can look current while comparing different experiments.
A model-adapter layer helps keep provider integration outside the test content. Each adapter can translate a common request into the provider’s API format, then return a consistent result containing the output, usage data and errors. Provider-specific settings should remain visible in the run record.
Two adapters may expose the same interface while using different tool-calling or structured-output options. Those differences can affect outcomes and belong in the comparison.
Reuse unchanged results, but rerun affected results. That rule preserves the guide’s budget-saving intent without treating every stored score as permanently comparable.
Keep Canary IDs Outside the Model’s Reach
Ruben recommends a unique canary marker in every case, intended both to signal that the material should stay out of training and to help investigate whether a model has encountered off-limits data. The guide explicitly says the evaluated model should not see the marker.
“Hidden” therefore has a concrete implementation requirement: the loader must exclude canaries from prompt-visible content, and the model’s tools must not expose the metadata containing them.
If the model can read a canary through an attached file or search tool, reproducing it tells the team little about training contamination.
A hidden marker is a useful audit signal, but it is not a complete contamination detector. A model’s failure to reproduce one cannot prove that the underlying task was absent from training.
That caution is supported by a July 2026 systematic review published at GEM. Erfan Nourbakhsh and colleagues reviewed 55 studies through late 2025 and found no detection method consistently reliable across contamination types, model-access settings and training stages.
The review distinguishes exact copying from syntactic, semantic and task-level contamination. A unique string addresses only part of that broader problem. The practical lesson is to treat canaries as one audit mechanism, with strict access controls, rather than a certificate that a benchmark is clean.
Build a Dashboard That Explains Failures
The guide recommends presenting results in a static dashboard. That is a useful publishing choice because the report can be shared without requiring every reader to run the harness.
An aggregate score alone leaves too much unexplained. A practical dashboard should expose:
- Pass rates by test family, tag and difficulty.
- Individual failures with expected and observed findings.
- Model and case versions.
- Run cost and latency.
- Missing, failed or stale runs.
The dashboard also needs a clear comparison set. If one model ran 100 easy cases and another ran 100 difficult cases, their headline percentages are not directly comparable.
I’d make the matched-case view the default and show broader coverage separately. This follows directly from selective reruns: once evaluations can grow independently, the reporting layer must prevent incomplete coverage from looking like superior performance. The guide connects independently runnable cases with dashboard reporting.
Frequently Asked Questions
4 questions
1Does Deterministic Scoring Make an LLM Deterministic?
No. It makes the verdict repeatable for a given output. The model may still generate different answers across calls.
