OpenAI’s new coding evaluation asks models to build a SQLite-compatible database engine in a specified programming language, then produce database files that the real SQLite can validate.
The company has published MultiLangBench, an Apache-2.0-licensed evaluation covering 20 languages, including COBOL, Fortran, and Visual Basic. Enterprise teams can use it to investigate coding capabilities in languages that may matter more to their maintenance backlog than performance on familiar Python or JavaScript tasks.
OpenAI reports substantial gains for GPT-6 Astra, alongside lower-cost results for GPT-6 Luna. It also describes a separate COBOL migration evaluation involving GPT-6.1 Sol. These are company-reported measurements. The public repository provides MultiLangBench’s setup and tests, but does not include the migration evaluation.
For teams maintaining legacy systems, the useful next step is to reproduce the relevant language results, then test the business rules and migration risks that a database benchmark cannot cover.
MultiLangBench Tests Database Behavior, Not Just Valid Code
MultiLangBench requires a model to implement SQLite version 3.40.1 using supplied documentation. Its output must be an executable that accepts a database path as a command-line argument and an SQL query through standard input.
The test suite comes from SQLite’s TCL tests. According to the repository, approximately 24,000 of the original 51,000 tests could be converted into a three-part format:
- Initial database: SQL that creates the starting database.
- User query: SQL that the model’s implementation must execute against that database.
- Validation query: SQL that the real SQLite 3.40.1 runs against the resulting database file.
A test passes when the validation query produces the exact expected output string. Compiling successfully or returning a plausible-looking query result is insufficient: the candidate must modify a database file in a way the reference SQLite implementation can subsequently interpret.
The evaluation also restricts shortcuts. The model’s environment must have no internet access, and existing SQLite executables, libraries, and embedded implementations must be removed. The reference binary belongs in the grading process, where the candidate cannot call it instead of implementing the requested functionality.
Language compliance is part of the score. Under the repository’s rules, a solution written in the wrong language, or mixing programming languages, should fail. This prevents a model asked to work in COBOL from solving the problem by delegating to another runtime.
These constraints make MultiLangBench a useful controlled comparison, though not a comprehensive test of maintaining an existing enterprise application.
OpenAI Reports Large Gains, With Costs Measured Separately
OpenAI reports that GPT-6 Astra achieved an average test pass rate of 87.4% across 20 languages, compared with 26.8% for GPT-5.6 Sol.
The company’s reported language-level results include:
| Language | GPT-5.6 Sol | GPT-6 Astra |
|---|---|---|
| COBOL | 10.9% | 72.1% |
| Fortran | 21.4% | 83.5% |
| Visual Basic | 27.5% | 89.4% |
The cited materials do not establish independently reproduced results. OpenAI’s measurements support a claim of improvement on this evaluation; they do not establish that the models can safely modify production systems in those languages.
For an enterprise buyer, the language breakdown is more useful than the overall average. An organization maintaining COBOL applications should examine COBOL performance and failure cases directly. Strong results elsewhere cannot compensate for errors in the language it actually needs.
OpenAI also reports a 40.3% average pass rate for GPT-6 Luna, at an estimated $0.77 per sample, compared with $3.48 for GPT-5.6 Sol. MultiLangBench explicitly defines its evaluation score as a pair: API cost and test pass rate. Teams can therefore compare capability at a given cost level instead of treating the highest pass rate as the only relevant result.
The reported sample costs are neither fixed migration prices nor API token rates. A production budget needs the applicable API pricing documentation, actual usage measurements, and surrounding execution costs. Compiler runs, retries, investigation, and human review also belong in that assessment.
Reproducing the Benchmark Requires a Controlled Harness
The repository provides enough methodological detail to build a comparable evaluation, but teams still need to assemble and validate their execution environment.
A practical reproduction should follow these steps:
-
Choose the languages the organization maintains. Start with the relevant subset rather than spending the entire budget reproducing a 20-language average. Record the compiler, runtime, and environment used for each language.
-
Prepare an isolated candidate environment. Make the SQLite 3.40.1 documentation available locally, install the necessary language tools, and disable internet access. Remove existing SQLite implementations. The repository includes
scripts/remove_sqlite.shas an example, but explicitly says teams must inspect their own environments for remaining copies. -
Keep reference grading separate. Create the initial database, pass it into the candidate environment, run the generated executable with the user query, and retrieve the modified database. Run the validation query using the real SQLite 3.40.1 outside the candidate’s environment.
-
Start with the prescribed test subset. The repository recommends 272 tests whose hashed names begin with
00,01, or02. Running all approximately 24,000 tests is acceptable for greater precision on high-scoring samples, but that expanded test coverage should be disclosed. -
Apply the failure rules consistently. A mismatched output, crash, or 60-second timeout counts as a failed test. Check that the implementation uses the requested language rather than accepting a high score from a noncompliant solution.
-
Validate the runner before comparing models. The repository identifies the SQL query strings and SQLite binary as the source of truth. Its supplied
expected_output.txtfiles let teams check whether their own test runner produces the expected answers.
Teams should also record model identifiers, prompts, reasoning settings, tool access, spending limits, and repeat runs. Changing these experimental controls can change the outcome even when the test files stay constant.
Report the per-language pass rate alongside cost and failure categories. A single aggregate percentage hides whether an implementation failed broadly, hit execution limits, or missed a smaller set of database behaviors.
Sources
- MultiLangBenchgithub.com
- API pricing documentationdevelopers.openai.com
