OpenAI Releases MultiLangBench for Legacy Coding Tests
The open SQLite benchmark tests models across 20 languages, while a separate COBOL migration evaluation remains outside the public release.
Listen
AI narration
12:02
0:00 / 12:02
AI SummaryGenerated from this article
OpenAI released MultiLangBench, an open-source evaluation testing AI models' ability to implement SQLite across 20 programming languages including COBOL, Fortran, and Visual Basic. Models must build a working database engine producing files that real SQLite can validate, helping enterprise teams assess coding capabilities in legacy languages. OpenAI reports GPT-6 Astra achieved 87.4% average pass rate across all languages, up from 26.8% for GPT-5.6 Sol, with COBOL performance rising from 10.9% to 72.1%. A separate unpublished COBOL migration evaluation shows GPT-6.1 Sol scoring 97.9% at higher reasoning effort. These company-reported measurements require independent verification and organization-specific business-rule testing before production use.
OpenAI’s new coding evaluation asks models to build a SQLite-compatible database engine in a specified programming language, then produce database files that the real SQLite can validate.
The company has published MultiLangBench, an Apache-2.0-licensed evaluation covering 20 languages, including COBOL, Fortran, and Visual Basic. Enterprise teams can use it to investigate coding capabilities in languages that may matter more to their maintenance backlog than performance on familiar Python or JavaScript tasks.
OpenAI reports substantial gains for GPT-6 Astra, alongside lower-cost results for GPT-6 Luna. It also describes a separate COBOL migration evaluation involving GPT-6.1 Sol. These are company-reported measurements. The public repository provides MultiLangBench’s setup and tests, but does not include the migration evaluation.
For teams maintaining legacy systems, the useful next step is to reproduce the relevant language results, then test the business rules and migration risks that a database benchmark cannot cover.
MultiLangBench Tests Database Behavior, Not Just Valid Code
MultiLangBench requires a model to implement SQLite version 3.40.1 using supplied documentation. Its output must be an executable that accepts a database path as a command-line argument and an SQL query through standard input.
The test suite comes from SQLite’s TCL tests. According to the repository, approximately 24,000 of the original 51,000 tests could be converted into a three-part format:
Initial database: SQL that creates the starting database.
User query: SQL that the model’s implementation must execute against that database.
Validation query: SQL that the real SQLite 3.40.1 runs against the resulting database file.
A test passes when the validation query produces the exact expected output string. Compiling successfully or returning a plausible-looking query result is insufficient: the candidate must modify a database file in a way the reference SQLite implementation can subsequently interpret.
The evaluation also restricts shortcuts. The model’s environment must have no internet access, and existing SQLite executables, libraries, and embedded implementations must be removed. The reference binary belongs in the grading process, where the candidate cannot call it instead of implementing the requested functionality.
Language compliance is part of the score. Under the repository’s rules, a solution written in the wrong language, or mixing programming languages, should fail. This prevents a model asked to work in COBOL from solving the problem by delegating to another runtime.
Work with Zeniteq
Let’s work together
We’re open to thoughtful collaborations with teams building in AI. Explore the ways we can work together.
These constraints make MultiLangBench a useful controlled comparison, though not a comprehensive test of maintaining an existing enterprise application.
OpenAI Reports Large Gains, With Costs Measured Separately
OpenAI reports that GPT-6 Astra achieved an average test pass rate of 87.4% across 20 languages, compared with 26.8% for GPT-5.6 Sol.
The company’s reported language-level results include:
Language
GPT-5.6 Sol
GPT-6 Astra
COBOL
10.9%
72.1%
Fortran
21.4%
83.5%
Visual Basic
27.5%
89.4%
The cited materials do not establish independently reproduced results. OpenAI’s measurements support a claim of improvement on this evaluation; they do not establish that the models can safely modify production systems in those languages.
For an enterprise buyer, the language breakdown is more useful than the overall average. An organization maintaining COBOL applications should examine COBOL performance and failure cases directly. Strong results elsewhere cannot compensate for errors in the language it actually needs.
OpenAI also reports a 40.3% average pass rate for GPT-6 Luna, at an estimated $0.77 per sample, compared with $3.48 for GPT-5.6 Sol. MultiLangBench explicitly defines its evaluation score as a pair: API cost and test pass rate. Teams can therefore compare capability at a given cost level instead of treating the highest pass rate as the only relevant result.
The reported sample costs are neither fixed migration prices nor API token rates. A production budget needs the applicable API pricing documentation, actual usage measurements, and surrounding execution costs. Compiler runs, retries, investigation, and human review also belong in that assessment.
Reproducing the Benchmark Requires a Controlled Harness
The repository provides enough methodological detail to build a comparable evaluation, but teams still need to assemble and validate their execution environment.
A practical reproduction should follow these steps:
Choose the languages the organization maintains. Start with the relevant subset rather than spending the entire budget reproducing a 20-language average. Record the compiler, runtime, and environment used for each language.
Prepare an isolated candidate environment. Make the SQLite 3.40.1 documentation available locally, install the necessary language tools, and disable internet access. Remove existing SQLite implementations. The repository includes scripts/remove_sqlite.sh as an example, but explicitly says teams must inspect their own environments for remaining copies.
Keep reference grading separate. Create the initial database, pass it into the candidate environment, run the generated executable with the user query, and retrieve the modified database. Run the validation query using the real SQLite 3.40.1 outside the candidate’s environment.
Start with the prescribed test subset. The repository recommends 272 tests whose hashed names begin with 00, 01, or 02. Running all approximately 24,000 tests is acceptable for greater precision on high-scoring samples, but that expanded test coverage should be disclosed.
Apply the failure rules consistently. A mismatched output, crash, or 60-second timeout counts as a failed test. Check that the implementation uses the requested language rather than accepting a high score from a noncompliant solution.
Validate the runner before comparing models. The repository identifies the SQL query strings and SQLite binary as the source of truth. Its supplied expected_output.txt files let teams check whether their own test runner produces the expected answers.
Teams should also record model identifiers, prompts, reasoning settings, tool access, spending limits, and repeat runs. Changing these experimental controls can change the outcome even when the test files stay constant.
Report the per-language pass rate alongside cost and failure categories. A single aggregate percentage hides whether an implementation failed broadly, hit execution limits, or missed a smaller set of database behaviors.
The COBOL Migration Scores Come From a Different Evaluation
OpenAI separately reports that GPT-6.1 Sol scored 86.6% at low reasoning effort and 97.9% at higher reasoning effort on a COBOL migration evaluation described as testing behavior preservation.
Implementing SQLite in COBOL and migrating an existing program pose different problems. A database reimplementation starts with a specified target and a reference implementation. Migration requires changing an existing program while retaining the behavior that matters. In an enterprise project, the difficult part may be identifying that behavior, including assumptions that were never written down.
The cited materials do not provide a public repository for the separate COBOL migration evaluation. Cloning MultiLangBench therefore does not let a team reproduce the GPT-6.1 Sol migration percentages. Its released tests cannot establish the migration tasks’ coverage, reveal their failure cases, or verify those scores.
The higher reported score should not be read as a recommendation to increase reasoning effort and remove review. It describes a result under OpenAI’s evaluation conditions. Whether additional reasoning improves a particular organization’s migration work, and at what cost, needs local measurement.
Add Business-Rule Tests Before Production Code Changes
The strongest enterprise use of MultiLangBench is as an initial capability check, followed by an evaluation built around the organization’s own systems.
For maintenance work, test representative changes in the actual language dialect, compiler, libraries, and deployment environment. For migration, run the existing and proposed implementations against the same inputs and compare the outputs and side effects that matter.
Useful organization-specific cases might include fixed-point rounding, date boundaries, file layouts, error handling, transaction behavior, and batch-processing order. These are examples of what a team could test, not capabilities that MultiLangBench has demonstrated.
Business-rule validation needs its own attention. Matching an existing program on a narrow set of ordinary inputs does not establish that a rewritten calculation handles exceptional cases correctly. Tests should include known incidents, boundary conditions, and rules confirmed by people responsible for the process.
Keep some acceptance cases outside the model’s development loop. Otherwise, repeated fixes can improve performance on visible tests without demonstrating that the implementation generalizes to unseen requirements.
Teams should distinguish three questions: Does the code compile? Does it pass the selected tests? Is the evidence sufficient to approve this particular production change? MultiLangBench addresses a controlled version of the second question. It cannot answer the third on an organization’s behalf.
OpenAI’s release makes its coding claim more inspectable through a published test suite and explicit grading rules. The separate migration results deserve a narrower reading until their evaluation is available. Legacy-code teams can use the benchmark to investigate failures in the languages they maintain, while requiring business-rule and migration tests before generated changes reach production.
