OpenAI MultiLangBench: Testing AI Coding Beyond Popular Languages

OpenAI released MultiLangBench on October 19, an evaluation that asks language models to implement SQLite 3.40.1 in a specified programming language. It measures API cost alongside test pass rate. The release targets a useful gap in coding evaluations: performance outside the languages that dominate familiar coding examples.

The accompanying research post reports improvements across 20 languages and discusses a separate COBOL migration evaluation. These are evaluations of existing models, not a new model launch or an API price change.

What MultiLangBench actually tests

The Apache-2.0 repository provides an unusually concrete task. A model receives SQLite documentation and must produce an implementation that can modify a database in response to SQL queries. It cannot use the internet or an existing SQLite implementation in its evaluation environment.

Each test starts with SQL that creates an initial database, then runs a user query through the candidate's implementation. The resulting database is checked using the real SQLite 3.40.1 outside the model's environment. A test passes only when the validation query produces the expected output exactly.

That tests more than whether a generated function looks plausible. The implementation has to preserve behavior that another program can inspect. Crashes, mismatched outputs and tests exceeding the documented 60-second timeout count as failures.

The requested programming language is also part of the task. An implementation in the wrong language, or a mixture of languages, fails the benchmark's language requirement. This prevents a model from avoiding a less familiar language by completing the work in one it handles more easily.

How many tests are included

The repository describes approximately 24,000 converted tests from SQLite's larger TCL test suite. Its recommended evaluation uses a 272-test subset because running the full collection can be expensive. Evaluators may use the full set for more precision on high-scoring samples, but should say so when reporting results.

That distinction belongs next to any score. A result on the recommended subset and a result on the entire converted collection should not be presented as though they cover identical cases.

The benchmark reports a pair of values: API cost and test pass rate. This makes it possible to compare how much performance a model obtains for the money spent on the evaluation, rather than sorting models by a percentage alone. It still does not include every cost a team would face deploying generated software.

What OpenAI reports about model performance

OpenAI reports that GPT-6 Astra at maximum reasoning effort averages 87.4% across 20 languages, compared with 26.8% for GPT-5.6 Sol. GPT-6 Luna at maximum reasoning effort reaches 40.3%, with an estimated sample cost of $0.77 compared with $3.48 for GPT-5.6 Sol.

Those are OpenAI's evaluation results. The dollar figures describe estimated sample costs in the reported setup; they are not new per-token API prices or a quote for migrating a repository. This article has not independently reproduced the runs.

The inspected repository README explains the evaluation and scoring method, but does not itself contain the post's language-results table. Readers should therefore distinguish the released evaluation artifact from the publisher's reported model comparisons.

A higher aggregate pass rate is useful evidence of progress, but it cannot establish that every language or codebase improved by the same amount. A team considering a less common language should inspect language-specific results and run representative cases from its own environment.

The COBOL results are a separate evaluation

OpenAI's post separately reports GPT-6.1 Sol reaching 86.6% in its COBOL migration evaluation at low reasoning effort and 97.9% at higher effort. These figures should not be merged with the SQLite benchmark scores.

The public MultiLangBench repository is the database-reimplementation artifact. It should not be described as a release of runnable COBOL migration tests based on the material inspected here.

For a production migration, behavioral compatibility has to extend beyond a benchmark's selected cases. Teams still need to test the actual program's data formats, calculations, integration points and failure behavior. A high evaluation score can justify a more serious trial while leaving the acceptance criteria for that trial unchanged.

Why this is useful for engineering teams

The practical contribution is a more explicit test of whether a model can carry a substantial specification into an implementation in a requested language. That is closer to many maintenance and modernization tasks than a short, isolated coding question.

There is also an evaluation-design lesson. An environment that accidentally leaves SQLite available could reward reuse of an existing implementation rather than the intended task. The repository therefore requires checking for embedded libraries and executables, not merely removing one command from the system path.

The next useful evidence will be reproducible results that keep language, test coverage, reasoning budget and cost visible together. That would make it easier to decide where coding agents can take on more work and where a strong headline score still hides expensive gaps.

I'm Paras. I work at the intersection of AI, engineering and distribution.

I help AI and developer-tool teams turn research results, technical decisions and product launches into clear explanations that developers and founders read and share.

Book a 30-minute call.