From Mexico, I’ve always thought the scariest code in any company is the code nobody wants to touch: the old billing job, the batch process that runs at 2 a.m., the program whose author left years ago. On October 19, OpenAI published a post on coding in low-resource programming languages like COBOL, Fortran and Visual Basic, with two new evaluations and some big jumps for its newest models. The numbers are eye-catching, but the part I find most useful is how they decided to test it.
Why these languages are hard for models
OpenAI’s point is simple: there’s far less COBOL and Fortran out there for models to learn from than Python or JavaScript, yet this code still runs essential services. OpenAI says COBOL supports more than 40% of online banking systems, 80% of in-person credit card transactions and 95% of ATM transactions, and Fortran is used in weather forecasting and nuclear reactor safety analysis. It also names the usual pain: documentation may be incomplete, the original developers may be gone, and important business rules may only exist in the code itself.
Two tests: build something new, keep something old
The first eval, MultiLangBench, asks a model to build a SQLite-compatible database engine from scratch in a given language, across 20 languages. The program gets a database and a SQL query, produces a modified database, and the real SQLite checks the result, using tests adapted from SQLite’s own test suite. At maximum reasoning effort, GPT-6 Astra averaged an 87.4% test pass rate, up from 26.8% for GPT-5.6 Sol. COBOL went from 10.9% to 72.1%, Fortran from 21.4% to 83.5%, and Visual Basic from 27.5% to 89.4%. COBOL is still the hardest language in the set. The cheaper GPT-6 Luna reached 40.3% at an estimated $0.77 per sample, against $3.48 for GPT-5.6 Sol.
The second eval is the one I care about. Models migrate existing COBOL programs to C++, Java, Kotlin, Python or Rust, and hidden tests require byte-for-byte parity on the observable outputs. A migration only counts if every test passes. GPT-6.1 Sol hit 86.6% at low reasoning effort, compared with 5.4% for GPT-5.6 Sol at the same setting, while costing 39% less. At higher effort it reached 97.9%, which OpenAI describes as close to saturating the eval.
The rounding example says it all
OpenAI gives one example that every team doing a migration should tape to the wall. COBOL’s fixed-point arithmetic discards extra decimal digits unless you explicitly ask it to round. A rewrite that rounds instead can run without a single error and still produce different financial results. That’s the real problem with legacy migration: the new code compiles, the old code was never “wrong,” and the difference only shows up in a total that’s off by a cent, a thousand times a day.
My take
I like that the test is about behavior, not about whether the new code looks nice. That’s the right bar. If I were planning a COBOL or Fortran migration, I’d copy the idea before I picked a model: capture real inputs and outputs from the old system first, then make “identical outputs” the only definition of done, whoever (or whatever) writes the new code.
But keep the caveats in view. These are OpenAI’s own evals on OpenAI’s own models, with no independent replication yet, and a test set is not your bank’s codebase with its job schedulers, file formats and decades of edge cases. OpenAI says so itself: applying this “to production software still requires expert review and testing against the system’s business rules and dependencies.” As Engadget and AI Affairs both note, these are results on OpenAI’s test programs under an all-tests-must-pass rule, and rewritten applications still need checking against real workloads. Still, going from 5.4% to 86.6% on a test this strict is a real signal that AI-assisted modernization is moving from demo to something worth a careful pilot.
Featured image: “Magnetic-core memory, at angle” by Bubba73 (Jud McCranie), Wikimedia Commons, CC BY-SA 4.0. Cropped, resized to 1400×900 and lightly sharpened; the modified image is shared under the same license.
