All the tested languages, outside of COBOL, tested within a narrow margin. I'm sure nobody will be porting their codebase to PHP from these results. This shows that language choice doesn't really matter for LLMs.

Or does it? Why did COBOL lag so much?

I suppose I would like to see tests on a wider assortment of languages now. I'd especially like to see it tested on more esoteric ones.

I notice that assembly language is not on this list. I've been working on a modern syntax for assembler, with C-like types[0], something of a successor to qhasm. I've written a race-the-beam renderer[1], which is a good fit because of its incredibly tight performance constraints. I've been doing a combination of coding by hand and Opus 5.5. Anecdotally, I find Claude can code in it just fine, something I'd like to make more quantitative.

Dan Luu has done that kind of evaluation[2], and found that LLMs do well in a range of languages, but do underperform in assembler. My hypothesis is that expressing programmer intent clearly, and having the compiler verify things like consistent placement of variables in registers, will fix the problems that hold LLMs back.

This has been a hobby project, but it feels ready to break containment. I'm open to a research opportunity where I'd be able to work on it for real - get in touch.

[0]: https://github.com/raphlinus/symbolasm

[1]: https://github.com/raphlinus/vic-2350

[2]: https://danluu.com/pl-tokens/

I’ve already worked with AI in niche languages without issue. AI is also very good at porting, so I expect to see more like the Bun and Ladybird rewrites.

I bet the language with the best metaprogramming will win out.

By "low-resource" they seem to mean "not much training data". I wonder if the fix for this was generic generalization ability, or just making a lot of training data for the specific languages involved. This distinction matters a lot for the question of whether new programming languages can be developed without having LLMs be bad at them.

Niche languages are fascinating, they recall a world that is disappearing, where it was necessary to think before writing, also because resources in terms of CPU and memory were scarce. I worked on IBM 3090 and the PL/1 language and I still remember the difficulty of getting print reports where you also had to calculate the number of columns needed to print a number. Now the arrival of data centers, and the birth of more modern languages like C# and Rust are replacing that world. Today speed is a fundamental element, and AI agents will end up replacing those who write code. But it is not the end, it is just the world changing. You are not a typist, but the one who thinks of new solutions.

So, Lingo is the programming language for Director / Shockwave. Most LLMs know very little about it, since its been MIA for a while, however, there's a niche community of people reverse engineering games and the entire Shockwave runtime for years now, and LLMs are accelerating these efforts.

In fact, because of these niche Shockwave communities, and they've admitted to it, the Habbo Origins client was able to be made. That effort was spearheaded by a player who reverse engineered their backend for years, and eventually built up a decompiler for Shockwave, and made their own game client run as a stand-alone EXE (Projector Mode for Shockwave iirc) which is what Origins did.

More recently, a project named "Shockless" emerged that took the Shockwave client for Habbo Origins, and rebuilt it in JS w/ PixiJS (I think?) in order to be a more special-purpose Shockwave runtime. Some months later (this month I think), Sulake released Keystone a new Origins client, essentially doing the same thing, itself coded in C++ but it looks like it took a very closely similar approach. Really fascinating seeing LLMs push reverse engineering communities forward to help preserve old tech and games, and in some cases, niche games forward from the grave.

I think the frontier LLMs have finally gotten to the point they can sort of pick up the syntax on the fly.

I made my own .NET based language using Opus 4.6 and it knew how to write code for it, since the rules were in the context window. I havent gone back to that project, since I've been trying to build other more ambitious projects, but its impressive how much LLMs can pull off as of early this year.

It is funny to me they mention Visual Basic, I've been on and off experimenting with rebuilding the VB6 IDE and eventually its compiler with a modern runtime using Claude, I've got a good chunk of the IDE, and I can load forms, I can even call VB6.exe to compile if you have it installed, but I had to pause since my old college copy of VB6 is kinda busted (I misconfigured something somewhere and it keeps crashing when I open a project).

I hope someone runs multilangbench for other models. I really want to see how open models in particular do on it. Benchmaxxing by focusing on the top 5 languages seems an easy

I thought LLMs had been good at this for a while? I vibed a language out and the LLM could write in it pretty well. I even got Qwen 3.8 to write it (though it was pretty slow, it managed to get it done)

'modernizing' is an interesting choice of word

It's kind of insane that one of OpenAI's internal test tasks is now "reimplement SQLite from scratch, preserving binary compatibility with existing database files".

Does "Visual Basic" in OpenAI's list also include VBA?

I need to see SQLite implemented in PowerShell

They needed to test what it would cost each model to make the tests pass at 100%.

Haskell got left out :(

> Our second evaluation asks models to migrate COBOL programs to C++, Java, Kotlin, Python, or Rust while preserving what each program does.

Reminds me of how Wall Street Raider [0] just simulates the original code instead of rewriting it.

[0]: https://news.ycombinator.com/item?id=46954696

It is such a ridiculously cool thing that we're reaching an era where the programming language you chose is as temporary a choice as the name of a variable.

> COBOL supports more than 40% of online banking systems, 80% of in-person credit card transactions and 95% of ATM transactions, while Fortran is used in weather forecasting and nuclear reactor safety analysis.

Can we not vibe code foundational bank infrastructure and nuclear reactor safety software? Pretty please?

Mildly offended that Forth didn’t get mentioned once in the article. /s