So a little bit of breakage is like a healthy "exercise."
Presumably, a functional system is supposed to have some kind of drills to fill in for that. But I have seen ones that don't!
The OG paper on this is called "The Ironies of Automation" (Bainbridge, 1983). It's a clunky but fascinating read. I give it a skim every couple years and usually come away with a bit of a fresh take on the problem.
https://web.archive.org/web/20200717054958if_/https://www.is...
Can also confirm that the process can be harrowing and extremely inefficient (days can turn into months) if you don't know what you're doing and don't have the necessary support.
https://netflix.github.io/chaosmonkey/ and similar can help.
That said ... most instability is introduced with normal changes, so every engineer can be a chaos monkey. ;-)
This was extremely tedious, and annoying, of course, but things would break in all sorts of educational ways.
And once in a while, the computer would do something truly magical, something I didn't even know computers can do.
LLMs still make mistakes, of course, but they make different kinds of errors than I do.
I've been trying to replicate the effect with software that reproduces my own type of errors, randomly omitting tokens from the program. (A strange type of fuzzing the program space.)
Some don't compile, some crash at runtime, most don't do anything interesting, but I know for a fact the process is worthwhile.
What would be really interesting would be to create a cheap verification process for this at scale, i.e. a "wall of TVs" each containing the output of a program, so that the "worthwhileness" of each one can be evaluated at a glance.
https://web.cs.ucdavis.edu/~rogaway/classes/188/materials/th...
I don't mean things like plumbing and roads, I mean things like, "what happens to a culture when people stop saying hello to each other?"
Unfortunately site would need some refurbishment for mobile… slide show doesn’t show, and page is floating.