cross-posted from: https://feddit.org/post/35091193
About the limits of reviewing and understanding code:
Ka-Ping constructs a model voting machine, and decides to see how hard it would be to verify that we know it behaves correctly. To push that exploration to its furthest, Ka-Ping Yee and David Wagner try an interesting experiment:
David Wagner and I decided to insert three bugs into Pvote to see if the reviewers would find them. We inserted what we thought would be an “easy” bug, a “medium” bug, and a “hard bug” to find, and chose each bug individually in such a way that an insider could conceivably exploit the bug to influence the results of an election. […]
We decided to insert all of these bugs in a 100-line region of a single file, lines 11 to 109 of Navigator.py, and told the reviewers to look in this region. We did this both because the navigator was the most interesting in terms of the program logic and because we knew the reviewers would have limited time. The new version of the code that we gave the reviewers contained all three bugs, but we did not tell the reviewers how many bugs there were.
Yoshi Kohno, Mark Miller, and Dan Sandler participated as reviewers on the third day of the review. Dan was very familiar with Python and found the “easy” and “medium” bugs quickly, within about 70 minutes. Yoshi Kohno and Mark Miller found the “easy” bug after about four hours of reviewing. None of the reviewers found the “hard” bug.
The reviewers spent a total of about 20 reviewer-hours focused on the task of finding the bugs in this 100-line section of Navigator.py.
[…]
If some of the best programmers in the world struggle to find bugs they even know must be there within a 100 line program, there is simply no hope for humans to review the volume of output from LLMs.
And so there is only one thing to do: don’t bother. At each step, remove yourself. You tell yourself you won’t, but you do. You give in to the chute and the shape of the vibe tunnel, and down you go.
I thought this would be an article about the old TV series depicting the life of Basil Fawlty, but no, it’s another comment on LLMs. Oh well.
Don’t mention the war. I mentioned it once, but I think I got away with it.
As far as I know I’m the only person who uses the term “vibe bobsled” and, well, I doubt it’s a term that’s particularly likely to catch on, but I find it personally useful.
Bobsledding, if you are unaware, is a particularly strange and interesting sport. It’s a lot of fun, but you don’t have a lot of agency in it. You sit in a bobsled, you go down an icy track, and really, there is only one way to go.
[…]
The vehicle is the LLM, you are the passenger. And I think the amount of agency people have over their journey is greatly reduced from what they feel like it is. More than just a slippery slope, it is a pre-crafted journey.
At the top of the chute, people tell themselves they’re going to use these tools as a kind of fancy autocomplete. As they descend, they say they’ll spin up some agents to explore ideas, but they’ll write the code themselves. Next their agents are generating the code for them, but don’t worry, but they’ll review all the output. Soon they’re plummeting downward and well, they don’t actually review the code being spat out much anymore, but they trust the agents, heck maybe the agents are actually better coders than they are they say. And where does it go from there? From “I don’t even code anymore” to “I don’t even prompt anymore”?
At every stage of the process, the coder in question removes themselves from the process of producing code, and gives in towards a faith-based initiative of code production, that the LLM knows and does a good job of what it’s doing. But what is the source of gravity pulling the sled along this icy chute?
It’s simple. Generation is not the slow part of coding. Theory-building and review are. And plausible-enough things are extremely hard to debug and understand. But the machines are so fast at producing things. If you are going to review their work, you aren’t really taking advantage of their most powerful property, which is speed. But theory-building and review are also the programmer’s most important role.
