Predicting what a compiler outputs is quiet hard, especially if you enable optimization flags or get near any undefined behavior. But I think you’re right to say that a compiler is more predictable than any LLM.
I don’t get the point why you should be unable to audit the output of an LLM? If its used as a tool like described the user should have the knowlage to audit any output
Then I can run it again, and the output will be the same.
I can run a LLM prompt.
Then I can run it again, and the output should be the same. The unpredictability is what makes the tech great - Nobody in decision making actually understands this, but the goal is not for identical outputs. Just like me and you, don’t respond to the same question in the same way every time
But I rarely want the output twice? At least in this domain of coding, the entire point is doing it once.
Secondly, I can definitely validate it in most ways that matter. Ironically, by compiling it for example, I make 100% certain the output was well defined code. Type checking, linting, tests, etc. all help ensure I got functionally valid code.
At that point I might not really care if the function name or a variable name is different, just like I don’t care that the compiler did some kind of heuristic optimization or minified the names or something.
In both cases, I have a target functionality, and I need to validate it was reached. The way I got there doesn’t matter (in most cases).
It’s a bit like Docker vs. Nix. Sure, docker isn’t really reproducible perfectly, but it fulfills it’s goal for most people most of the time for a fraction of the time and effort taken.
Predicting what a compiler outputs is quiet hard, especially if you enable optimization flags or get near any undefined behavior. But I think you’re right to say that a compiler is more predictable than any LLM. I don’t get the point why you should be unable to audit the output of an LLM? If its used as a tool like described the user should have the knowlage to audit any output
I can run a compiler.
Then I can run it again, and the output will be the same.
I can run a LLM prompt.
Then I can run it again, and the output should be the same. The unpredictability is what makes the tech great - Nobody in decision making actually understands this, but the goal is not for identical outputs. Just like me and you, don’t respond to the same question in the same way every time
But I rarely want the output twice? At least in this domain of coding, the entire point is doing it once.
Secondly, I can definitely validate it in most ways that matter. Ironically, by compiling it for example, I make 100% certain the output was well defined code. Type checking, linting, tests, etc. all help ensure I got functionally valid code.
At that point I might not really care if the function name or a variable name is different, just like I don’t care that the compiler did some kind of heuristic optimization or minified the names or something.
In both cases, I have a target functionality, and I need to validate it was reached. The way I got there doesn’t matter (in most cases).
It’s a bit like Docker vs. Nix. Sure, docker isn’t really reproducible perfectly, but it fulfills it’s goal for most people most of the time for a fraction of the time and effort taken.
At least clean it up so it’s maintainable, introducing tech debt is a long term impact
For local LLMs you can set the seed and generate the same output every time.
You can get the same output from the LLM if you put the appropriate guardrails and context in place. That’s what is happening in the enterprise space.
I was just pointing out Linus was conflating the two to make his point. I don’t agree with this, but everything else he said was fine.
100%, or we’d commit prompts instead of code lol