If I ask "why did the LLM output “for” as the XXXXth word of its utterance, can Linus say why?
If you have the model that was used, and the seed used for the generation, then yes you can trace the weights and figure out how everything was generated.
AI is deterministic. And compilers can be just as “nondeterministic” if you don’t know what compiler was used and what flags were enabled.
Because weight x = y isn’t really a satisfactory answer to the “why” in question though. To really answer that you need to refer to the pretraining and context, which in a lot of cases today, and likely forever, since there’s no reason for anyone to cold start model training ever again, is necessarily opaque. A lot of the best why answers are going to be given by the model itself (which… obviously circular). Trust and attribution are just really hard in this paradigm. Not necessarily so different from the challenge of trusting/attributing human output, except that humans are more diverse and in general difficult to manipulate.
If you have the model that was used, and the seed used for the generation, then yes you can trace the weights and figure out how everything was generated.
AI is deterministic. And compilers can be just as “nondeterministic” if you don’t know what compiler was used and what flags were enabled.
AI is not deterministic at all, only if you keep the seed values for generating random numbers the same all over the pipeline.
It’s literally a huge stochastic model.
Because weight x = y isn’t really a satisfactory answer to the “why” in question though. To really answer that you need to refer to the pretraining and context, which in a lot of cases today, and likely forever, since there’s no reason for anyone to cold start model training ever again, is necessarily opaque. A lot of the best why answers are going to be given by the model itself (which… obviously circular). Trust and attribution are just really hard in this paradigm. Not necessarily so different from the challenge of trusting/attributing human output, except that humans are more diverse and in general difficult to manipulate.