What’s stopping people from using open sourced trained models? Like, why are tokens going to be needed in the future? Isn’t this their whole business model?
Honestly, I have a little tin-foil-hat theory that the used their infine supply of venture to fuck the memory market to make sure people couldn’t run giant chinese models on solar and eat their lunch.
you mean like Sam Altman’s ‘dirty DRAM’ deal where he secretly negotiated with Samsung and Hynix to (together) buy 40% of the world’s memory production, announced both deals the same day (neither company knew about the other), and bought the RAM not as sticks or even chips but as finished wafers, useless until they are sliced, binned, and packaged into chip casings, a task OpenAI has neither the equipment nor the expertise to accomplish?
training a top-level model takes harvesting petabytes / exabytes of revelant data and then months of continuous 100% power on a large server farm costing many millions of dollars.
It’s a one-and-done process, then the normal Q&A usage has vastly lower requirements.
Local models generally means you download the result of all that , after it’s done, (the PT in ChatGPT stands for pre-trained) and then generate answers locally, but the training wasn’t done locally. The generating answers step is called “inference”.
Hardware cost to do inference with a top-level model is within reach of most small businesses. Hardware cost to do training of a top-level model is orders of magnitude higher.
So, there are open-source models that are trained by universities, and then used by affluent but not billionaire people at home.
There are open-source models that can do inference (answer questions) on a typical 8GB gaming graphics card, but they aren’t in the same order of magnitude of quality as the famous top-level models.
Thank you! That’s the best answer I’ve seen on any platform!
What would you recommend as a local model? Want to dip my toes in but I’m woefully ignorant. And again, I’ve done my time in IT, so you don’t have to dumb it down, uh, too much.
The last time I ran one locally was a few months ago, so this might be somewhat out of date, but I’ll say Qwen.
Generally there are two numbers to look at first, the number of parameters (number of nodes + much larger number of connections between nodes) and the quantisation (number of bits that define each parameter)
So an 8 billion parameter model at 8-bit quantisation will require 8GB of working RAM, probably VRAM for fast results at home (at 8 bits to the Byte)
and also a 16 billion parameter model at 4 bits quantisation will also fill the same 8GB.
in my experience, high bit depth enables the thing to respond in more complex terms, and more parameters means more “knowledge” is there to be accessed. I have better results from phrasing questions that’ll have short answers and leaning more towards more parameters than bit depth.
The next number to consider is “context window”, which is basically how far back into your chat the thing can “remember” and have previously given information influence the next answer.
A year to two ago, when I was trying a lot of LLMs for coding tips, Google’s seemed the best for larger projects, simply because it had a much bigger context window than any other, and so could “consider” longer and more interconnected scripts as a whole, even if perhaps Claude was “smarter” with a small problem. I don’t know if this is still true in autumn 2026.
They’ve (commercial models) started keeping a per-account or per-chat log of important facts that exists alongside the context window, but doesn’t carry as much detail. Good for seeming to get to know you, not as good for writing new code that won’t conflict with a long-established bit of old code.
It’s not really practical to have million+ token context windows on home/gaming hardware.
(usually a token is a word, or a couple of tokens are a long or rare word, or a token is a punctuation mark)
LM Studio is good software with generally sensible defaults for downloading pre-trained models and running them on your own PC.
They’re the output of all that data centre processing. The model is fixed at that point and can’t “learn” anything more, so the pressure is always on to release a newer model that resulted from more data with better algorithms, but still small enough to be able to run on a consumer grade GPU
For enterprise, I think same reasons why companies using paid software instead of open software. Someone else to maintain and blamed if something doesn’t work.
The hardware requirements are substantial. If companies or individuals want to run models locally, especially at the speed you’d expect from the big name models, then everyone needs a pretty serious GPU on their work station. That or the company needs to build and maintain a server capable of running whatever loads they expect to see.
None of this is impossible of course. But you pay for the tokens one way or another. Either you pay the upfront cost of the hardware and continuing cost of the electricity to run it, or you pay with the inconvenience of running a model very slowly on lower end hardware, or you pay a subscription for someone else to worry about all that.
This is why local models are still niche. Only hobbyists nerds are willing to go through the trouble of setting up and running a local model. Most companies still are not sure if AI is actually worth it for them. So it’s much easier to pay for tokens than the giant upfront cost required to build you own infrastructure.
If the numbers the AI companies are putting up as far as infrastructure cost vs revenue are anything to go by, it probably isn’t worth it to most companies.
They don’t need to, OpenAI and Anthropic are running at a huge loss to keep it cheaper to use them than to buy the comparable hardware required to replace them. The only way running open source models is cheaper is if you ignore hardware costs, or if you have a very specific and unusual usage pattern (i.e. constant 100% capacity, and not a clock cycle more or less), or if you go with another cloud provider who are also running at a loss.
If you want it well rounded I can see the argument for enterprise but for me personally I programmed one on my workstation and it’s trained on very specific things I think it barely uses up all 24gb vram I have available.
come on, I’ve trained toy size models too, you can’t seriously say they’re even vaguely approaching the capability of models that takes a whole massive server farm weeks or months to train.
If you do one simple task like I do in my job yes you can make a small one purpose built. Work in reading comprehension before commenting. If you re read which I know might be hard for you, you can see I said make it for a specific task. Example for you since you’re going to try and counter because you’re an idiot. I have to fill in the same multiple fields on an excel table for state documents. This is pulled from a sql database I literally have a model where it does that for me with natural language processing instead of me being a tool and using python exclusively when I already did that to make the easy simple tool. I suggest Amazon for reading comprehension before commenting books. If you message back I’ll send you some recommendations I tell to my students.
Right, and what proportion of enterprise use cases genuinely need an appreciable fraction of that power (because they are wide ranging and non-repeating)?
Anyone that says these things aren’t impressive is kidding themselves. But how do we get from impressive to productivity enhancing in excess of cost? I find smaller specialised models more plausible on that front, but jury is still out on the financials, we’ll find out I guess!
What’s stopping people from using open sourced trained models? Like, why are tokens going to be needed in the future? Isn’t this their whole business model?
You need 200GB of video memory :)
Honestly, I have a little tin-foil-hat theory that the used their infine supply of venture to fuck the memory market to make sure people couldn’t run giant chinese models on solar and eat their lunch.
Of course, why do you think they are running around with warnings about “rogue AI destroying humanity”?
They hope something of what they throw at the wall sticks and some legislations limit their competition.
By the way, this kind of throwing money at the wall unfortunately works (if you have enough money).
But also - with this kind of applications being useful, memory prices would jump regardless.
OK, I’m just guessing.
you mean like Sam Altman’s ‘dirty DRAM’ deal where he secretly negotiated with Samsung and Hynix to (together) buy 40% of the world’s memory production, announced both deals the same day (neither company knew about the other), and bought the RAM not as sticks or even chips but as finished wafers, useless until they are sliced, binned, and packaged into chip casings, a task OpenAI has neither the equipment nor the expertise to accomplish?
training a top-level model takes harvesting petabytes / exabytes of revelant data and then months of continuous 100% power on a large server farm costing many millions of dollars.
That was my understanding. So how do local models work? (Been fucking with computers since the 80s, AI is all new to me.)
It’s a one-and-done process, then the normal Q&A usage has vastly lower requirements.
Local models generally means you download the result of all that , after it’s done, (the PT in ChatGPT stands for pre-trained) and then generate answers locally, but the training wasn’t done locally. The generating answers step is called “inference”.
Hardware cost to do inference with a top-level model is within reach of most small businesses. Hardware cost to do training of a top-level model is orders of magnitude higher.
So, there are open-source models that are trained by universities, and then used by affluent but not billionaire people at home.
There are open-source models that can do inference (answer questions) on a typical 8GB gaming graphics card, but they aren’t in the same order of magnitude of quality as the famous top-level models.
Thank you! That’s the best answer I’ve seen on any platform!
What would you recommend as a local model? Want to dip my toes in but I’m woefully ignorant. And again, I’ve done my time in IT, so you don’t have to dumb it down, uh, too much.
The last time I ran one locally was a few months ago, so this might be somewhat out of date, but I’ll say Qwen.
Generally there are two numbers to look at first, the number of parameters (number of nodes + much larger number of connections between nodes) and the quantisation (number of bits that define each parameter)
So an 8 billion parameter model at 8-bit quantisation will require 8GB of working RAM, probably VRAM for fast results at home (at 8 bits to the Byte)
and also a 16 billion parameter model at 4 bits quantisation will also fill the same 8GB.
in my experience, high bit depth enables the thing to respond in more complex terms, and more parameters means more “knowledge” is there to be accessed. I have better results from phrasing questions that’ll have short answers and leaning more towards more parameters than bit depth.
The next number to consider is “context window”, which is basically how far back into your chat the thing can “remember” and have previously given information influence the next answer.
A year to two ago, when I was trying a lot of LLMs for coding tips, Google’s seemed the best for larger projects, simply because it had a much bigger context window than any other, and so could “consider” longer and more interconnected scripts as a whole, even if perhaps Claude was “smarter” with a small problem. I don’t know if this is still true in autumn 2026.
They’ve (commercial models) started keeping a per-account or per-chat log of important facts that exists alongside the context window, but doesn’t carry as much detail. Good for seeming to get to know you, not as good for writing new code that won’t conflict with a long-established bit of old code.
It’s not really practical to have million+ token context windows on home/gaming hardware.
(usually a token is a word, or a couple of tokens are a long or rare word, or a token is a punctuation mark)
LM Studio is good software with generally sensible defaults for downloading pre-trained models and running them on your own PC.
They’re the output of all that data centre processing. The model is fixed at that point and can’t “learn” anything more, so the pressure is always on to release a newer model that resulted from more data with better algorithms, but still small enough to be able to run on a consumer grade GPU
For enterprise, I think same reasons why companies using paid software instead of open software. Someone else to maintain and blamed if something doesn’t work.
The hardware requirements are substantial. If companies or individuals want to run models locally, especially at the speed you’d expect from the big name models, then everyone needs a pretty serious GPU on their work station. That or the company needs to build and maintain a server capable of running whatever loads they expect to see.
None of this is impossible of course. But you pay for the tokens one way or another. Either you pay the upfront cost of the hardware and continuing cost of the electricity to run it, or you pay with the inconvenience of running a model very slowly on lower end hardware, or you pay a subscription for someone else to worry about all that.
This is why local models are still niche. Only hobbyists nerds are willing to go through the trouble of setting up and running a local model. Most companies still are not sure if AI is actually worth it for them. So it’s much easier to pay for tokens than the giant upfront cost required to build you own infrastructure.
If the numbers the AI companies are putting up as far as infrastructure cost vs revenue are anything to go by, it probably isn’t worth it to most companies.
They thought they sold tokens, they actually sell marginally (6 months lead) improved models.
They don’t need to, OpenAI and Anthropic are running at a huge loss to keep it cheaper to use them than to buy the comparable hardware required to replace them. The only way running open source models is cheaper is if you ignore hardware costs, or if you have a very specific and unusual usage pattern (i.e. constant 100% capacity, and not a clock cycle more or less), or if you go with another cloud provider who are also running at a loss.
If you want it well rounded I can see the argument for enterprise but for me personally I programmed one on my workstation and it’s trained on very specific things I think it barely uses up all 24gb vram I have available.
come on, I’ve trained toy size models too, you can’t seriously say they’re even vaguely approaching the capability of models that takes a whole massive server farm weeks or months to train.
If you do one simple task like I do in my job yes you can make a small one purpose built. Work in reading comprehension before commenting. If you re read which I know might be hard for you, you can see I said make it for a specific task. Example for you since you’re going to try and counter because you’re an idiot. I have to fill in the same multiple fields on an excel table for state documents. This is pulled from a sql database I literally have a model where it does that for me with natural language processing instead of me being a tool and using python exclusively when I already did that to make the easy simple tool. I suggest Amazon for reading comprehension before commenting books. If you message back I’ll send you some recommendations I tell to my students.
Right, and what proportion of enterprise use cases genuinely need an appreciable fraction of that power (because they are wide ranging and non-repeating)?
Anyone that says these things aren’t impressive is kidding themselves. But how do we get from impressive to productivity enhancing in excess of cost? I find smaller specialised models more plausible on that front, but jury is still out on the financials, we’ll find out I guess!