I’m going to add to the previous responses that a token is also the billing unit used.
If you hire a lawyer, they charge a rate based on their predetermined “billable hours” arrangement. They beak down their jobs into small time segments, often much smaller than an actual hour. Similar to how a mechanic might use a predetermined labor cost based on a time estimate chart; swapping a battery takes X time, but rebuilding an engine takes Y time, with the actual number of hours and minutes not being useful in terms of billing because neither lawyers or mechanics can precisely track exactly how much of the time they’re digging working on your specific job is being spent “hands on” vs the time necessary to do all the little things that come up to interrupt the work. So they bill by the abstracted time a job takes when stripped of real world necessity.
Tokens are, as others said, chunks of data. When you enter a prompt, and get a response, those chunks get tallied up and billed for. From what I’ve read (and I’m not in the field, not even as s hobbyist/amateur; I just wanted to know exactly what you asked) the various companies bill by large chunks, usually per million tokens, or a similar chunks.
As an example, a given word might only need three tokens to process as input, and another three as output. But, because the cost of doing that work is more intensive to output (generate) a response, output costs more per unit. Now, I’m not saying the capitalist hellscape reality of that is a good thing, I’m just saying that’s the reasoning of the capitalist twats that run the companies. Anywhere other than lemmy, I might not need that disclaimer lol.
Now, why they chose token as the jargon, I’ve never looked to see if I could find an answer, but that’s just me wondering just enough to type it, but not enough to go looking ;)
To add on, when using GitHub Copilot you can see a distinction between AI “Credits” and Tokens (in the context window for that conversation). One reason could be that each Model architecturally has a different tokenizer, so one what one Model considers as one Token is different from another. To normalize the overall compute costs across different Models, the AI Credit unit is used as an abstraction.
To expand on this, using tokens as a unit of billing is quite reasonable as it is also the unit of (inference) cost for LLMs.
The inference is essentially nothing but a loop of “Hey if the first X tokens are this, what would token X+1 most likely be?” until the LLM says its done. Every single loop is a ton of math which, on ridiculoudly expensive hardware, uses electricity and produces heat. So its a combination of those costs, and then we haven’t even taken into account the cost to train the model.
To add on, when using GitHub Copilot you can see a distinction between AI “Credits” and Tokens (in the context window for that conversation). One reason is that each Model architecturally has a different tokenizer, so one what one Model considers as one Token is different from another. To normalize the overall compute costs across different Models, the AI Credit unit is used as an abstraction.
If you plugged this question into an LLM here’s how it would be broken into input tokens:
BOS (Beginning of Sequence) | Tf | is | a | token | ? | EOS (End of Sequence)
So this sentence would be 7 input tokens into an LLM. The sentence before this one, which is my response, would be 15 output tokens. The big LLM providers charge you per token used (usually the 1000 tokens used). Some LLM providers charge one price for input tokens and another for output tokens.
Here’s how that translates to actual dollars to you. Here’s an price list from Oracle Cloud, a large company providing LLM service. Lets take the Google Gemini 2.5 Pro LLM model hosted at Oracle Cloud:
Oracle Cloud Infrastructure Generative AI - Google - Gemini 2.5 Pro - Input Tokens - Text, Image, Audio, and Video less than 200K input tokens = $1.25/1,000,000 tokens
Oracle Cloud Infrastructure Generative AI - Google - Gemini 2.5 Pro - Output Tokens - Text Output less than 200K input tokens = $10/1,000,000 tokens
So your input question would have cost you $0.0000175 and my single sentence response to you would have cost you $0.00015.
Some LLMs (like those discussed in the article) separate each letter as a separate token for processing as input. In that case your same question in one of those LLMs would be 13 tokens instead of 7.
It’s the smallest unit an LLM can process. If you write a long florid prompt, it’s more tokens and therefore costs more for the LLM to interpret. There’s literally a “caveman” mod where it makes the LLM respond in caveman-speak, which means loss tokens and therefore cheaper. What’s odd (did not RTFA) is that some languages are cheaper, IIRC even has a super-caveman mode where it responds in Chinese/Japanese because that costs fewer tokens.
In the end LLM are just predicting what is statistically the most likely follow-up for a sentence, according to what they’ve been trained on. That’s how they work.
For this they cut sentences into bits called tokens. The text they are working on is a sequence of tokens, and they predict the next one.
A token could be a whole word (generally if it’s short and usual), often it’s a syllable, and if you’re trying to use uncommon words/names they might even cut them down letter-by-letter in part to try and fit them into something they can swallow.
Of course tokens would be very language dependent, and some languages just use more characters/longer words and sentences than others on average.
Tf is a token?
I’m going to add to the previous responses that a token is also the billing unit used.
If you hire a lawyer, they charge a rate based on their predetermined “billable hours” arrangement. They beak down their jobs into small time segments, often much smaller than an actual hour. Similar to how a mechanic might use a predetermined labor cost based on a time estimate chart; swapping a battery takes X time, but rebuilding an engine takes Y time, with the actual number of hours and minutes not being useful in terms of billing because neither lawyers or mechanics can precisely track exactly how much of the time they’re digging working on your specific job is being spent “hands on” vs the time necessary to do all the little things that come up to interrupt the work. So they bill by the abstracted time a job takes when stripped of real world necessity.
Tokens are, as others said, chunks of data. When you enter a prompt, and get a response, those chunks get tallied up and billed for. From what I’ve read (and I’m not in the field, not even as s hobbyist/amateur; I just wanted to know exactly what you asked) the various companies bill by large chunks, usually per million tokens, or a similar chunks.
As an example, a given word might only need three tokens to process as input, and another three as output. But, because the cost of doing that work is more intensive to output (generate) a response, output costs more per unit. Now, I’m not saying the capitalist hellscape reality of that is a good thing, I’m just saying that’s the reasoning of the capitalist twats that run the companies. Anywhere other than lemmy, I might not need that disclaimer lol.
Now, why they chose token as the jargon, I’ve never looked to see if I could find an answer, but that’s just me wondering just enough to type it, but not enough to go looking ;)
To add on, when using GitHub Copilot you can see a distinction between AI “Credits” and Tokens (in the context window for that conversation). One reason could be that each Model architecturally has a different tokenizer, so one what one Model considers as one Token is different from another. To normalize the overall compute costs across different Models, the AI Credit unit is used as an abstraction.
To expand on this, using tokens as a unit of billing is quite reasonable as it is also the unit of (inference) cost for LLMs.
The inference is essentially nothing but a loop of “Hey if the first X tokens are this, what would token X+1 most likely be?” until the LLM says its done. Every single loop is a ton of math which, on ridiculoudly expensive hardware, uses electricity and produces heat. So its a combination of those costs, and then we haven’t even taken into account the cost to train the model.
To add on, when using GitHub Copilot you can see a distinction between AI “Credits” and Tokens (in the context window for that conversation). One reason is that each Model architecturally has a different tokenizer, so one what one Model considers as one Token is different from another. To normalize the overall compute costs across different Models, the AI Credit unit is used as an abstraction.
If you plugged this question into an LLM here’s how it would be broken into input tokens:
BOS (Beginning of Sequence) | Tf | is | a | token | ? | EOS (End of Sequence)
So this sentence would be 7 input tokens into an LLM. The sentence before this one, which is my response, would be 15 output tokens. The big LLM providers charge you per token used (usually the 1000 tokens used). Some LLM providers charge one price for input tokens and another for output tokens.
Here’s how that translates to actual dollars to you. Here’s an price list from Oracle Cloud, a large company providing LLM service. Lets take the Google Gemini 2.5 Pro LLM model hosted at Oracle Cloud:
Oracle Cloud Infrastructure Generative AI - Google - Gemini 2.5 Pro - Input Tokens - Text, Image, Audio, and Video less than 200K input tokens = $1.25/1,000,000 tokens
Oracle Cloud Infrastructure Generative AI - Google - Gemini 2.5 Pro - Output Tokens - Text Output less than 200K input tokens = $10/1,000,000 tokens
So your input question would have cost you $0.0000175 and my single sentence response to you would have cost you $0.00015.
Some LLMs (like those discussed in the article) separate each letter as a separate token for processing as input. In that case your same question in one of those LLMs would be 13 tokens instead of 7.
It’s the smallest unit an LLM can process. If you write a long florid prompt, it’s more tokens and therefore costs more for the LLM to interpret. There’s literally a “caveman” mod where it makes the LLM respond in caveman-speak, which means loss tokens and therefore cheaper. What’s odd (did not RTFA) is that some languages are cheaper, IIRC even has a super-caveman mode where it responds in Chinese/Japanese because that costs fewer tokens.
In the end LLM are just predicting what is statistically the most likely follow-up for a sentence, according to what they’ve been trained on. That’s how they work.
For this they cut sentences into bits called tokens. The text they are working on is a sequence of tokens, and they predict the next one.
A token could be a whole word (generally if it’s short and usual), often it’s a syllable, and if you’re trying to use uncommon words/names they might even cut them down letter-by-letter in part to try and fit them into something they can swallow.
Of course tokens would be very language dependent, and some languages just use more characters/longer words and sentences than others on average.