What Is a Token in AI? How Many Words a Token Is, and Why There Is No Industry Standard for AI's Billing Unit

The bottom line first: there is no industry-wide definition of a token. A token comes from a vocabulary, a kind of splitting table, that is fixed when each model is trained. Give the same text to different models and they split it into different numbers of tokens; even different generations of models from the same company can differ. So “how many words is a token?” only has rules of thumb, not a fixed measure.

What the industry has actually standardized is the pricing structure, a price per million tokens with input and output billed separately, not the token itself. That difference becomes critical the moment you try to build a cost spreadsheet.

What a token is: the goal is compression, not meaning

A token is the smallest unit a model splits text into for processing. The splitting comes from a family of algorithms called BPE (Byte Pair Encoding), in three steps:

1. Run statistics over a huge body of text and repeatedly merge the character combinations that most often appear together.
2. The merges produce a vocabulary, usually between 50,000 and 200,000 entries.
3. All later input is matched against this vocabulary and split accordingly.

The key point is that the goal of the algorithm is compression, not understanding meaning. Common strings, such as “the” and “and” in English, common Chinese words like 的 (“of”) and 因為 (“because”), or “function” in code, count as a single token each. Rare words get broken up: names, specialized terms, and typos are often split into three to five tokens.

So a token is neither a character nor a word. It is the smallest component that particular model recognizes. That is the premise for every cost question that follows.

Why a common standard is impossible

I found this strange at first, too: it is 2026, so how can such a basic billing unit have no standard?

The answer is that the vocabulary is part of the model’s weights and cannot be changed after training. It is essentially a part of the model, not an external specification, so standardization has no technical foothold. You cannot ask two companies to use the same vocabulary; that would be asking them to use the same model.

That leads to a situation that is even easier to trip over: when a company releases a new model generation, it may change the tokenizer too. Claude changed its tokenizer with the Opus 4.7 generation, so the same text produces a different token count than on the previous generation. If your cost model was built last year, the table may already be off after a generation change, and it will not throw an error. Your estimates will just drift quietly.

Rules of thumb: fine for estimates, not for quotes

This table has only one use: helping you decide whether something is worth doing. If you just need a ballpark figure to decide whether to invest, it is enough. But the moment quotes, budgets, or commitments to clients are involved, do not use it, because its error can reach a factor of two for Chinese text and code.

Why Chinese text loses out, and by an unpredictable amount

In UTF-8 encoding, one Chinese character takes three bytes. If the model’s vocabulary contains the word you are using, two or three characters might merge into a single token. If it does not, the split falls back to something close to byte by byte, and a single character can use two to three tokens.

So the token efficiency of Chinese text depends entirely on how much Chinese the model was trained on. This is a cost factor that rarely gets explained clearly: the same passage of Chinese can cost more than twice as many tokens on one vendor’s model as on another’s, and you cannot see that from the price list.

This is also why I do not put much faith in comparing unit prices. A low unit price does not mean cheap: a model whose Chinese token efficiency is 30% worse is still more expensive even if its unit price is 20% lower.

The pricing structure: the part the whole industry shares

Taking Anthropic’s official API in mid-2026 as an example, Claude Opus 5 costs about $5 for input and $25 for output, Sonnet 5 is $2 and $10, and Haiku 4.5 is $1 and $5 (per million tokens). Prices will change. I list these numbers not for you to quote them, but so you can see the structure: output always costs four to five times as much as input.

That structure directly changes design decisions. A feature that produces long reports and a feature that only classifies and outputs a few words can differ in cost by an order of magnitude, even with the same input. To cut costs, look at the output column first.

The most underestimated point: costs are not linear

This is the point I think most deserves its own section.

The model itself has no memory. Every turn of a conversation has to resend the entire history so the model knows what you have been talking about. So when you type “OK” in the twentieth turn, you are actually billed for the full text of the previous nineteen turns as input again.

Mathematically, the cost of a long conversation grows roughly quadratically, not linearly. In practice, that leads to three conclusions:

• The cost of a long conversation is concentrated toward the end, not spread evenly. A turn that feels cheap at turn three may cost ten times as much at turn thirty.
• The system prompt is a fixed cost billed again every turn, not something you pay once at the start. A detailed persona setup is paid for every time you say anything.
• Caching exists precisely because of this. Fixing the unchanging prefix in place and reading the repeated part back at about one-tenth of the price is currently the most effective tactic for long conversations.

There is only one way to get an exact number

Measure with that model’s own tokenizer.

Anthropic provides a count_tokens endpoint, and you must pass the model ID you will actually use when calling it, because different models give different numbers. On the OpenAI side, the equivalent is the tiktoken package.

There is a common source of error here: using tiktoken to estimate Claude usage underestimates it by roughly 15% to 20%, and the gap is larger for Chinese text and code. The two companies’ splitting tables are different, so they should never stand in for each other. I have seen more than one cost spreadsheet built this way, and the result was an actual bill about 20% higher than the estimate, with no obvious explanation.

Four situations, four ways to calculate

The difference between these four is not precision, but what decision you will make with the number. When deciding whether to do something, being 20% off does not matter. When deciding how much to charge a client, being 20% off comes out of your profit.

Back to the opening question: what is a token? It is the smallest component that a particular model recognizes, and every model recognizes different things. Accept that, and your cost model will rest on the right foundation.

Frequently asked questions

How many words is one token?

There is no fixed answer. In English, one token is roughly 4 characters, or about 0.75 words. In Chinese, one character is roughly 0.6 to 1.5 tokens. The range is wide because each Chinese character takes three bytes in UTF-8, and whether a word is in that particular model’s vocabulary can change the result by more than a factor of two. These numbers are only good for deciding whether to do something, not for quoting a price.

Why is there no industry standard for tokens?

Because the vocabulary is part of the model’s weights and cannot be changed after training. It is essentially a part of the model, not an external specification, so standardization has no technical foothold; asking two companies to use the same vocabulary is asking them to use the same model. The same company may also change its tokenizer between generations, so a cost model built last year may already be inaccurate this year, without any error message.

Can I use tiktoken to estimate Claude usage?

It is not recommended. tiktoken is OpenAI’s tokenizer, and using it for Claude underestimates usage by roughly 15% to 20%, with larger gaps for Chinese text and code. There is only one way to get an exact number: measure with that model’s own tokenizer. For Anthropic, that is the count_tokens endpoint, and you must pass the model ID you will actually use.

Why do long conversations get more and more expensive?

Because the model itself has no memory, and every turn resends the entire history. When you type “OK” in the twentieth turn, you are billed for the full text of the previous nineteen turns as input again, so costs grow roughly quadratically rather than linearly. It follows that the system prompt is a fixed cost billed every turn, not something you pay once at the start; prompt caching exists to reduce exactly that part.

Is a model with a lower unit price always cheaper?

Not necessarily. The token efficiency of Chinese text depends on how much Chinese the model was trained on, and the same Chinese passage can produce more than twice as many tokens on one vendor’s model as on another’s, which you cannot see on the price list. A model whose Chinese token efficiency is 30% worse is still more expensive even if its unit price is 20% lower. To compare prices, measure the same sample with each vendor’s tokenizer.

Keywords: what is a token in AI, AI tokens, tokens to words, tokenizer, BPE, AI cost, LLM pricing, prompt caching, count_tokens