Skip to content
CodexPal
AI & LLMs · Beginner · 3 min read

Tokens explained: why the same text costs differently on each model

Tokenizers split text differently, so one prompt can be a different number of tokens on each model. Rules of thumb, real counting methods and why language matters.

By CodexPal Editorial3 min read
One sentence split into different-sized token blocks by two tokenizers.

AI models do not read words; they read tokens. Each model family has its own tokenizer, which decides how text is split. That is why a token count, and therefore a cost, measured on one model does not transfer exactly to another.

What a token is

A tokenizer breaks text into pieces from a fixed vocabulary. Common words may be a single token. Rare words, names, code and numbers are often split into several. Spaces and punctuation are usually part of tokens too.

Google’s documentation gives a working rule: a token is about four characters, and 100 tokens are about 60–80 English words. It is a useful first estimate, not a billing figure.

Estimate tokens for your text →

Different tokenizers, different counts

Anthropic notes that Claude Opus 4.7 and later models use a newer tokenizer that produces approximately 30% more tokens for the same text than earlier Claude models, with the exact increase depending on content. It advises recounting prompts with the model you plan to use rather than reusing older counts.

The same applies across providers. When you compare prices, compare cost per task measured on each model, not just the price per million tokens.

Language and format matter

Tokenizer vocabularies are built from training data, and English is usually well represented. Text in other languages, especially those written in non-Latin scripts, can use more tokens for the same meaning. Code, tables, long numbers and unusual formatting can also be token-heavy.

If your users write in several languages, measure representative samples in each one. The difference can change both cost and how much fits in the context window.

Count tokens properly

For accurate numbers, use the provider’s own method. Anthropic offers a free token-counting endpoint that accepts the same input as a message. Gemini provides a count_tokens method, and every major API reports input and output tokens in each response.

python
import anthropic

client = anthropic.Anthropic()
count = client.messages.count_tokens(
    model="claude-sonnet-5-5",
    system="You are a concise support assistant.",
    messages=[{"role": "user", "content": "Where is my order?"}],
)
print(count.input_tokens)

The takeaway

Use the four-characters rule for quick planning, then count real prompts with the provider’s tool for the exact model and languages you will use.

Go to the source

Sources & further reading

Primary references checked on November 12, 2026. Examples and diagrams are illustrative unless stated otherwise.

  1. Google — understand and count tokens
  2. Anthropic — token counting
  3. Anthropic — Claude API pricing
Found this useful? Pass it along.