What is a token?

AI models do not read words. They read tokens.

Tokens

The Gemini API documentation says that Gemini and other generative AI models process input and output at a granularity called a token. Tokens can be single characters, like z, or whole words, like cat, and long words are broken into several tokens. The set of all tokens a model uses is its vocabulary, and splitting text into tokens is tokenization. Anthropic's glossary describes tokens in the same way: the smallest units a language model works with, which can be words, subwords, characters or even bytes.

Rules of thumb for English

For Gemini models, the documentation says a token is about 4 characters, and 100 tokens is about 60 to 80 English words. The estimator on this site uses both rules and shows the range: characters divided by 4, and 1.25 to 1.67 tokens per word.

Why other languages differ

Anthropic's glossary says that for Claude the number of characters a token represents depends on the model and on the language. So the English rules do not carry over to Hindi, Punjabi or Chinese: such text usually takes more tokens for the same meaning, which is why the estimator warns when most of the text is outside the Latin alphabet. We give no fixed figure for other scripts because we found none in the official documents.

Why it matters

1. Price. Most APIs charge per million input tokens and per million output tokens.
2. Limits. Every model has a context window: the most tokens it can handle in one request, input and output together.
3. Speed. More tokens take longer to process and to produce.

This is an independent guide. It is not connected with Google, Anthropic, OpenAI or any company named here. Facts were read on 14 November 2026 from the pages named on each page. The numbers are estimates from rules of thumb, not the counts of any maker's tokenizer.