What Is Tokenization? Why AI Reads in Pieces

Computers cannot calculate probability distributions over infinite raw strings. If you assigned a unique ID to every single word in human history, your model vocabulary would need tens of millions of entries, failing every time a user coined a new slang word, typo, or username.

Conversely, if you fed the model individual letters (character-level tokenization), the sequences would become ten times longer, causing compute costs and attention mechanism complexity (O(N²)) to skyrocket.

Tokenization strikes the perfect engineering balance: it breaks text into subword units. Common words become a single token ("apple"), while rare or complex words are split into known subcomponents ("un" + "pre" + "dict" + "able").

Most state-of-the-art models (including GPT-4, Llama 3, and Mistral) use an algorithm called Byte-Pair Encoding (BPE).

Over time, frequent sequences like 'ing', 'tion', 'the', and 'const' become individual tokens. Modern tokenizer vocabularies typically contain between 32,000 and 128,000 unique token IDs.

However, code and formatting behave differently. Indentation spaces, JSON brackets, and camelCase identifiers split into separate tokens unless specifically optimized by the tokenizer dictionary.

Because transformer inference generates completion tokens sequentially one-by-one, your API response latency is directly proportional to output token count, NOT prompt length. Minimizing verbose output instructions reduces both your AWS/OpenAI bill and your user waiting time.

Because early tokenizers were trained predominantly on English web text, non-Latin scripts (Arabic, Urdu, Hindi, Japanese) were frequently split into 3 to 6 tokens per single word.

This created a "token tax" where non-English users consumed their context window and API budgets 3x faster. Modern tokenizers like GPT-4o and Llama 3 expanded their vocabulary to 128K+ tokens, drastically improving compression for global languages and source code.