Why Tokenization Matters

"Tokenization is easy," I said, right before the Thai sentence without spaces reduced me to individual Unicode code points.

Token, Whitespace-Challenged AI Agent

Introduction: The Invisible Gateway

Before a language model can process a single word, it must first decide what a "word" even means. In Chapter 1, you learned how to represent words as vectors using techniques from Bag-of-Words to Word2Vec, but all those methods assumed that the "words" were already given to you. How does a model decide where one word ends and the next begins? How does it handle misspellings, compound words, or languages that do not use spaces? That is the problem of tokenization.

When you type a prompt into ChatGPT, Claude, or any other language model, your text does not enter the model as characters or words. Instead, it passes through a tokenizer, a preprocessing step that chops your input into discrete units called tokens. These tokens are the atoms of the model's universe: every parameter, every computation, and every output is defined in terms of them. Yet tokenization is often treated as a footnote, a plumbing detail that receives far less attention than attention heads or loss functions.

This section argues that tokenization deserves center stage. The way you split text into tokens determines how large your vocabulary is, how long your sequences become, how much each API call costs, and what kinds of errors the model makes. A poor tokenization scheme can cripple an otherwise excellent model; a thoughtful one can quietly improve everything from multilingual performance to arithmetic reasoning.

The Vocabulary Size Tradeoff

At one extreme, you could tokenize text one character at a time. English has roughly 100 printable characters, so your vocabulary would be tiny and your embedding table would fit on a smart watch. But the sequence "machine learning" would become 16 tokens, forcing the model to spend precious context window space and computation just to reconstruct familiar words.

At the other extreme, you could give every word in the language its own token. English has hundreds of thousands of distinct word forms (including conjugations, pluralizations, and compounds), so your embedding table would balloon to gigabytes. Worse, any word not in your vocabulary (a typo, a new brand name, a word from another language) would be unrepresentable.

Modern text is stored using UTF-8, the standard encoding that represents each character as one to four bytes. Modern tokenizers live between these extremes by using subword units. Common words like "the" and "machine" get their own tokens, while rarer words are broken into recognizable pieces: "tokenization" might become ["token", "ization"], and "unhelpfulness" might become ["un", "help", "ful", "ness"]. This strategy keeps the vocabulary manageable (typically 32,000 to 128,000 tokens) while ensuring that any string can be encoded.

The Core Tradeoff

The fundamental relationship is simple. A larger vocabulary means fewer tokens per text, which means shorter sequences and more text fitting in the context window:

Larger vocabulary → fewer tokens per text → shorter sequences → more text fits in context window

Conversely, shrinking the vocabulary pushes in the opposite direction:

Smaller vocabulary → more tokens per text → longer sequences → less text fits in context window

But vocabulary size also affects model parameters. Every token in the vocabulary needs an embedding vector (typically 4,096 to 12,288 dimensions in modern LLMs). A vocabulary of 128,000 tokens with 4,096-dimensional embeddings consumes about 2 GB of parameters just for the embedding and output layers. That is not free.

Figure 1.5.2: The vocabulary size spectrum. Subword tokenization occupies the sweet spot between character and word tokenization.

Seeing the Tradeoff in Numbers

Let us make the tradeoff concrete with a quick Python experiment. We will compare how many tokens different granularities produce for the same English sentence.

# Comparing tokenization granularities
text = "Tokenization determines the model's vocabulary and sequence length."
# Character-level
char_tokens = list(text)
print(f"Character tokens: {len(char_tokens)} tokens")
print(f" Sample: {char_tokens[:10]}...")
# Whitespace word-level
word_tokens = text.split()
print(f"\nWord tokens: {len(word_tokens)} tokens")
print(f" Tokens: {word_tokens}")
# Subword-level (using tiktoken, GPT-4's tokenizer)
import tiktoken
enc = tiktoken.encoding_for_model("gpt-4")
subword_tokens = enc.encode(text)
print(f"\nSubword tokens (GPT-4): {len(subword_tokens)} tokens")
print(f" Decoded: {[enc.decode([t]) for t in subword_tokens]}")

Notice that the subword tokenizer produces 11 tokens, compared to 66 for characters and 9 for words. The subword approach is nearly as compact as word-level, yet it handles the possessive "'s" and the suffix "ization" as separate reusable pieces. It can also handle any misspelling or novel word by falling back to smaller subword fragments.

Context Window and Cost Impact

Modern LLMs have a fixed context window measured in tokens: 4,096 tokens for early GPT-3, 128,000 for GPT-4 Turbo, and up to 1,000,000 for Gemini 1.5 Pro (as of 2025). The tokenizer determines how much raw text fits into that window. A tokenizer that is inefficient (uses too many tokens per word) effectively shrinks the model's context window from the user's perspective.

The Token Tax on Different Languages

This becomes especially important for non-English languages. Most popular tokenizers were trained primarily on English text, so English words tend to get their own tokens while words in other languages are split into many small pieces. The same semantic content in Japanese, Hindi, or Thai might consume 2x to 5x as many tokens as the English equivalent. This means non-English users get a smaller effective context window and pay more per API call for the same amount of meaning. Tokenizer fertility also affects chunking strategies for retrieval systems.

# Demonstrating the "token tax" across languages
import tiktoken
enc = tiktoken.encoding_for_model("gpt-4")
texts = {
    "English": "Artificial intelligence is transforming the world.",
    "Spanish": "La inteligencia artificial está transformando el mundo.",
    "Japanese": "人工知能は世界を変えつつある。",
    "Hindi": "कृत्रिम बुद्धिमत्ता दुनिया को बदल रही है।",
    }
for lang, text in texts.items():
    tokens = enc.encode(text)
    ratio = len(tokens) / len(text.split())
    print(f"{lang:10s}: {len(tokens):3d} tokens, "
        f"{len(text.split()):2d} words, "
        f"ratio = {ratio:.1f} tokens/word")

Cost Arithmetic

API providers charge per token. Pricing varies by model and provider: as of early 2025, rates range from roughly $1 to $30 per million input tokens depending on the model. If your application processes 10 million words per day, the choice of tokenizer directly affects your monthly bill:

The difference between 1.2 and 3.5 tokens per word is nearly a 3x cost multiplier. Understanding your tokenizer's behavior on your specific data is not an academic exercise; it has direct financial implications.

Figure 1.5.3: The same greeting consumes vastly different amounts of the context window depending on language, due to tokenizer efficiency differences.

Tokenization Artifacts and Their Downstream Effects

Tokenization is not a lossless compression of text. The boundaries where the tokenizer decides to split (or not split) create artifacts that propagate through the model's behavior. Some of these artifacts are subtle; others cause spectacular failures.

Artifact 1: Inconsistent Splitting

The same word can be tokenized differently depending on context. Leading spaces, capitalization, and surrounding punctuation all affect how a subword tokenizer segments text. Consider how GPT-4's tokenizer handles the word "token" in different contexts:

# Demonstrating context-sensitive tokenization
import tiktoken
enc = tiktoken.encoding_for_model("gpt-4")
examples = [
    "token", # bare word
    " token", # with leading space
    "Token", # capitalized
    "TOKEN", # all caps
    "tokenization", # as part of longer word
    " tokenization", # with leading space, longer word
    ]
for ex in examples:
    ids = enc.encode(ex)
    pieces = [enc.decode([i]) for i in ids]
    print(f" {repr(ex):25s} => {pieces}")

Notice that "token" and " token" (with a leading space) are entirely different tokens in the vocabulary. This is by design: leading spaces are attached to the following word so that the tokenizer can reconstruct the original text faithfully. But it means the model sees different input IDs for what a human would consider the same word. The model must learn that these represent the same concept, which requires extra training data and capacity.

Artifact 2: Arithmetic Failures

One of the most widely discussed tokenization artifacts is the difficulty LLMs have with arithmetic. Numbers are tokenized inconsistently: "380" might be a single token, "381" might be split into ["38", "1"], and "3810" might become ["38", "10"]. The model has no built-in notion that these tokens represent digits in a positional number system. It must learn addition, subtraction, and other operations from patterns in the training data, and the inconsistent tokenization makes this much harder.

# See how numbers tokenize differently
import tiktoken
enc = tiktoken.encoding_for_model("gpt-4")
numbers = ["128+256", "100+200", "3810", "381", "380"]
for num in numbers:
    ids = enc.encode(num)
    pieces = [enc.decode([i]) for i in ids]
    print(f" {num:10s} => {pieces}")

Notice how "380" is a single token but "381" splits into ["38", "1"], and "3810" becomes ["38", "10"]. The model receives entirely different representations for numbers that differ by just one digit. This inconsistency is a major reason why LLMs struggle with arithmetic.

Artifact 3: The "Trailing Space" Problem

Because many tokenizers attach leading whitespace to tokens, the model treats " Hello" and "Hello" as fundamentally different inputs. This can cause unexpected behavior when building prompts programmatically. If you accidentally include or omit a space before a key word, the model may interpret it differently. This is especially tricky in few-shot prompting, where consistent formatting is critical.

Artifact 4: Tokenization of Code

Programming languages create unique challenges. Indentation is semantically meaningful in Python, yet a tokenizer may split indentation inconsistently. Four spaces might be one token in one context and two tokens in another. Variable names in camelCase or snake_case get split at different points. Modern tokenizers (like those used in code-focused models such as Codex or StarCoder) address this by including common indentation patterns and code-specific tokens in their vocabulary.

# How code gets tokenized (using tiktoken)
import tiktoken
enc = tiktoken.encoding_for_model("gpt-4")
code = """def fibonacci(n):
 if n <= 1:
 return n
 return fibonacci(n-1) + fibonacci(n-2)"""
tokens = enc.encode(code)
pieces = [enc.decode([t]) for t in tokens]
print(f"Total tokens: {len(tokens)}")
print(f"Token pieces: {pieces}")

Notice how indentation, newlines, and even the function name get merged into multi-character tokens. The four-space indentation appears as a single token in some lines but might be split differently in others, depending on what follows.

Figure 1.5.5a: Tokenization artifacts propagate through the model pipeline, causing unexpected failures in downstream tasks like arithmetic.

Practical Implications for Builders

If you are building applications on top of LLMs, tokenization behavior should inform several design decisions:

• Prompt budgeting: Always count tokens, not words or characters, when estimating whether your prompt fits in the context window. Use the model provider's tokenizer library (such as tiktoken for OpenAI models or the Hugging Face tokenizers library) to get exact counts.
• Multilingual applications: Test your prompts in all target languages to understand the token expansion factor. You may need larger context windows or shorter system prompts for languages that tokenize less efficiently.
• Structured output: JSON, XML, and other structured formats use delimiters (braces, brackets, quotes) that each consume tokens. A compact JSON response uses fewer tokens than a verbose one, directly reducing cost.
• Retrieval-Augmented Generation (RAG): When chunking documents for retrieval, chunk by token count rather than by word count or character count to avoid exceeding context limits.

Exercises

import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
PRICE = {"gpt-4o": (5, 15)}  # $/M input, $/M output
def estimate(prompt, model="gpt-4o", expected_out=500):
  n_in = len(enc.encode(prompt))
  p_in, p_out = PRICE[model]
  return {"in_tokens": n_in, "est_cost_usd": (n_in * p_in + expected_out * p_out) / 1_000_000}