Special Tokens, Chat Templates, and Tiktoken

My chat template puts the system prompt in just the right place.

Token, Boundary-Confused AI Agent

Special Tokens

Beyond the subword vocabulary, every tokenizer includes a set of special tokens that serve structural purposes. These tokens are never produced by the subword algorithm itself; they are manually added to the vocabulary and carry specific meanings that the model learns during training. Understanding special tokens is essential for correctly formatting inputs and interpreting outputs.

Common Special Tokens

As you can see, the same concept (marking sequence boundaries) appears under many different names across different model families.

Chat Templates

Modern LLMs that support conversation (ChatGPT, Claude, Llama Chat, Mistral Instruct) use a chat template that wraps user messages, system prompts, and assistant responses in a specific format using special tokens. The model was trained to expect this exact format, and deviating from it can degrade performance or cause unexpected behavior. We explore how to use these templates effectively through LLM APIs (Section 11.1) and prompt engineering (Chapter 12).

Example: ChatML Format

The ChatML format (used by some OpenAI models) wraps each message with role tags:

# ChatML template structure
template = """<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
What is tokenization?<|im_end|>
<|im_start|>assistant
"""
# The model generates its response here, ending with <|im_end|>
print(template)

You now understand how tokenizers are trained, but knowing the algorithm is only half the story. When you actually deploy an LLM, you will encounter a different set of questions: What are those mysterious <|system|> tokens? Why does my Japanese prompt cost four times as much as the English version? How do I format a multi-turn conversation correctly? This section covers five practical topics that connect tokenization theory to real-world usage: special tokens, chat templates, multilingual fertility, multimodal tokenization, and API cost estimation.

Example: Llama-3 Chat Format

Llama-3 uses a distinct set of special tokens to delimit system, user, and assistant turns in multi-turn conversations.

# Llama 3 chat template
template = """<|begin_of_text|><|start_header_id|>system<|end_header_id|>
You are a helpful assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>
What is tokenization?<|eot_id|><|start_header_id|>assistant<|end_header_id|>
"""

Notice that the special tokens differ between models, and the exact placement of newlines matters. The Hugging Face transformers library provides a apply_chat_template() method that handles this formatting automatically:

# Using Hugging Face chat templates
from transformers import AutoTokenizer
# Load tokenizer with vocabulary matching the model
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is tokenization?"},
]
formatted = tokenizer.apply_chat_template(
    messages,
    tokenize=False, # return string, not token IDs
    add_generation_prompt=True # add the assistant header
)
print(formatted)

<|im_start|>system
You are a helpful assistant.
<|im_end|>

<|im_start|>user
Explain BPE vs WordPiece
<|im_end|>

<|im_start|>assistant
Response...
<|im_end|>

The Tiktoken Library

tiktoken is OpenAI's open-source tokenizer library, written in Rust with Python bindings for performance. It implements the BPE tokenizers used by GPT-3.5, GPT-4, GPT-4o, and related models. For any application that interacts with OpenAI's APIs, tiktoken is the authoritative tool for counting tokens, estimating costs, and debugging tokenization behavior. It is also widely used as a general-purpose BPE tokenizer for non-OpenAI workflows because of its speed and simplicity.

Installation and Basic Usage

The following snippet installs tiktoken, loads a model-specific encoding, and tokenizes a sample string.

# Install tiktoken
# pip install tiktoken
import tiktoken
# Load by model name (recommended)
enc = tiktoken.encoding_for_model("gpt-4o")
# Or load by encoding name directly
enc_cl100k = tiktoken.get_encoding("cl100k_base") # GPT-4, GPT-3.5
enc_o200k = tiktoken.get_encoding("o200k_base") # GPT-4o
# Encode text to token IDs
text = "Tokenizers split text into subword units."
tokens = enc.encode(text)
print(f"Text: {text}")
print(f"Token IDs: {tokens}")
print(f"Token count: {len(tokens)}")
# Decode token IDs back to text
decoded = enc.decode(tokens)
print(f"Decoded: {decoded}")
# Inspect individual tokens
for token_id in tokens:
    token_bytes = enc.decode_single_token_raw(token_id)
    print(f" {token_id:6d} -> {token_bytes}")

Two key details deserve attention. First, tiktoken is significantly faster than pure Python tokenizers because the core BPE algorithm runs in Rust. Tokenizing a million characters takes roughly 100ms with tiktoken versus 2 to 5 seconds with a pure Python implementation. This matters for batch processing and real-time cost estimation. Second, different OpenAI models use different encoding schemes: cl100k_base (100,256 token vocabulary) for GPT-4 and GPT-3.5, and o200k_base (200,019 token vocabulary) for GPT-4o. Always match the encoding to the model you are calling, or use encoding_for_model() to let tiktoken select automatically.

# pip install transformers
from transformers import AutoTokenizer
# Load the tokenizer that ships with a specific model
tok = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-1B")
text = "Tokenization determines cost and context usage."
ids = tok.encode(text)
print("Token IDs:", ids)
print("Tokens:", tok.convert_ids_to_tokens(ids))
print("Decoded:", tok.decode(ids))
print(f"Token count: {len(ids)}")

Tip: SentencePiece in Practice

Load a SentencePiece model directly (used by T5, ALBERT, and Llama 1/2). SentencePiece treats text as a raw Unicode byte stream and scores any segmentation $S = (t_1, \ldots, t_m)$ of a word under a unigram language model:

$$ P(S) = \prod_{i=1}^{m} P(t_i), \qquad S^{*} = \arg\max_{S} P(S) $$

Decoding picks the highest-probability segmentation via Viterbi. For training-time data augmentation, the library also samples a segmentation from a temperature-smoothed distribution (subword regularization, Kudo 2018):

$$ P_{\alpha}(S) \propto \bigl(\prod_i P(t_i)\bigr)^{\alpha}, \qquad 0 < \alpha \le 1 $$

Lower $\alpha$ flattens the distribution and surfaces alternative segmentations; $\alpha = 1$ recovers the deterministic Viterbi tokenization.

# pip install sentencepiece
import sentencepiece as spm
# Load a pre-trained SentencePiece model (e.g., from a T5 download)
# sp = spm.SentencePieceProcessor(model_file="spiece.model")
# Or train a tiny one for demonstration
import tempfile, os
tmp = tempfile.NamedTemporaryFile(mode="w", suffix=".txt", delete=False)
tmp.write("Language models learn subword tokenization.\n" * 100)
tmp.close()
spm.SentencePieceTrainer.train(
    input=tmp.name, model_prefix="demo_sp", vocab_size=64,
    model_type="bpe"
)
sp = spm.SentencePieceProcessor(model_file="demo_sp.model")
print("Pieces:", sp.encode("Language models", out_type=str))
os.unlink(tmp.name)

You now know how text becomes token IDs. In Chapter 2, you will learn how those token sequences are processed: first by recurrent neural networks that read one token at a time, then by the attention mechanism that lets the model look at all tokens simultaneously.