Text Preprocessing & Classical Representations

I turned "the cat sat on the mat" into a 50,000-dimensional sparse vector. The cat was unimpressed.

Lexica, Hopelessly Sparse AI Agent

From Text to Numbers

Here is a question that sounds trivial until you actually try to answer it: how do you turn a sentence into numbers? The word "cat" is not inherently closer to "dog" than to "refrigerator" in any mathematical sense. Yet somehow, you need to feed text into a model that only understands vectors. This is the foundational challenge of NLP.

A machine learning model cannot read text. It operates on vectors of numbers. So every NLP system must convert raw, messy human text (with all its typos, abbreviations, Unicode characters, and grammatical variations) into a numerical representation that a model can process. This section covers the preprocessing pipeline that cleans up the mess, and the earliest (and simplest) approaches to representing the result mathematically.

The Text Preprocessing Pipeline

Here is a typical classical NLP preprocessing pipeline, from raw text to features:

Each step in this pipeline makes a deliberate tradeoff. Let us understand them before writing code:

• Unicode normalization: Converts "café" to "cafe" and "naïve" to "naive". Without this, the same word in different Unicode forms would be treated as different tokens.
• Lowercasing: Makes "The", "the", and "THE" identical. We lose the information that a capitalized word might be a proper noun, but we reduce vocabulary size significantly.
• Stop word removal: Words like "the", "is", "at", "which" appear in virtually every document and carry little distinguishing information. Removing them reduces noise. But be careful: "to be or not to be" becomes meaningless without stop words!
• Stemming vs. Lemmatization: Both reduce words to a base form, but differently:
  • Stemming (Porter Stemmer): chops off suffixes using rules. "running" → "run", "studies" → "studi", "university" → "univers". Fast but crude; often produces non-words.
  • Lemmatization (WordNet): looks up the actual dictionary base form. "running" → "run", "studies" → "study", "better" → "good". Slower but produces real words. This is what we will use.

Figure 1.2.1: Stemming vs. lemmatization. Stemming chops suffixes with rules and often produces non-words; lemmatization uses a dictionary to find the true base form.

Let us implement the full pipeline in Python:

# Complete text preprocessing pipeline
import re
import unicodedata
import nltk
from nltk.corpus import stopwords
from nltk.stem import WordNetLemmatizer
nltk.download('stopwords', quiet=True)
nltk.download('wordnet', quiet=True)
def preprocess(text: str) -> list[str]:
    """Full preprocessing pipeline: raw text to clean token list."""
    # Step 1: Unicode normalization (e.g., "café" to "cafe")
    text = unicodedata.normalize('NFKD', text)
    # Step 2: Lowercase
    text = text.lower()
    # Step 3: Remove non-alphabetic characters (keep spaces)
    text = re.sub(r'[^a-z\s]', '', text)
    # Step 4: Tokenize (split on whitespace)
    tokens = text.split()
    # Step 5: Remove stop words
    stop_words = set(stopwords.words('english'))
    tokens = [t for t in tokens if t not in stop_words]
    # Step 6: Lemmatize
    # Note: WordNetLemmatizer defaults to noun lookup when no POS tag is provided.
    # This means "running" stays as "running" (it is not a noun). To lemmatize verbs
    # correctly, you must pass pos='v'. Compare the spaCy version below, which handles
    # POS tagging automatically.
    lemmatizer = WordNetLemmatizer()
    tokens = [lemmatizer.lemmatize(t) for t in tokens]
    return tokens
# Example
raw = "The cats were running quickly across the beautiful gardens!"
print(preprocess(raw))
# Output: ['cat', 'running', 'quickly', 'across', 'beautiful', 'garden']
# Notice "running" is NOT lemmatized to "run" because no POS tag was provided.
# NLTK's WordNetLemmatizer assumes every word is a noun by default.

# Bag-of-Words with scikit-learn: CountVectorizer tokenizes documents
# and builds a term-frequency matrix where each row is a document vector.
from sklearn.feature_extraction.text import CountVectorizer
documents = [
    "The cat sat on the mat",
    "The dog sat on the log",
    "The cat chased the dog",
]
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(documents)
print("Vocabulary:", vectorizer.get_feature_names_out())
print("Vectors:")
print(X.toarray())
# Output:
# Vocabulary: ['cat' 'chased' 'dog' 'log' 'mat' 'on' 'sat' 'the']
# Vectors:
# [[1 0 0 0 1 1 1 2] "The cat sat on the mat"
# [0 0 1 1 0 1 1 2] "The dog sat on the log"
# [1 1 1 0 0 0 0 2]] "The cat chased the dog"

Here is the same pipeline using spaCy, the industry standard for modern NLP preprocessing. Notice how much simpler it is, because spaCy handles tokenization, POS tagging, and lemmatization in one pass:

# Same pipeline using spaCy (modern, production-grade)
import spacy
# Load the English model (run: python -m spacy download en_core_web_sm)
nlp = spacy.load("en_core_web_sm")
def preprocess_spacy(text: str) -> list[str]:
    """Preprocessing with spaCy: tokenize, lemmatize, remove stop words in one pass."""
    doc = nlp(text)
    return [token.lemma_.lower() for token in doc
        if not token.is_stop
        and not token.is_punct
        and token.is_alpha]
raw = "The cats were running quickly across the beautiful gardens!"
print(preprocess_spacy(raw))
# Output: ['cat', 'run', 'quickly', 'beautiful', 'garden']
# Bonus: spaCy gives you POS tags, entities, and dependencies for free
doc = nlp("Apple released the iPhone 16 in San Francisco.")
for ent in doc.ents:
    print(f" {ent.text:20s} -> {ent.label_}")
        # Output:
            # Apple -> ORG
            # iPhone 16 -> PRODUCT (may vary by model)
            # San Francisco -> GPE

Bag-of-Words (BoW)

The simplest way to turn text into numbers: count how many times each word appears. Each document becomes a vector where each dimension corresponds to a word in the vocabulary.

Let us walk through the process step by step. Given three documents:

1. "The cat sat on the mat"
2. "The dog sat on the log"
3. "The cat chased the dog"

Step 1: Build the vocabulary by collecting all unique words: {cat, chased, dog, log, mat, on, sat, the}. That is 8 unique words, so every document will become an 8-dimensional vector.

Step 2: For each document, count how many times each vocabulary word appears. The word "the" appears twice in document 1, so its entry is 2. The word "chased" does not appear at all, so its entry is 0.

Step 3: The result is a matrix where each row is a document and each column is a word. Notice how sparse it is: most cells are zero. This is the fundamental inefficiency of BoW.

# TF-IDF: weight terms by how important they are to each document.
# Common words like "the" get down-weighted; distinctive words get amplified.
from sklearn.feature_extraction.text import TfidfVectorizer
vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(documents)
print("TF-IDF vectors (rounded):")
import numpy as np
print(np.round(X.toarray(), 2))
# Notice: "the" gets a LOW weight (appears in every doc)
# while "mat", "log", "chased" get HIGH weights (distinctive)

N-grams: Partially Solving the Word Order Problem

One partial fix for BoW's word-order blindness is to count n-grams (sequences of n consecutive words) instead of individual words. Bigrams (n=2) like "dog bites" and "bites man" at least capture some local word order:

# Bigrams capture some word order
from sklearn.feature_extraction.text import CountVectorizer
bigram_vec = CountVectorizer(ngram_range=(1, 2)) # unigrams + bigrams
X = bigram_vec.fit_transform(["dog bites man", "man bites dog"])
print("Features:", bigram_vec.get_feature_names_out())
print("Vectors:")
print(X.toarray())
# Features: ['bites' 'bites dog' 'bites man' 'dog' 'dog bites' 'man' 'man bites']
# Vectors:
# [[1 0 1 1 1 1 0] "dog bites man" has "dog bites" and "bites man"
# [1 1 0 1 0 1 1]] "man bites dog" has "man bites" and "bites dog"
# Now the vectors ARE different! But the vocabulary explodes quadratically.

Three fatal flaws of Bag-of-Words:

1. No word order: "Dog bites man" and "Man bites dog" have the exact same vector
2. No semantics: "Happy" and "joyful" are as far apart as "happy" and "refrigerator". Because BoW treats each word as an independent dimension, there is no mechanism to know that two different words share meaning; the only information is whether a word is present and how often.
3. Massive dimensionality: With a vocabulary of 100,000 words, each document is a 100,000-dimensional vector that is mostly zeros

TF-IDF: Weighting Words by Importance

TF-IDF (Term Frequency, Inverse Document Frequency) improves on raw counts by giving more weight to words that are distinctive for a document. Common words like "the" get downweighted because they appear everywhere; rare but informative words get upweighted.

Where:

• $TF(t, d)$ = how often term t appears in document d
• $DF(t)$ = how many documents contain term t
• $N$ = total number of documents

Worked example: Using our three documents from above. Consider the word "the":

• TF("the", doc1) = 2 (appears twice in "The cat sat on the mat")
• DF("the") = 3 (appears in all three documents)
• IDF = log(3/3) = log(1) = 0 → "the" gets zero weight! It is useless for distinguishing documents.

Now consider the word "chased":

• TF("chased", doc3) = 1
• DF("chased") = 1 (only appears in one document)
• IDF = log(3/1) = log(3) ≈ 1.1 → high weight! "Chased" is distinctive.

This is the core intuition: words that appear everywhere are useless; words that appear in only some documents are informative. TF-IDF automatically discovers this.

TF-IDF Variants: Sublinear Scaling and Normalization

The formula above is the simplest version. In practice, two refinements are standard:

Sublinear TF scaling: Raw term frequency can be misleading. A word that appears 20 times in a document is not 20 times more important than a word that appears once. The sublinear variant replaces TF with $1 + \log(TF)$, compressing the scale:

With this formula, a word appearing once gets weight 1.0, a word appearing 10 times gets weight 2.0, and a word appearing 100 times gets weight 3.0. This prevents extremely frequent terms from dominating the vector representation.

Document length normalization: Longer documents naturally have higher term frequencies. To make vectors comparable, each document's TF-IDF vector is typically normalized to unit length (L2 normalization). This ensures that a 10,000-word article and a 50-word abstract can be fairly compared using cosine similarity.

Both refinements ship as flags on scikit-learn's TfidfVectorizer, so the sublinear-TF and L2-normalized variant is a one-line change from the default:

from sklearn.feature_extraction.text import TfidfVectorizer

docs = [
    "the cat sat on the mat",
    "the dog chased the cat",
    "the senator filed a bill",
]
# sublinear_tf=True applies 1 + log(tf); norm='l2' makes rows unit-length
vec = TfidfVectorizer(sublinear_tf=True, norm="l2")
X = vec.fit_transform(docs)
print(X.shape)             # (3, vocab_size)
print(vec.get_feature_names_out()[:5])

Figure 1.2.2a: TF-IDF weights for a sample document. Words that appear in many documents (like "the") receive zero weight, while rare, distinctive words (like "chased") receive high weight.

# One-hot encoding: see the problem for yourself
from sklearn.preprocessing import LabelEncoder, OneHotEncoder
import numpy as np
words = ["cat", "dog", "fish", "democracy"]
label_enc = LabelEncoder()
integer_encoded = label_enc.fit_transform(words)
onehot_enc = OneHotEncoder(sparse_output=False)
onehot = onehot_enc.fit_transform(integer_encoded.reshape(-1, 1))
print("One-hot vectors:")
for word, vec in zip(words, onehot):
    print(f" {word:12s} -> {vec}")
    # Compute distances: every pair is equally far apart!
    from scipy.spatial.distance import euclidean
    print(f"\nDistance cat to dog: {euclidean(onehot[0], onehot[1]):.2f}")
    print(f"Distance cat to democracy: {euclidean(onehot[0], onehot[3]):.2f}")
    # Both print 1.41 (sqrt of 2): cat is as "far" from dog as from democracy!

One-Hot Encoding and Its Limitations

The most basic word representation: each word is a vector with a single 1 and all other entries 0. With a vocabulary of 50,000 words, "cat" might be:

# One-hot encoding: every word becomes a sparse high-dimensional vector
# The problem: two words that mean the same thing are EQUALLY DIFFERENT
# from each other as two completely unrelated words.
import numpy as np

vocab = ["cat", "kitten", "feline", "automobile", "car"]
V = len(vocab)
one_hots = {word: np.eye(V)[i] for i, word in enumerate(vocab)}

# Distance between any two distinct words is the same constant.
def dist(w1, w2):
    return float(np.linalg.norm(one_hots[w1] - one_hots[w2]))

print(f"cat vs kitten   : {dist('cat', 'kitten'):.3f}")
print(f"cat vs automobile: {dist('cat', 'automobile'):.3f}")
print(f"cat vs car      : {dist('cat', 'car'):.3f}")
# All print 1.414 -- one-hot encoding has no notion of similarity.

The problem is devastating: every word is equally different from every other word. The Euclidean distance between any two one-hot vectors is always √2, regardless of which two words you pick (because each pair differs in exactly two positions). The distance between "cat" and "dog" is identical to the distance between "cat" and "democracy." There is no notion of similarity, no shared meaning, no semantic structure whatsoever.

From Words to Topics: LSA and LDiA

Bag-of-Words and TF-IDF treat each vocabulary word as an independent dimension. The result is long, sparse, and semantically blind: "artist" and "painter" land in completely unrelated coordinates, and "bank" gets the same dimension regardless of whether the document discusses rivers or money. A natural fix is to compress the vocabulary axis into a much shorter list of latent topics, where each topic is a soft cluster of words that tend to co-occur. A typical topic representation might describe a document as "80% sport, 20% news" instead of as a 50,000-dimensional count vector.

The bridge from sparse word-count vectors to dense, learned embeddings (Section 1.3) runs through two classical algorithms that compute these topic representations: Latent Semantic Analysis (LSA) via truncated SVD and Latent Dirichlet Allocation (LDiA) as a probabilistic mixture model.

The Document-Term Matrix

Both methods start from the same input: a document-term matrix (DTM) of shape $N \times K$, where $N$ is vocabulary size and $K$ is the number of documents. Each entry holds the affinity between a word and a document, typically a raw count, a binary presence indicator, or a TF-IDF weight. The DTM is the same Bag-of-Words table from earlier, transposed and viewed as a single matrix to be factorized.

LSA via Truncated SVD

LSA approximates the DTM as the product of two low-rank matrices: a document-topic matrix and a topic-term matrix, joined by a diagonal matrix of singular values that ranks topic importance. The user picks the rank $r$, which becomes the number of topics. Rows of the document-topic matrix can then be used as dense $r$-dimensional document features for any downstream classifier or retriever, and low singular values flag noisy, incoherent topics that can safely be discarded.

Formally, the full singular value decomposition factorizes the document-term matrix as $X = U \Sigma V^{\top}$, where $U \in \mathbb{R}^{N \times N}$ holds the term-topic directions, $\Sigma$ is a diagonal matrix of singular values sorted in decreasing order, and $V \in \mathbb{R}^{K \times K}$ holds the document-topic directions. LSA keeps only the top $r$ singular values:

$$ X \;\approx\; X_r \;=\; U_r\, \Sigma_r\, V_r^{\top} $$

The Eckart-Young theorem guarantees that $X_r$ is the best rank-$r$ approximation of $X$ under the Frobenius norm; that is, it minimizes $\|X - X_r\|_F^{2}$ over all rank-$r$ matrices. The rows of $V_r \Sigma_r$ are the dense $r$-dimensional document embeddings, and the columns of $U_r \Sigma_r$ are the corresponding term embeddings in the same shared topic space.

In scikit-learn this is one call to TruncatedSVD on a TF-IDF matrix:

# LSA = TF-IDF + Truncated SVD. The result is a dense doc-topic matrix.
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.decomposition import TruncatedSVD
docs = [
    "the cat sat on the mat",
    "the dog chased the cat",
    "the senator filed a bill",
    "congress voted on the bill",
]
tfidf = TfidfVectorizer().fit_transform(docs)
lsa = TruncatedSVD(n_components=2)      # 2 topics
doc_topic = lsa.fit_transform(tfidf)  # shape: (n_docs, 2)
print(doc_topic.round(2))
# Animal docs cluster on topic 0; politics docs cluster on topic 1.

LSA is still used in production, notably in semantic search and in retrieval baselines where a 200-dimensional dense vector is cheaper to index than a 50,000-dimensional sparse one. It is also the historical bridge from purely lexical retrieval (TF-IDF, BM25) to the dense vector retrieval covered in Chapter 31.

LDiA as a Bayesian Mixture

Latent Dirichlet Allocation (LDiA) models each topic as a distribution over words and each document as a mixture of those topic distributions. Where LSA factorizes a matrix, LDiA performs full Bayesian inference over the topic-word and document-topic distributions, typically with Gibbs sampling (covered later in Chapter 4's decoding-and-sampling material) or with variational inference. The number of topics is a hyperparameter, swept via topic coherence or held-out perplexity. LDiA remains the standard for interpretable topic models in archive analysis, survey free-text coding, and product-review mining where the deliverable is a human-readable topic, not a raw embedding.

Formally, LDiA places a Dirichlet prior on each document's topic mixture and on each topic's word distribution. For a corpus with $K$ topics and vocabulary of size $V$, the generative story for a document of length $N$ is:

Here $\theta_d \in \Delta^{K-1}$ is the document's topic mixture, $\phi_k \in \Delta^{V-1}$ is topic $k$'s word distribution, $z_{d,n}$ is the latent topic assignment for the $n$-th token, and $w_{d,n}$ is the observed word. The hyperparameters $\alpha$ and $\beta$ control sparsity: small $\alpha$ pushes each document toward a few dominant topics, and small $\beta$ pushes each topic toward a few dominant words. Inference recovers the posterior $p(\theta, \phi, z \mid w)$, which yields the interpretable topic-word and document-topic distributions used downstream.

In gensim the call is a one-liner once the corpus is in bag-of-words form:

from gensim.corpora import Dictionary
from gensim.models import LdaModel

docs = [
    ["cat", "sat", "mat"],
    ["dog", "chased", "cat"],
    ["senator", "filed", "bill"],
    ["congress", "voted", "bill"],
]
dictionary = Dictionary(docs)
corpus = [dictionary.doc2bow(d) for d in docs]

lda = LdaModel(
    corpus=corpus, id2word=dictionary,
    num_topics=2, alpha="auto", eta="auto",
    passes=20, random_state=42,
)
for k, words in lda.print_topics(num_words=4):
    print(f"Topic {k}: {words}")
# Topic 0: 0.18*"cat" + 0.13*"dog" + 0.12*"mat" + 0.10*"chased"
# Topic 1: 0.19*"bill" + 0.14*"congress" + 0.13*"senator" + 0.10*"voted"

So here is where we stand: Bag-of-Words and TF-IDF can turn text into numbers, and they are surprisingly effective for tasks like search and simple classification. But they have a fatal flaw: they have no concept of meaning. "Happy" and "joyful" are as unrelated as "happy" and "refrigerator." The vectors are sparse (mostly zeros), high-dimensional (one dimension per word in the entire vocabulary), and semantically blind.

What we need is a representation where similar words are close together, where the geometry of the vector space reflects the meaning of the words. Enter word embeddings.

Exercises

import numpy as np, math, collections
docs = [d.split() for d in corpus]
vocab = sorted({w for d in docs for w in d})
idx = {w: i for i, w in enumerate(vocab)}
tf = np.zeros((len(docs), len(vocab)))
for i, d in enumerate(docs):
  for w, c in collections.Counter(d).items(): tf[i, idx[w]] = c
df = (tf > 0).sum(axis=0)
idf = np.log((1 + len(docs)) / (1 + df)) + 1   # smoothed IDF
tfidf = tf * idf