Word Embeddings: Word2Vec, GloVe & FastText

King minus man plus woman equals queen. I tried this with my coworkers and HR got involved.

Lexica, Analogically Reckless AI Agent

The Distributional Hypothesis

In 2013, Tomas Mikolov and colleagues at Google published a paper that would reshape all of NLP. The idea was elegantly simple: instead of defining word representations by hand, learn them from data. The result was Word2Vec: dense vectors where semantically similar words are geometrically close.

"You shall know a word by the company it keeps."

J.R. Firth, 1957 (the distributional hypothesis)

This idea (the distributional hypothesis) says that words appearing in similar contexts tend to have similar meanings. Why does this work? Consider: the words "cat" and "dog" both appear near "pet," "veterinarian," "cute," "fed," and "walked." A word you have never seen, like "wug," that also appears near "pet" and "fed" is probably an animal too. Context is a remarkably reliable proxy for meaning. This principle is the foundation of all modern word representations, including the embeddings inside GPT-4 and Claude.

Word2Vec: How It Works

Word2Vec comes in two flavors. We will focus on Skip-gram, which is simpler to understand and more widely used.

The idea: given a center word, predict the surrounding context words. Let us trace through a concrete example.

Take the sentence: "the cat sat on the mat" with a window size of 2. The model slides a window across the sentence, and at each position, creates training pairs:

Figure 1.3.1: The Skip-gram sliding window. The center word "sat" is paired with each context word within a window of size 2, generating training pairs.

After processing billions of such pairs, words that frequently appear in similar contexts (like "cat" and "dog," which both appear near "the," "sat," "chased") end up with similar vectors. Words that never share context (like "cat" and "economics") end up far apart.

The Architecture (It Is Surprisingly Simple)

Skip-gram is a shallow neural network with just one hidden layer:

1. Input: One-hot vector for the center word (dimension = vocabulary size V)
2. Hidden layer: Multiply by weight matrix $W$ (V × d): this produces a d-dimensional vector. This IS the word embedding.
3. Output: Multiply by another matrix $W'$ (d × V), apply softmax: this gives a probability distribution over all words in the vocabulary

Figure 1.3.2: The Skip-gram neural network. A one-hot center word vector is multiplied by the embedding matrix W to produce a hidden representation, which is then used to predict context words.

Formally, the network computes the probability of each context word by applying a softmax over the output of the two weight matrices:

Negative Sampling: Making Training Tractable

The naive softmax over a vocabulary of 100,000+ words is extremely expensive. Negative sampling simplifies this: instead of updating all 100K output weights, we only update the weights for the correct context word (positive) and a small random sample of "negative" words (typically 5 to 20).

In plain English: make the dot product between the center word and the real context word large (positive), and make the dot product with random words small (negative). The key insight is that this approximation works because the vast majority of vocabulary words are irrelevant to any given context. Sampling just a handful of negatives is representative of the full vocabulary, much like polling a few thousand people can predict a national election.

Think of it like a multiple-choice test: instead of ranking every word in the dictionary, the model only needs to pick the right answer from a handful of options. If it can reliably distinguish the correct context word from five random distractors, it has learned something meaningful about word relationships.

# Skip-gram forward pass implemented from scratch with NumPy.
# Demonstrates that the embedding for the center word is just a matrix row
# lookup (W^T x with x one-hot), and that a softmax converts the output
# logits into a probability distribution over context-word candidates.
import numpy as np

V, d = 5, 3                                # tiny vocabulary, 3-dim embeddings
vocab = ["the", "cat", "sat", "on", "mat"]

rng = np.random.default_rng(0)
W = rng.standard_normal((V, d)) * 0.1      # input embedding matrix (rows = words)
W_out = rng.standard_normal((d, V)) * 0.1  # output projection matrix

center_idx = vocab.index("sat")            # row 2 of W
x = np.zeros(V); x[center_idx] = 1.0       # one-hot center vector

h = W.T @ x                                # hidden vector == W[center_idx]
assert np.allclose(h, W[center_idx])       # the embedding IS a row of W

logits = W_out.T @ h                       # length-V scores for context words
probs = np.exp(logits) / np.exp(logits).sum()

for word, p in zip(vocab, probs):
    print(f"P({word} | sat) = {p:.3f}")

P(the | sat) = 0.198
P(cat | sat) = 0.201
P(sat | sat) = 0.200
P(on | sat) = 0.202
P(mat | sat) = 0.199

Training Word2Vec from Scratch

Let us train a Word2Vec model using Gensim on a real corpus:

# Train Word2Vec with gensim: build a skip-gram model on a small corpus,
# then query the resulting vectors for nearest neighbors and analogies.
from gensim.models import Word2Vec
from gensim.utils import simple_preprocess
# Sample corpus (in practice, use millions of sentences)
corpus = [
    "the king ruled the kingdom with wisdom",
    "the queen ruled the kingdom with grace",
    "the prince and princess lived in the castle",
    "the man worked in the field every day",
    "the woman worked in the market every day",
    "a dog chased a cat across the garden",
    "the cat sat on the mat near the dog",
    "paris is the capital of france",
    "berlin is the capital of germany",
    "tokyo is the capital of japan",
]
# Tokenize
sentences = [simple_preprocess(s) for s in corpus]
# Train Word2Vec (Skip-gram with negative sampling)
model = Word2Vec(
    sentences,
    vector_size=50, # embedding dimensions
    window=3, # context window size
    min_count=1, # minimum word frequency
    sg=1, # 1 = Skip-gram, 0 = CBOW
    negative=5, # number of negative samples
    epochs=100, # training epochs
)
# Explore the learned embeddings
print("Vector for 'king':", model.wv['king'][:5], "...")
print("Most similar to 'king':", model.wv.most_similar('king', topn=3))
print("Most similar to 'cat':", model.wv.most_similar('cat', topn=3))
# Peek inside: the embedding matrix is just a numpy array
print(f"\nEmbedding matrix shape: {model.wv.vectors.shape}")
# Output: (num_words, 50): each row is one word's vector

# Measuring cosine similarity between word vectors
from numpy import dot
from numpy.linalg import norm
def cosine_sim(a, b):
    return dot(a, b) / (norm(a) * norm(b))
# Using pre-trained Word2Vec vectors
print(cosine_sim(wv['cat'], wv['dog'])) # ~0.76 (both are pets)
print(cosine_sim(wv['cat'], wv['king'])) # ~0.13 (unrelated)
print(cosine_sim(wv['king'], wv['queen'])) # ~0.65 (both are royalty)
print(cosine_sim(wv['paris'], wv['france'])) # ~0.77 (capital-country)

Measuring Similarity: Cosine Similarity

Before we explore analogies, we need to understand how similarity is measured between word vectors. The standard metric is cosine similarity: the cosine of the angle between two vectors.

Figure 1.3.5: Geometric intuition for cosine similarity. Two vectors with a small angle between them (high cosine) are semantically similar; a large angle (low cosine) indicates unrelated meanings.

from sentence_transformers import SentenceTransformer, util
model = SentenceTransformer("all-MiniLM-L6-v2")
sims = util.cos_sim(
    model.encode(["A cat sat on the mat", "Dogs are great pets"]),
    model.encode(["Felines resting on rugs", "The stock market crashed"])
)
print(sims) # high similarity for semantically related pairs

# Word analogy: king - man + woman = ?
# (Using pre-trained vectors for reliable results)
import gensim.downloader as api
# Download pre-trained Word2Vec (trained on Google News, 3M words)
# NOTE: This download is approximately 1.7 GB. It may take several minutes.
wv = api.load('word2vec-google-news-300')
# The famous analogy
result = wv.most_similar(positive=['king', 'woman'], negative=['man'], topn=1)
print(result) # [('queen', 0.7118)]
# More analogies
print(wv.most_similar(positive=['paris', 'germany'], negative=['france'], topn=1))
# [('berlin', 0.7327)] Paris is to France as Berlin is to Germany
print(wv.most_similar(positive=['walking', 'swam'], negative=['swimming'], topn=1))
# [('walked', 0.7458)] captures verb tense relationships too!

# 3CosMul analogy scoring with gensim: same inputs, different combiner.
wv.most_similar_cosmul(
    positive=['king', 'woman'],
    negative=['man'],
    topn=1,
)  # typically returns [('queen', score)] more reliably than most_similar()

Word embeddings do more than just place similar words nearby in vector space. The geometric structure they learn encodes surprisingly rich relationships between concepts. Perhaps the most striking demonstration of this is the ability to solve word analogies through simple vector arithmetic.

The Magic of Word Analogies

The most striking property of word embeddings is that they capture relationships as vector arithmetic:

# Loading pre-trained GloVe vectors
import gensim.downloader as api
# Download GloVe (trained on Wikipedia + Gigaword, 400K vocab, 100d)
glove = api.load('glove-wiki-gigaword-100')
# Same interface as Word2Vec
print("GloVe similarity cat/dog:", glove.similarity('cat', 'dog'))
print("GloVe analogy king-man+woman:",
    glove.most_similar(positive=['king', 'woman'], negative=['man'], topn=1))

GloVe: Global Vectors for Word Representation

GloVe (Pennington et al., 2014, Stanford) takes a fundamentally different approach from Word2Vec. Instead of learning from individual (center, context) pairs one at a time, GloVe first builds a global co-occurrence matrix (a giant table counting how often every word appears near every other word across the entire corpus) and then factorizes this matrix into low-dimensional vectors.

Think of it this way: Word2Vec learns by reading one sentence at a time (local context). GloVe first compiles all the statistics, then learns from the complete picture (global statistics). Neither approach is strictly better; they tend to produce similar-quality embeddings, but the mathematical foundations are quite different.

In more precise terms, GloVe trains word vectors so that the dot product of two word vectors equals the logarithm of how often they co-occur. The objective function penalizes any discrepancy between the predicted (dot product) and observed (log co-occurrence count) values, weighted so that neither extremely rare nor extremely frequent pairs dominate the training.

Where $X_{ij}$ is how often word $i$ co-occurs with word $j$, $w_{i}$ and $\tilde{w}_{j}$ are the two word vectors, $b_{i}$ and $\tilde{b}_{j}$ are bias terms, and $f(X_{ij})$ is a weighting function that caps the influence of very frequent pairs (it equals 0 when $X_{ij} = 0$, grows sublinearly, and plateaus at 1 for frequent pairs). The objective forces the dot product of two word vectors to approximate the log of their co-occurrence count.

The real power of GloVe comes from the insight that ratios of co-occurrence probabilities encode meaning:

import matplotlib.pyplot as plt
# Cosine similarity heatmap: compute pairwise similarities between word
# vectors and visualize which words cluster together in embedding space.
import seaborn as sns
import numpy as np
words = ['king', 'queen', 'man', 'woman', 'cat', 'dog', 'paris', 'france']
vectors = np.array([wv[w] for w in words])
# Compute cosine similarity matrix
from sklearn.metrics.pairwise import cosine_similarity
sim_matrix = cosine_similarity(vectors)
# Plot heatmap
plt.figure(figsize=(8, 6))
sns.heatmap(sim_matrix, xticklabels=words, yticklabels=words,
annot=True, fmt=".2f", cmap="RdYlGn", vmin=-0.2, vmax=1)
plt.title("Word Similarity Matrix (Cosine Similarity)")
plt.tight_layout()
plt.show()
# You'll see bright blocks where royalty words cluster together,
# and where animals cluster together, confirming the embedding structure

FastText: Subword Embeddings

FastText (Facebook/Meta, 2016) extends Word2Vec with a critical improvement: it represents each word as a bag of character n-grams. This subword approach foreshadows the subword tokenization methods (BPE, WordPiece) covered in Chapter 1 that modern LLMs rely on. The word "running" with n=3 becomes: <ru, run, unn, nni, nin, ing, ng>

Figure 1.3.7a: FastText subword decomposition. The word "running" is split into character n-grams, each mapped to a vector, and the vectors are summed to produce the final word embedding.

Formally, let $\mathcal{G}_w$ denote the set of character n-grams in word $w$ (with $w$ itself included as a special "whole word" n-gram). FastText represents $w$ as the sum of n-gram vectors:

and trains with the Skip-gram negative-sampling objective in which the context score for a (word, context) pair becomes

Here $z_g \in \mathbb{R}^d$ is the embedding of n-gram $g$ and $u_c$ is the context vector for word $c$. Because $v_w$ is built by summing n-gram vectors, any unseen word still receives an embedding from whichever of its n-grams the model has already learned, which is the property that makes FastText robust to morphology and to typos.

Why this matters:

• Handles unseen words: Even if "unfriend" never appeared in training data, FastText can compose its vector from subwords like "un-", "friend", "-end" etc.
• Morphologically rich languages: Turkish, Finnish, Arabic, where a single word can have many inflected forms, benefit enormously from subword sharing.
• Typos and misspellings: "runnng" is still close to "running" because they share most subword n-grams.

# FastText handles out-of-vocabulary words
from gensim.models import FastText
# Train on same corpus
ft_model = FastText(
    sentences,
    vector_size=50,
    window=3,
    min_count=1,
    sg=1,
    epochs=100,
)
# FastText can produce vectors for UNSEEN words!
print("Vector for 'kingdoms' (never in training data):")
print(ft_model.wv['kingdoms'][:5]) # Works! Uses subword info from 'kingdom'
# Word2Vec would crash: KeyError: "word 'kingdoms' not in vocabulary"

We have now seen three distinct approaches to learning word embeddings, each with its own philosophy: Word2Vec learns from local context windows, GloVe from global co-occurrence statistics, and FastText from subword structure. How do they stack up against each other in practice?

Comparing the Three Approaches

Let us put Word2Vec, GloVe, and FastText side by side:

Visualizing Embeddings

High-dimensional embeddings can be projected to 2D for visualization using t-SNE (preserves local structure) or UMAP (preserves both local and global structure, and is much faster).

# t-SNE projection: compress 100-dimensional word vectors to 2D
# and scatter-plot them to reveal semantic clusters visually.
from sklearn.manifold import TSNE
import matplotlib.pyplot as plt
# Get vectors for a subset of words
words = ['king', 'queen', 'man', 'woman', 'prince', 'princess',
    'cat', 'dog', 'paris', 'france', 'berlin', 'germany', 'tokyo', 'japan']
vectors = [wv[w] for w in words]
# Project to 2D with t-SNE
tsne = TSNE(n_components=2, random_state=42, perplexity=5)
vectors_2d = tsne.fit_transform(vectors)
# Plot
plt.figure(figsize=(10, 8))
for i, word in enumerate(words):
    plt.scatter(vectors_2d[i, 0], vectors_2d[i, 1])
    plt.annotate(word, (vectors_2d[i, 0]+0.5, vectors_2d[i, 1]+0.5), fontsize=12)
    plt.title("Word Embeddings Projected to 2D with t-SNE")
    plt.show()

import umap
reducer = umap.UMAP(n_components=2, random_state=42, n_neighbors=5)
vectors_2d = reducer.fit_transform(vectors)
# Plot with the same matplotlib code as above

Generalized Embeddings: Beyond Language

Word2Vec asks one question: given a token sequence, learn a vector per token that predicts neighbors. Once stated that way, nothing in the algorithm is intrinsically about language. Any setting that produces sequences of discrete items can be embedded with the exact same training loop. This is what makes the distributional hypothesis click for many readers: the company a token keeps may be linguistic, but it can equally well be musical, social, or transactional.

Two canonical non-NLP examples bring the point home:

• Graph embeddings via random walks. Treat nodes as tokens and random walks over the graph as sentences. Train Word2Vec on the walks and the resulting node vectors place graph-neighbors and structurally-similar nodes close together. This is the core idea behind DeepWalk and node2vec, used in recommendation, fraud detection, and protein-interaction analysis.
• Song embeddings from playlists. Treat each song's ID as a token and each user's playlist as a sentence. Train Word2Vec on millions of playlists. The learned vector for a song captures which other songs people group it with, with no audio features, no genre labels, and no metadata at all.

# Songs-as-tokens, playlists-as-sentences. Word2Vec does the rest.
from gensim.models import Word2Vec
# playlists: List[List[str]], each inner list is a playlist of song IDs
model = Word2Vec(
    sentences=playlists,
    vector_size=100,
    window=10,
    min_count=5,
    sg=1,
)
model.save("songs.w2v")
model.wv.most_similar("song_id_12345", topn=10)
# Returns the 10 songs whose playlist-company most resembles the seed track.

Exercises

from torch import nn
import torch
vocab = build_vocab(corpus); V = len(vocab)
emb = nn.Embedding(V, 100); ctx = nn.Embedding(V, 100)
opt = torch.optim.Adam(list(emb.parameters()) + list(ctx.parameters()), lr=1e-3)
for epoch in range(N):
    for center, context in pairs:
        negs = sample_negatives(5, V)
        e = emb(center); c = ctx(context); n = ctx(negs)
        pos_loss = -torch.log(torch.sigmoid((e * c).sum()))
        neg_loss = -torch.log(torch.sigmoid(-(e.unsqueeze(0) * n).sum(-1))).sum()
        loss = pos_loss + neg_loss
        opt.zero_grad(); loss.backward(); opt.step()