Contextual Embeddings: ELMo & the Path to Transformers

A word's meaning depends on context, they said. So does my performance review, but nobody built ELMo for that.

Lexica, Context-Dependent AI Agent

The Polysemy Problem

Word2Vec, GloVe, and FastText share a fundamental limitation: each word gets exactly one vector, regardless of context. But language is deeply contextual.

Consider the word "bank":

• "I deposited money at the bank." → financial institution
• "We sat on the river bank." → edge of a river
• "Don't bank on it." → to rely on

With Word2Vec, all three uses map to the same vector: a compromise that captures none of the meanings well. This is called the polysemy problem.

Figure 1.4.2: Static vs. contextual embeddings. Word2Vec assigns one fixed vector per word regardless of context, while contextual models produce different vectors for each usage.

The polysemy problem is not merely an engineering annoyance; it is the central challenge that motivated the next generation of embedding models. The solution came from a conceptual leap: instead of assigning one vector per word type (the dictionary entry), assign a different vector for every word token (each occurrence in context).

ELMo: Embeddings from Language Models (2018)

ELMo (Peters et al., 2018) was published under a name that hints at the playful culture of NLP research: it stands for Embeddings from Language Models, but the acronym was chosen to reference the Sesame Street character. Its successor, BERT, kept the tradition alive.

ELMo (Peters et al., 2018) was the first widely successful contextual embedding model. The key idea: run the entire sentence through a deep bidirectional LSTM, and use the hidden states as word representations. Since the LSTM has seen the whole sentence, each word's representation is influenced by its context.

How it works:

1. Train a bidirectional language model (forward LSTM + backward LSTM) on a large corpus
2. For each word in a sentence, extract hidden states from all layers
3. The ELMo embedding is a learned weighted combination of all layer representations

Figure 1.4.3: The ELMo computation graph. Input tokens pass through a character embedding layer, then through forward and backward LSTMs. The final ELMo vector for each token is a learned weighted sum across all layers.

The breakthrough: "bank" in "river bank" now gets a different vector than "bank" in "bank account", because the LSTM hidden states are conditioned on the entire sentence.

Why Different Layers Capture Different Information

A remarkable finding from the ELMo paper: different layers of the LSTM capture different types of linguistic information:

• Layer 0 (token embeddings): Captures basic word identity and morphological features, similar to Word2Vec. "Running" is close to "runner" and "runs."
• Layer 1 (first LSTM): Captures syntactic information: part of speech, grammatical role. The model has learned whether a word is a noun, verb, or adjective from context.
• Layer 2 (second LSTM): Captures semantic information: word sense disambiguation, topical context. This is the layer that distinguishes "river bank" from "bank account."

Figure 1.4.5: The information hierarchy across ELMo layers. Lower layers capture morphology and word identity, middle layers encode syntax, and upper layers represent semantics and word sense.

This is why ELMo uses a weighted combination of all layers (the α weights in the diagram above): different downstream tasks benefit from different layers. A POS (part-of-speech) tagger might weight Layer 1 heavily, while a sentiment classifier might rely more on Layer 2. The weights are learned during fine-tuning for each specific task.

Contextual Embeddings in Code

While ELMo itself is rarely used directly today (BERT and transformers have superseded it), we can demonstrate the concept of contextual embeddings using Hugging Face Transformers, which makes it easy to extract hidden states from any model:

# Demonstrating contextual embeddings: same word, different vectors
from transformers import AutoTokenizer, AutoModel
import torch
from scipy.spatial.distance import cosine
# Load a BERT model (the modern successor to ELMo)
# NOTE: First run will download bert-base-uncased (~420 MB).
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModel.from_pretrained("bert-base-uncased")
def get_word_embedding(sentence, word):
    """Extract the contextual embedding for a specific word in a sentence."""
    inputs = tokenizer(sentence, return_tensors="pt")
    with torch.no_grad():
        outputs = model(**inputs)
        # Find the token position for our word
        # WARNING: This assumes the target word is a single token. If BERT's tokenizer
        # splits the word into subwords (e.g., "deposit" -> "dep" + "##osit"), this
        # lookup will fail. For production code, you would need to handle multi-token
        # words by averaging their subword embeddings.
        tokens = tokenizer.tokenize(sentence)
        word_idx = tokens.index(word) + 1 # +1 for [CLS] token
        # Return the hidden state at that position
        return outputs.last_hidden_state[0, word_idx].numpy()
        # "bank" in two different contexts
        bank_river = get_word_embedding("I sat by the river bank", "bank")
        bank_money = get_word_embedding("I went to the bank to deposit money", "bank")
        # Measure how different the two "bank" vectors are
        distance = cosine(bank_river, bank_money)
        print(f"Cosine distance between 'bank' in different contexts: {distance:.3f}")
        # Output: ~0.35: substantially different vectors for the same word!
        # Compare: "bank" (river) is closer to "shore" than to "bank" (money)
        shore = get_word_embedding("We walked along the shore", "shore")
        print(f"Distance bank(river) to shore: {cosine(bank_river, shore):.3f}")
        print(f"Distance bank(river) to bank(money): {cosine(bank_river, bank_money):.3f}")
        # bank(river) is CLOSER to "shore" than to bank(money)!
        # This is exactly what contextual embeddings solve.

from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = model.encode([
    "I sat by the river bank",
    "I went to the bank to deposit money",
    "We walked along the shore"
])
# embeddings.shape: (3, 384), ready for cosine similarity

From ELMo to Transformers: What Changed

Let us compare the approaches side by side to see the progression clearly:

The key limitation of ELMo was its reliance on LSTMs, which process text sequentially (one word at a time). This made training slow and limited the model's ability to capture very long-range dependencies. The Transformer architecture (Chapter 3) solves this by processing all words simultaneously using self-attention.

Summary: The Representation Journey

This module traced the evolution of how we represent text for machines. Each step solved a problem that the previous approach could not handle:

Step back, and the path from sparse to dense to contextual representations becomes a single arc. Figure 1.4.7 lines up the five milestones (BoW, TF-IDF, Word2Vec, ELMo, Transformers) so you can see at a glance which limitation each new representation was designed to fix.

Figure 1.4.7a: The evolution of text representation. Each step solved a specific limitation: BoW lost word order, TF-IDF added importance weighting, Word2Vec captured semantics, ELMo added context, and Transformers enabled parallel contextual processing.