Multilingual & Cross-Cultural LLMs

A truly multilingual model must master the grammar of a thousand languages, the idioms of a hundred cultures, and the quiet indignity of being evaluated almost exclusively in English.

Bert, Polyglot AI Agent

1. Multilingual Pre-Training: How One Model Learns Many Languages

Modern multilingual LLMs are trained on corpora containing text in dozens to hundreds of languages. The model is never explicitly told which language it is processing; it must learn to handle all languages within a shared parameter space. This approach produces a remarkable phenomenon: cross-lingual transfer, where knowledge learned in one language becomes available in others.

1.1 Cross-Lingual Transfer

Cross-lingual transfer occurs because languages share deep structural similarities despite surface-level differences. When a model learns that "The cat sat on the mat" has a subject-verb-object structure in English, this structural knowledge partially transfers to French ("Le chat s'est assis sur le tapis") and even to languages with different word orders, because the underlying conceptual relationships are similar. Cross-lingual transfer also extends to the embedding space, where multilingual models cluster semantically similar concepts across languages (see Section 18.1 for embedding fundamentals).

The mechanism behind cross-lingual transfer operates at multiple levels:

• Shared vocabulary: Subword tokenizers (BPE, SentencePiece) create overlapping token sets across languages. Cognates, borrowed words, numbers, and shared scripts provide anchor points that align representations across languages.
• Structural alignment: Languages that share syntactic structures (e.g., SVO word order) develop more aligned internal representations. The model learns abstract "slots" for subjects, verbs, and objects that work across languages.
• Conceptual universals: Certain concepts (numbers, spatial relationships, causality) are expressed in all languages. Training on parallel or similar content in multiple languages forces the model to develop language-agnostic representations of these concepts.

Figure 7.4.2: Multilingual models map different languages into a shared representation space, enabling knowledge learned in one language to transfer to others.

1.2 The Curse of Multilinguality

Cross-lingual transfer is not free. Training a single model on many languages introduces a fundamental tension known as the curse of multilinguality: for a fixed model capacity, adding more languages improves low-resource language performance (through transfer from high-resource languages) but degrades high-resource language performance (because capacity is shared across more languages).

This trade-off can be expressed informally as:

The interference term grows with the number of languages and the dissimilarity between them. Languages with different scripts, typological features, or morphological systems create more interference than closely related languages. Research by Conneau et al. (2020) showed that for XLM-R, performance on English decreased measurably when the model was trained on 100 languages versus 10, despite the English data remaining constant.

2. Low-Resource Language Challenges

Of the roughly 7,000 languages spoken worldwide, the vast majority are considered "low-resource" in the context of NLP. A language is low-resource when there is insufficient digital text data to train a capable model. The distribution is extremely skewed: English alone accounts for roughly 50% of internet content, and the top 10 languages cover over 80%. Thousands of languages have virtually no digital presence.

2.1 The Data Scarcity Problem

Low-resource languages face several compounding challenges: Code Fragment 7.4.1 below puts this into practice.

• Limited training data: Some languages have fewer than 10,000 sentences available in any digital form, compared to trillions of tokens for English
• Tokenization inefficiency: Tokenizers trained primarily on English and other Latin-script languages produce highly inefficient tokenizations for languages with different scripts. A single word in Thai, Tibetan, or Khmer may require 5 to 10 tokens, while the equivalent English word uses 1 to 2 tokens. This means the model's effective context window is much smaller for these languages.
• Evaluation gaps: Benchmarks for low-resource languages are scarce or nonexistent, making it difficult to measure progress or identify problems
• Dialect and variety: Many low-resource languages have significant dialectal variation, and the available data may not represent the variety spoken by the target users

# Demonstrating tokenization inefficiency across languages
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B")

# The same concept in different languages
sentences = {
 "English": "The weather is beautiful today.",
 "French": "Le temps est magnifique aujourd'hui.",
 "Chinese": "今天天气很好。",
 "Thai": "วันนี้อากาศดีมาก",
 "Amharic": "ዛሬ የአየር ሁኔታ ቆንጆ ነው።",
 "Khmer": "អាកាសធាតុល្អនៅថ្ងៃនេះ។",
}

print(f"{'Language':<12} {'Tokens':>6} {'Chars/Token':>11}")
print("-" * 35)
for lang, text in sentences.items():
 tokens = tokenizer.encode(text)
 ratio = len(text) / len(tokens)
 print(f"{lang:<12} {len(tokens):>6} {ratio:>11.2f}")

# Typical output:
# Language Tokens Chars/Token
# -----------------------------------
# English 7 4.71
# French 9 4.11
# Chinese 10 0.70
# Thai 23 0.78
# Amharic 33 0.52
# Khmer 43 0.47

2.2 Solutions for Low-Resource Languages

Several strategies have been developed to improve LLM performance on low-resource languages:

• Cross-lingual transfer: Leverage the model's knowledge from high-resource languages. If the model understands sentiment analysis in English, it can partially transfer this capability to Swahili through shared representations.
• Data augmentation: Use machine translation, back-translation, or LLM-generated synthetic data to expand the training set for low-resource languages
• Few-shot prompting: Provide a few examples of the target task in the low-resource language, leveraging the model's in-context learning ability
• Community-driven data collection: Projects like Masakhane (for African languages) and AmericasNLP (for indigenous American languages) coordinate community efforts to create datasets and benchmarks

3. Cultural Bias in LLMs

Language models inherit the cultural perspectives embedded in their training data. Since most training data originates from English-language internet sources, these models tend to encode Western (and specifically American) cultural norms, values, and assumptions as defaults. This manifests in several ways.

3.1 Western-Centric Defaults

When asked culturally dependent questions, LLMs overwhelmingly default to Western contexts: Code Fragment 7.4.2 below puts this into practice.

• Geography and politics: "What is the capital?" assumes the user means the United States. "The president" defaults to the U.S. president.
• Cultural norms: Advice about family, relationships, business etiquette, or social situations reflects Western individualist values, which may be inappropriate or offensive in collectivist cultures.
• Historical perspective: Historical events are presented from Western viewpoints. The same event may be described very differently in different cultural traditions.
• Religious and philosophical assumptions: Concepts like "morality," "fairness," and "justice" carry culturally specific meanings that models tend to resolve toward Western liberal interpretations.

# Demonstrating cultural bias in LLM responses
# (Illustrative; actual responses vary by model)

prompts_and_bias = [
 {
 "prompt": "What should I bring to a dinner party?",
 "typical_response": "Wine or dessert",
 "cultural_note": "In many Asian cultures, bringing fruit or "
 "specialty items from your region is preferred. "
 "In some Middle Eastern cultures, the host may "
 "be offended by gifts implying they cannot "
 "provide enough."
 },
 {
 "prompt": "How should I greet my colleague's parents?",
 "typical_response": "A firm handshake and eye contact",
 "cultural_note": "In Japan, a bow is appropriate. In many South "
 "Asian cultures, touching elders' feet is a sign "
 "of respect. In some Middle Eastern cultures, "
 "cross-gender handshakes may be inappropriate."
 },
 {
 "prompt": "What is a healthy breakfast?",
 "typical_response": "Oatmeal, eggs, or yogurt with fruit",
 "cultural_note": "In Japan: miso soup, rice, grilled fish. "
 "In India: idli, dosa, upma. "
 "In Mexico: chilaquiles, huevos rancheros. "
 "The concept of 'healthy' itself varies by culture."
 },
]

# A culturally aware system should detect the user's context
# and adapt responses accordingly, or explicitly acknowledge
# that the answer is culturally dependent.

3.2 Measuring Cultural Bias

Researchers have developed several approaches to quantify cultural bias in LLMs:

• World Values Survey alignment: Compare LLM responses on value-laden questions to responses from different countries in the World Values Survey. Models consistently align most closely with responses from the United States and Western Europe.
• Cultural probing: Ask models about culturally specific practices, holidays, food, and customs across different cultures. Measure the depth and accuracy of knowledge as a function of the culture's representation in training data.
• Stereotype evaluation: Test whether models associate nationalities, ethnicities, or religions with specific (often stereotypical) attributes. Benchmarks like BBQ (Bias Benchmark for QA) and CrowS-Pairs provide standardized tests.

4. Multilingual Evaluation Benchmarks

Evaluating multilingual LLMs requires benchmarks that span multiple languages and tasks. Several important benchmarks have emerged:

Figure 7.4.3: Illustrative performance gap on multilingual QA benchmarks. Low-resource languages can trail English by 40+ percentage points on the same task.

5. Adapting English-Centric Models to New Languages

Given the dominance of English in LLM training data, a practical question arises: how can we adapt an existing English-centric model to serve a new target language well? Several techniques have proven effective.

5.1 Vocabulary Extension

The first bottleneck is often the tokenizer. A tokenizer trained primarily on English will fragment text in other scripts into many small, semantically meaningless tokens. Vocabulary extension adds new tokens specific to the target language: Code Fragment 7.4.3 below puts this into practice.

# Vocabulary extension for a new language
from transformers import AutoTokenizer, AutoModelForCausalLM
from tokenizers import trainers, Tokenizer, models
import torch

def extend_tokenizer(base_model_name, target_corpus_path, n_new_tokens=10000):
 """
 Extend a model's tokenizer with tokens from a new language.
 Steps:
 1. Train a new tokenizer on the target language corpus
 2. Find tokens in the new tokenizer that are not in the base
 3. Add the most frequent new tokens to the base tokenizer
 4. Resize the model's embedding matrix
 """
 base_tokenizer = AutoTokenizer.from_pretrained(base_model_name)
 base_vocab = set(base_tokenizer.get_vocab().keys())

 # Train a tokenizer on the target language
 target_tokenizer = Tokenizer(models.BPE())
 trainer = trainers.BpeTrainer(
 vocab_size=32000,
 special_tokens=["[UNK]", "[PAD]"],
 )
 target_tokenizer.train([target_corpus_path], trainer)
 target_vocab = target_tokenizer.get_vocab()

 # Find new tokens not in the base vocabulary
 new_tokens = [
 tok for tok in target_vocab
 if tok not in base_vocab
 ]

 # Sort by frequency and take top N
 new_tokens = sorted(
 new_tokens,
 key=lambda t: target_vocab[t],
 reverse=True,
 )[:n_new_tokens]

 # Add to base tokenizer
 base_tokenizer.add_tokens(new_tokens)

 # Resize model embeddings
 model = AutoModelForCausalLM.from_pretrained(base_model_name)
 model.resize_token_embeddings(len(base_tokenizer))

 # Initialize new embeddings as average of existing ones
 with torch.no_grad():
 avg_embedding = model.get_input_embeddings().weight[
 :len(base_vocab)
 ].mean(dim=0)
 for i in range(len(base_vocab), len(base_tokenizer)):
 model.get_input_embeddings().weight[i] = avg_embedding

 return model, base_tokenizer

# After extension, fine-tune on target language data

5.2 Continued Pre-Training

After extending the vocabulary, the model needs exposure to text in the target language. Continued pre-training (also called language-adaptive pre-training) trains the model on a large corpus of target language text using the same next-token prediction objective as the original pre-training. This approach is simpler than training from scratch and leverages the model's existing knowledge.

Key considerations for continued pre-training include:

• Data mixing: Include some English data alongside the target language to prevent catastrophic forgetting of English capabilities. A common ratio is 70% target language, 30% English.
• Learning rate: Use a smaller learning rate than original pre-training (typically 1/10 to 1/5 of the original peak, e.g. 2e-5) to preserve existing knowledge while learning new language patterns.
• Data volume: Even a few billion tokens of target language data can produce significant improvements, especially when combined with vocabulary extension.
• Regularization: Standard weight decay (0.1) and gradient clipping (max norm 1.0) help prevent catastrophic forgetting during adaptation.
• Precision: Use bf16 mixed precision for training efficiency, consistent with modern pre-training practice.

5.3 Cross-Lingual Instruction Tuning

After language-adaptive pre-training, the model understands the target language but may not follow instructions well in it. Cross-lingual instruction tuning fine-tunes the model on instruction-following examples in the target language. These can be:

• Translated: Machine-translate existing English instruction datasets (cheap but may introduce translation artifacts)
• Native: Collect instruction-response pairs written natively in the target language (expensive but higher quality)
• Mixed: Combine translated and native examples, with native examples for culturally sensitive topics

6. Multilingual Model Families

Several model families have been designed with multilingual capability as a core goal rather than an afterthought:

pip install torch

import torch
import torch.nn.functional as F
import math

def scaled_dot_product_attention(Q, K, V, mask=None):
 """
 Q, K, V: (batch, heads, seq_len, d_k)
 mask: (seq_len, seq_len) boolean, True = positions to mask out
 Returns: (batch, heads, seq_len, d_k), attention weights
 """
 d_k = Q.size(-1)
 scores = torch.matmul(Q, K.transpose(-2, -1)) / math.sqrt(d_k)
 if mask is not None:
 scores = scores.masked_fill(mask, float("-inf"))
 weights = F.softmax(scores, dim=-1)
 output = torch.matmul(weights, V)
 return output, weights

# Quick test
batch, heads, seq_len, d_k = 2, 4, 8, 16
Q = torch.randn(batch, heads, seq_len, d_k)
K = torch.randn(batch, heads, seq_len, d_k)
V = torch.randn(batch, heads, seq_len, d_k)

# Causal mask: prevent attending to future positions
causal = torch.triu(torch.ones(seq_len, seq_len, dtype=torch.bool), diagonal=1)
out, attn = scaled_dot_product_attention(Q, K, V, mask=causal)
print(f"Output shape: {out.shape}") # (2, 4, 8, 16)
print(f"Attention shape: {attn.shape}") # (2, 4, 8, 8)
print(f"Row sums (should be 1.0): {attn[0, 0].sum(dim=-1)}")

import torch.nn as nn

class ManualMultiHeadAttention(nn.Module):
 def __init__(self, d_model, num_heads):
 super().__init__()
 assert d_model % num_heads == 0
 self.d_model = d_model
 self.num_heads = num_heads
 self.d_k = d_model // num_heads

 self.W_q = nn.Linear(d_model, d_model, bias=False)
 self.W_k = nn.Linear(d_model, d_model, bias=False)
 self.W_v = nn.Linear(d_model, d_model, bias=False)
 self.W_o = nn.Linear(d_model, d_model, bias=False)

 def forward(self, x, mask=None):
 batch, seq_len, _ = x.shape
 # Project and reshape to (batch, heads, seq_len, d_k)
 Q = self.W_q(x).view(batch, seq_len, self.num_heads, self.d_k).transpose(1, 2)
 K = self.W_k(x).view(batch, seq_len, self.num_heads, self.d_k).transpose(1, 2)
 V = self.W_v(x).view(batch, seq_len, self.num_heads, self.d_k).transpose(1, 2)

 # Apply attention from Step 1
 attn_out, weights = scaled_dot_product_attention(Q, K, V, mask=mask)

 # Concatenate heads and project
 concat = attn_out.transpose(1, 2).contiguous().view(batch, seq_len, self.d_model)
 output = self.W_o(concat)
 return output, weights

# Test
d_model, num_heads = 64, 4
mha = ManualMultiHeadAttention(d_model, num_heads)
x = torch.randn(2, 10, d_model)
out, weights = mha(x)
print(f"MHA output: {out.shape}") # (2, 10, 64)
print(f"Attn weights: {weights.shape}") # (2, 4, 10, 10)

# The library way: 3 lines
builtin_mha = nn.MultiheadAttention(d_model, num_heads, batch_first=True, bias=False)
builtin_out, builtin_weights = builtin_mha(x, x, x)
print(f"Built-in output: {builtin_out.shape}") # (2, 10, 64)

# Copy weights from manual to built-in to verify numerical match
with torch.no_grad():
 # PyTorch packs Q, K, V projections into one in_proj_weight
 builtin_mha.in_proj_weight.copy_(
 torch.cat([mha.W_q.weight, mha.W_k.weight, mha.W_v.weight], dim=0)
 )
 builtin_mha.out_proj.weight.copy_(mha.W_o.weight)

builtin_out2, _ = builtin_mha(x, x, x)
manual_out, _ = mha(x)
diff = (builtin_out2 - manual_out).abs().max().item()
print(f"Max absolute difference: {diff:.2e}") # Should be ~1e-6 or smaller