Multilingual Evaluation, Adaptation & Model Families

The first 80 percent of a multilingual model comes from data. The last 20 percent, the part that decides whether a Burmese teenager actually trusts your chatbot, comes from caring enough to evaluate it in Burmese.

Bert, Polyglot AI Agent

7.4.4 Multilingual Evaluation Benchmarks

Evaluating multilingual LLMs requires benchmarks that span multiple languages and tasks. Several important benchmarks have emerged:

Figure 7.4.4b: Illustrative performance gap on multilingual QA benchmarks. Low-resource languages can trail English by 40+ percentage points on the same task.

7.4.5 Adapting English-Centric Models to New Languages

Given the dominance of English in LLM training data, a practical question arises: how can we adapt an existing English-centric model to serve a new target language well? Several techniques have proven effective.

7.4.5.1 Vocabulary Extension

The first bottleneck is often the tokenizer. A tokenizer trained primarily on English will fragment text in other scripts into many small, semantically meaningless tokens. Vocabulary extension adds new tokens specific to the target language:

# Vocabulary extension for a new language
from transformers import AutoTokenizer, AutoModelForCausalLM
from tokenizers import trainers, Tokenizer, models
# Load base English-centric model
base_tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B")
base_model = AutoModelForCausalLM.from_pretrained("meta-llama/Meta-Llama-3-8B")
# Train a new tokenizer on target language data (e.g., Hindi)
# target_corpus = load_dataset(...) # large Hindi text corpus
# new_vocab = train_new_vocab(target_corpus, vocab_size=20000)
# Identify tokens in new vocab not in base vocab
# to_add = [t for t in new_vocab if t not in base_tokenizer.vocab]
# base_tokenizer.add_tokens(to_add)
# Resize model embeddings to accommodate new tokens
# base_model.resize_token_embeddings(len(base_tokenizer))
# Initialize new embeddings (e.g., as mean of related existing embeddings)
# or with small random values

7.4.5.2 Continued Pretraining

After vocabulary extension, the model needs exposure to text in the target language. Continued pretraining (also called domain-adaptive pretraining or language-adaptive pretraining) trains the model on a large corpus of target language text, with several considerations:

• Data mixing: Mix target language data (e.g., 70%) with English data (e.g., 30%) to prevent catastrophic forgetting of English capabilities
• Learning rate: Use a smaller learning rate than initial pretraining (typically 10x to 100x smaller) to avoid disrupting existing knowledge
• Duration: Continued pretraining typically lasts for 1B to 100B tokens, depending on the target language's distance from the base model's training data
• Evaluation: Monitor both target language performance and English performance throughout training to detect catastrophic forgetting early

7.4.5.3 Cross-Lingual Instruction Tuning

After language-adaptive pretraining, the model understands the target language but may not follow instructions well in it. Cross-lingual instruction tuning fine-tunes the model on instruction-following examples in the target language. These can be:

• Translated: Machine-translate existing English instruction datasets (cheap but may introduce translation artifacts)
• Native: Collect instruction-response pairs written natively in the target language (expensive but higher quality)
• Mixed: Combine translated and native examples, with native examples for culturally sensitive topics

7.4.6 Multilingual Model Families

Several model families have been designed with multilingual capability as a core goal rather than an afterthought:

Exercises

from langdetect import detect
specialists = {"hi": "sarvam-2b", "ja": "rakuten-7b", "ar": "jais-13b"}
def route(msg):
  try: lang = detect(msg)
  except: lang = "en"
  return call_model(specialists.get(lang, "gpt-4o"), msg)