Scaling Frontiers: What Comes Next

"We keep scaling until the electricity bill becomes the research contribution."

Frontier, Budget Busting AI Agent

1. The Data Wall

The Chinchilla scaling laws (Hoffmann et al., 2022), covered in Section 06.3, established that optimal training requires scaling data and compute proportionally. For a model with N parameters, you need roughly 20N tokens of training data to achieve compute-optimal performance. This creates a straightforward problem: where do those tokens come from?

How Much Data Exists?

Villalobos et al. (2024) at Epoch AI conducted the most systematic analysis of this question. Their estimates suggest that the total stock of publicly available, high-quality text on the internet is between 4 and 15 trillion tokens, depending on quality thresholds. Lower-quality text (scraped web pages without filtering) extends this to perhaps 30 to 50 trillion tokens, but with rapidly diminishing quality.

To put this in perspective: GPT-4 is estimated to have been trained on approximately 13 trillion tokens. Llama 3 (405B) was trained on 15 trillion tokens. If the Chinchilla-optimal data requirement for a 10-trillion-parameter model is 200 trillion tokens, we have a problem. The internet does not contain enough high-quality text to support the next order-of-magnitude increase in model size.

The data wall is not just about volume. It is also about diversity. As models absorb more of the available text, each additional token provides less novel information. The marginal value of the 15-trillionth token is far less than the marginal value of the 1-trillionth token, because it is increasingly likely to be a paraphrase of content already seen.

Responses to the Data Wall

Frontier labs are pursuing several strategies simultaneously:

• Multi-epoch training. Training on the same data multiple times. Early evidence suggested this was harmful, but recent work shows that with proper learning rate scheduling and data ordering, models can extract additional value from repeated passes (Muennighoff et al., 2023). The penalty for repetition is real but manageable for 2 to 4 epochs.
• Data curation and filtering. Rather than gathering more data, improve the quality of existing data. Techniques include perplexity filtering, deduplication, classifier-based quality scoring, and domain rebalancing. The FineWeb dataset (Penedo et al., 2024) demonstrated that aggressive quality filtering of Common Crawl can produce training data that matches or exceeds curated datasets at a fraction of the size.
• Synthetic data generation. Using existing models to generate training data for the next generation. This is covered in detail in Chapter 13, but the frontier implications deserve attention here.
• Multimodal data. Supplementing text with images, video, audio, and code. A single YouTube video contains both visual and auditory information that, when transcribed and aligned, provides a richer learning signal than text alone.
• Private and licensed data. Partnerships with publishers, news organizations, and data aggregators. These deals (e.g., OpenAI's agreements with the Associated Press and News Corp) are partly about data access and partly about legal risk mitigation.

2. Synthetic Data: Promise and Peril

The most controversial response to the data wall is synthetic data: using model-generated text to train the next generation of models. The promise is obvious. If a frontier model can generate high-quality text, that text can be used to train a larger model, creating a virtuous cycle that decouples scaling from the finite stock of human-generated data.

Where Synthetic Data Works

Synthetic data has proven effective in specific, constrained domains:

• Mathematics and code. Models can generate vast quantities of mathematical proofs and programming problems with verifiable solutions. The verification step (can the proof be checked? does the code pass tests?) provides a natural quality filter. This is the strategy behind approaches like AlphaProof and the mathematical reasoning improvements in recent models.
• Instruction tuning. The Evol-Instruct approach (Xu et al., 2023), covered in Section 13.2, generates increasingly complex instruction-response pairs by iteratively rewriting prompts. This has been widely adopted, with WizardLM and its successors demonstrating that synthetic instruction data can match or exceed human-written instructions for fine-tuning.
• Distillation. Using a larger model's outputs to train a smaller model, as covered in Chapter 16. This is technically synthetic data, and it works well because the teacher model provides a richer training signal (soft probabilities) than the original hard labels.

Model Collapse: The Danger

Shumailov et al. (2024), published in Nature, demonstrated a sobering phenomenon they termed "model collapse." When models are trained on data generated by previous model generations, the distribution of the training data narrows progressively. Rare events, tail phenomena, and minority perspectives are gradually lost. After several generations of recursive training, the model's output distribution converges to a narrow mode that no longer reflects the diversity of the original human-generated data.

The analogy is photocopying a photocopy: each generation loses fidelity, and the losses compound. The authors showed this effect across multiple model families and training setups, suggesting it is a fundamental property of iterative synthetic data generation rather than an artifact of a particular approach.

The practical implication is that synthetic data can supplement human data but probably cannot fully replace it. A training pipeline that relies primarily on synthetic data will need robust mechanisms to detect and counteract distribution narrowing, such as diversity metrics, tail-event preservation, and periodic infusion of human-generated data.

3. The Three Axes of Scaling

Historically, scaling meant one thing: bigger pretraining runs (more parameters, more data, more compute). But the frontier has diversified into three distinct scaling axes, each with different economics and different diminishing returns profiles.

Axis 1: Pretraining Compute

This is the traditional scaling axis. Train a larger model on more data. The cost grows roughly quadratically with model size (because both the number of parameters and the data requirement increase). Frontier training runs in 2025 are estimated to cost $100 million to $1 billion. The returns, as measured by benchmark improvements, are diminishing: each doubling of compute yields smaller absolute gains.

Axis 2: Data Quality and Curation

The Llama strategy (Touvron et al., 2023) demonstrated an alternative: train a smaller model for longer on higher-quality data. Llama 1 (65B) was trained on 1.4 trillion tokens, far exceeding the Chinchilla-optimal data allocation for its size. This "over-training" approach produces models that are less compute-efficient during training but more token-efficient during inference, because the resulting model is smaller and cheaper to serve.

This insight has reshaped the industry. Most practitioners serve models, not train them. A model that costs twice as much to train but is half the size at serving time is often the better economic choice. The data quality axis offers a way to improve performance without increasing model size, by investing in better data curation, filtering, and curriculum design.

Axis 3: Test-Time Compute

The most significant recent development is the emergence of test-time compute scaling, as discussed in the context of reasoning models in Section 07.4. Models like OpenAI's o1 and o3, DeepSeek-R1, and Anthropic's extended thinking demonstrate that spending more compute at inference time (through chain-of-thought reasoning, search, verification, and self-correction) can dramatically improve performance on hard tasks.

The economics of test-time scaling are fundamentally different from pretraining scaling. Pretraining is a one-time cost amortized over all future queries. Test-time compute is a per-query cost that scales linearly with usage. This means test-time scaling is most cost-effective for hard, high-value queries where the additional compute is justified by the value of a correct answer.

The interplay between pretraining and test-time compute creates interesting trade-offs. A smaller, well-trained model with generous test-time compute can sometimes match a much larger model with minimal test-time compute. The optimal allocation depends on the query distribution: if most queries are easy, invest in pretraining; if a few queries are very hard and high-value, invest in test-time compute.

The table below summarizes how these three scaling axes differ in their cost structure, return profiles, and ideal use cases.

Notice that pretraining compute and data quality are one-time investments, while test-time compute is an ongoing per-query expense. This asymmetry is central to the economic calculus of frontier model development.

4. Alternative Architectures

Beyond scaling existing transformers along these three axes, a separate question looms: is the transformer itself the right architecture? The transformer has dominated language modeling since 2017, but several alternative architectures are challenging its supremacy, particularly for long-context and efficiency-critical applications.

State Space Models (SSMs)

State space models, particularly the Mamba architecture (Gu and Dao, 2023), offer a fundamentally different approach to sequence modeling. Instead of the quadratic attention mechanism, SSMs process sequences through a linear recurrence that scales linearly with sequence length. This makes them dramatically more efficient for long sequences.

The key insight behind Mamba is the "selective state space" mechanism, which allows the model to selectively propagate or forget information based on the input content. This addresses the principal limitation of earlier SSMs, which processed all tokens identically regardless of content relevance.

As of early 2026, SSMs have demonstrated competitive performance with transformers on many benchmarks, with clear advantages in throughput for long sequences. However, they have not yet matched transformers on tasks that require complex in-context learning or precise information retrieval from context. Hybrid architectures that combine transformer attention layers with SSM layers are emerging as a pragmatic compromise, offering the efficiency of SSMs for most processing while retaining attention for tasks that require it.

Mixture of Experts (MoE) at Scale

Mixture of experts is not a new idea, but its application at frontier scale has been transformative. Models like Mixtral (Jiang et al., 2024), DeepSeek-V2, and reportedly GPT-4 use sparse MoE architectures where only a fraction of the model's parameters are activated for each token. This decouples total model capacity (total parameters) from per-token compute (active parameters).

The scaling implications are significant. A 1.8-trillion-parameter MoE model that activates only 100 billion parameters per token has the knowledge capacity of a dense 1.8T model with the inference cost of a dense 100B model. This allows MoE models to scale capacity without proportionally scaling inference cost, although training cost and memory requirements still scale with total parameters.

Open questions for MoE scaling include: expert load balancing at extreme scale, the optimal number and granularity of experts, routing stability during training, and whether MoE models exhibit different emergence patterns than dense models.

Diffusion Language Models

A more speculative direction is applying diffusion processes (successful in image generation) to language modeling. Unlike autoregressive models that generate tokens left-to-right, diffusion language models generate all tokens simultaneously through iterative refinement, starting from noise and progressively denoising.

Early results (e.g., MDLM by Sahoo et al., 2024) are promising but still lag behind autoregressive models on standard benchmarks. The potential advantage is in tasks that benefit from bidirectional generation, such as infilling, editing, and constrained generation. If diffusion language models can close the quality gap, they may offer qualitatively different capabilities from autoregressive models.

5. The Chinchilla Trap and the Llama Strategy

An important subtlety in the scaling debate is the distinction between compute-optimal training and inference-optimal training. The Chinchilla scaling laws tell you how to allocate a fixed compute budget to minimize training loss. But in practice, models are trained once and served millions (or billions) of times. The total cost of ownership is dominated by inference cost, not training cost.

This creates what we might call the "Chinchilla trap": if you follow the compute-optimal recipe, you end up with a large model that is expensive to serve. The Llama strategy (and its successors) deliberately violates the Chinchilla prescription by training a smaller model on more data than is compute-optimal. The resulting model has slightly higher training loss than a Chinchilla-optimal model trained with the same total compute, but it is much cheaper to serve because it has fewer parameters.

The Chinchilla trap illustrates a broader lesson: optimizing for one metric (training compute efficiency) can be counterproductive when the real objective (total cost of ownership) involves a different metric. This lesson recurs throughout the scaling frontier: the optimal strategy depends critically on what you are optimizing for.

6. Multimodal Scaling

A growing body of evidence suggests that training on multiple modalities (text, images, video, audio, code) improves performance on each individual modality compared to training on that modality alone. This is not merely because more data is available; it is because different modalities provide complementary learning signals.

For example, visual grounding helps language models understand spatial relationships. Code training improves logical reasoning. Audio training with transcripts provides a richer model of natural language than text alone. The "scaling" in multimodal scaling is not just about more tokens; it is about a richer information-per-token that accelerates learning.

The practical challenge is tokenization and alignment. How do you represent a video frame, a spectrogram, and a paragraph of text in a shared token space that allows the model to learn cross-modal relationships? The approaches surveyed in Chapter 27 (native multimodal encoders, adapter-based fusion, interleaved pretraining) are all active areas of scaling research.

Exercises