Production Training Infrastructure

"Sixteen thousand GPUs, a seven-hour MTBF, and one heroic checkpoint script standing between you and a wasted week. Save early, save sharded, save async."

Sched, 3 a.m. Checkpoint Resurrection AI Agent

59.5.1 Mean Time Between Failures

The numerical reality of large-scale training: at any non-trivial cluster size, hardware failures are continuous, not exceptional. Empirical per-node failure rates from public post-mortems:

The Llama-3 paper reports 419 hardware interruptions in 54 days on 16,000 H100s, or roughly one node failure every 3 hours of wall-clock time across the cluster. The dominant causes were GPU hardware failures (58%), GPU HBM failures (17%), network switch failures (9%), and host system issues (16%). All of these are time-correlated to silicon temperature, which is why Meta's cluster monitoring includes a per-rack thermal model.

59.5.2 Checkpointing Strategies

A checkpoint is the only way to bound the cost of a failure. The state of the art is sharded asynchronous checkpoints written every few minutes to a parallel file system.

59.5.2.1 What Goes Into a Checkpoint

A complete checkpoint must capture everything needed to bit-identically resume training: model parameters, optimizer state (master weights, $m$, $v$, scheduler step), data-loader state (current shard, sample index, RNG seed), and the random number generator state of every rank. Forgetting any one of these creates subtle divergence after restart: the most common bug is forgetting the data-loader's epoch index, so that on restart you re-train on the same first batch and overshoot or stall the loss curve.

59.5.2.2 Synchronous vs Asynchronous

The simplest checkpoint is synchronous: every rank gathers its shard of state, writes to disk, and waits until done before resuming training. On a 70B model with 100 GB of state per rank, writing to a parallel file system at 5 GB/s takes 20 seconds per rank. Even with 1000 parallel writers, the synchronous step blocks training for 20-30 seconds. At a 10-minute cadence, that is 4% overhead.

Asynchronous checkpointing decouples the write from the training step. The state is first copied to a CPU staging buffer (fast, $\sim$1 second using GPUDirect-Storage or pinned host memory), then a background process writes to disk while training continues. Modern frameworks (PyTorch's torch.distributed.checkpoint.async_save, Megatron's save_checkpoint_async) implement this. The training-step overhead drops from 20 seconds to roughly 1.

59.5.2.3 Sharded Checkpoint Format

With FSDP / TP / PP, each rank holds a different shard of the parameters. The naive "rank 0 collects everything and writes one file" approach does not scale; rank 0 becomes a memory bottleneck and a single-point-of-failure for the checkpoint. The production pattern is the sharded checkpoint: each rank writes its own shard file independently, with a metadata file describing the global tensor layout. PyTorch's DTensor and distributed.checkpoint APIs implement this: a 70B model's checkpoint is ~1000 files (one per rank), each ~100 MB, written in parallel.

A nice property: a sharded checkpoint can be resharded on load. A run trained with $TP=8, PP=8, DP=16$ can resume on a cluster with $TP=4, PP=16, DP=8$ as long as the total tensor shapes match. This flexibility is essential when clusters are reconfigured between training phases (e.g., switching from pretraining to fine-tuning).

59.5.2.4 Checkpoint Cadence

How often to checkpoint? If failures arrive at rate $1/\tau$ and a checkpoint takes time $C$, then expected wasted work per failure is $(T_{\text{interval}} + C)/2$, and the per-step overhead from writing checkpoints is $C / T_{\text{interval}}$. Total cost as a fraction of compute:

Minimizing, the optimal interval is $T^{\star}_{\text{interval}} = \sqrt{2C\tau}$. For $C=20$ seconds and $\tau=7$ hours $\approx 25{,}000$ seconds, the optimal interval is $\sqrt{2 \cdot 20 \cdot 25000} = 1000$ seconds, about 17 minutes. The actual frontier-run cadence is typically 5-15 minutes; sub-5 minutes starts to compete with the optimizer step for I/O bandwidth.

59.5.3 Fault-Tolerant Schedulers and Elastic Agents

When a node dies, three things must happen, in order, in a few minutes:

1. Detect the failure. NCCL hangs (it does not abort cleanly), so the supervisor needs a timeout-based watchdog. Each rank's process should periodically report a heartbeat to a coordinator; missing heartbeats triggers the recovery path.
2. Provision a replacement. The cluster scheduler (Kubernetes, Slurm, AWS Batch) must have a hot pool of standby nodes or be able to provision one quickly. Spot-instance training (PyTorch + AWS Spot) makes this routine.
3. Restart with the new node. The whole world must re-form the NCCL communicator with the new rank assignment, load the latest checkpoint, and resume.

59.5.3.1 PyTorch Elastic (TorchElastic)

PyTorch's torchrun with --max-restarts=N and --rdzv-backend=c10d implements (1) and (3) at the framework level. The rdzv (rendezvous) backend coordinates which ranks are alive; if a rank dies, the rendezvous re-forms and the surviving ranks restart from the latest checkpoint. The world-size can shrink (continue with fewer ranks) or stay the same (a replacement joins). This is the lowest-level fault-tolerance primitive in the PyTorch ecosystem; everything else builds on it.

59.5.3.2 Cluster Scheduler Integration

The job scheduler (the layer above the framework) handles provisioning:

• SkyPilot (Yang et al., 2023): a cloud-agnostic abstraction that runs training jobs on any cloud's spot capacity, automatically requesting a replacement when a spot instance is reclaimed. Most academic labs use SkyPilot in 2026.
• Kubeflow Training Operator: the Kubernetes-native solution. The PyTorchJob CRD includes elasticity (elasticPolicy.minReplicas) and works with the cluster autoscaler.
• AWS Batch / SageMaker Training Compiler: AWS-native, with hot-standby spot capacity. Common at companies running on AWS H100 / Trainium.
• MosaicML Composer (now Databricks): a Python framework that wraps PyTorch with a training callback model and integrates with the MosaicML cloud's elastic provisioning. Notable for being one of the first to operationalize spot-aware training.

59.5.3.3 Spot and Preemptible Training

The economic story: AWS spot instances are 50-70% cheaper than on-demand. If your training run can survive arbitrary node preemption (which it can, with the machinery above), you cut your training cost by half. This is now standard. The mathematics: with 7-hour MTBF baseline failures and spot preemption adding another, say, 4-hour preemption rate, you double the failure rate. As long as checkpoint recovery costs less than the on-demand premium, spot wins.

59.5.4 Observability: MFU and the NCCL Flame Graph

You cannot fix what you cannot see. Production training surfaces three classes of signal: throughput (am I running at full speed?), quality (is the loss curve healthy?), and system health (is anything degrading?).

59.5.4.1 Model FLOPs Utilization (MFU)

The headline throughput metric is MFU (Chowdhery et al., PaLM 2022): the ratio of useful FLOPs your model computed per second to the theoretical peak FLOPs of your hardware. For a transformer with $P$ parameters processing $T$ tokens with sequence length $L$:

and MFU is:

Modern frontier training achieves 35-55% MFU on H100s in BF16. The Llama-3 paper reports 38-43% MFU at 405B scale; Megatron's published benchmarks reach 50-55% on smaller models. The remaining headroom is communication overlap and kernel efficiency; the Hopper architecture's full peak is unreachable in dense BF16 because of memory bandwidth bottlenecks. FP8 training (DeepSeek-V3, Llama-4) lifts the ceiling.

59.5.4.2 Per-Rank Loss Tracking

The training loss is computed locally on each rank and aggregated for logging. Frontier teams log per-rank loss separately as well, because a per-rank loss spike on a single rank is the smoking gun for a corrupted data shard, a numerically unstable kernel on one GPU, or a degraded NCCL link. The "log only the average" pattern hides exactly the signal you need.

59.5.4.3 Gradient Norms and Spike Detection

The global gradient norm $\| g \|_2$ is logged every step. A healthy run has a slowly decreasing norm; spikes correlate with bad batches (often noisy or duplicated data) and predict loss divergences. The standard recipe: clip gradients at $\max(1.0, \mu_g + 4\sigma_g)$ where $\mu_g, \sigma_g$ are running estimates of the gradient norm. Skip the optimizer step if the norm exceeds, say, $20 \cdot \mu_g$; this is the "skip step on spike" practice that turned around several public 100B+ training runs in 2023.

59.5.4.4 NCCL Flame Graphs

When a step suddenly slows from 1.2 s to 1.8 s, the culprit is almost always NCCL. NCCL exposes per-collective profiling via the NCCL_DEBUG=INFO environment variable and the nvtx Python module; PyTorch's profiler aggregates these into a flame graph. The pathological patterns to look for:

• One slow rank. Every rank waits on the slowest in a collective. If rank 47's all-gather takes 50 ms while every other rank takes 5 ms, GPU 47 is overheating, has a degraded NVLink, or is hosting a parasitic process. Pull it from the cluster.
• Buffer fragmentation. After hours of training, the CUDA caching allocator fragments and NCCL can no longer allocate its preferred contiguous buffer, falling back to bounce-buffered transfers. A periodic torch.cuda.empty_cache() at known synchronization points fixes this.
• Stragglers from network contention. On rail-suboptimal topologies, two large all-reduces from different runs (or different training groups) can collide on the same switch, dropping each to half throughput. The solution is rail-aligned scheduling, which most modern clusters now enforce.

59.5.5 The Tooling Stack

The 2026 production stack converged on a small set of well-supported tools:

59.5.5.1 Weights & Biases and MLflow at Scale

For a 1000-rank job, the simplest mistake is to have every rank push metrics to W&B independently; you get a 1000x rate-limit issue and ambiguous "which rank are these numbers from?" plots. The pattern is to have rank 0 alone publish metrics, with per-rank logs aggregated locally and pushed in batches. W&B's wandb.init(group=run_id, job_type=f"rank-{rank}") pattern lets you keep per-rank data for debugging while keeping the main run cleanly scalar. MLflow's distributed tracking server pattern is similar.

pip install wandb
import os, wandb, torch.distributed as dist

rank = int(os.environ["RANK"])
run = wandb.init(
    project="llama-pretrain",
    group=os.environ["RUN_ID"],          # one group per training run
    job_type=f"rank-{rank}",
    mode="online" if rank == 0 else "disabled",  # only rank 0 publishes
)
if rank == 0:
    wandb.define_metric("loss", summary="min")
    wandb.define_metric("mfu", summary="mean")

for step, batch in enumerate(loader):
    loss = train_step(batch)
    if rank == 0 and step % 10 == 0:
        wandb.log({
            "loss": loss.item(),
            "lr": scheduler.get_last_lr()[0],
            "tokens_per_sec": tokens_seen / elapsed,
            "mfu": measured_flops / peak_flops,
        }, step=step)

59.5.5.2 Throughput Dashboards

The single most useful real-time dashboard for a frontier run is a stacked plot of: tokens/s (throughput), MFU, per-rank loss min/max/median, gradient norm, and an indicator of whether NCCL is exhibiting buffer-fragmentation slowdowns. A drop in any of these is the leading signal for a problem; the on-call engineer's job is to map the drop to the root cause within 15 minutes. Most frontier teams have a runbook of "the loss spiked, here are the 10 things to check"; the runbook gets longer with every training run.

59.5.6 Real-World Post-Mortems

The most informative training-stack lessons come from public post-mortems. Three worth knowing in 2026:

59.5.6.1 OPT-175B: 35+ Loss Divergences, 2022

Meta's OPT-175B (Zhang et al., 2022) was the first public training-stack post-mortem at frontier scale. The team published its training logbook, which documented 35 distinct loss divergences across the run, each requiring a manual rewind to a checkpoint and a learning-rate cut. Root causes included: (a) a fp16 numerical overflow in the activation gradient of one attention head; (b) several incidents of data shard duplication that drove the loss into a tight loop on a single example; (c) a bad batch where one document had been incorrectly tokenized at training time. The fixes that emerged became standard practice: skip-step-on-spike, per-rank loss logging, document-level deduplication.

59.5.6.2 BLOOM-176B: NVLink Saturation on Jean Zay

The BLOOM training (BigScience, 2022) discovered mid-run that the Jean Zay supercomputer's NVLink topology had only 4-GPU fanout rather than the assumed 8. The original plan of $TP=8$ was infeasible; the team rolled out $TP=4, PP=12$ instead, which preserved the per-GPU memory budget but introduced a larger pipeline bubble. The cost was a roughly 15% throughput hit; the lesson is to validate the interconnect topology against the parallelism plan before the production run starts, not after.

59.5.6.3 Llama-3 405B: Thermal-Correlated Failures, 2024

Meta's Llama-3 paper reports that GPU failure rate tripled during the hot afternoon hours of the cluster's geographic location, correlating tightly with rack-level inlet air temperature. The fix was to add thermal headroom to the cluster's HVAC, which dropped the daily failure spike from $\sim 3\times$ baseline to $\sim 1.2\times$. This is a "non-ML" problem (HVAC engineering) that the ML team had to drive because it dominated their job's MTBF. A second finding: nearly half the NCCL hangs were on a single rack with a flaky ToR switch; the cluster topology validation pass had missed it because synthetic benchmarks were under the threshold that exposed the flakiness.

59.5.7 Cost Discipline

The economic shape of a training run: a 16k-H100 cluster at $$4/hr per GPU = $64k/hr$. A 30-day run costs $$46M$. Every 1% of MFU lost to inefficiency is $$460k$. Every hour of downtime from a slow checkpoint recovery is $$64k$. These numbers concentrate the mind.

The operational rules of thumb that emerged from this discipline:

• Burn-in test before any production run: run the full job at production scale for 1 hour, on the same data, and validate (a) MFU at target, (b) checkpoint write-read round-trip works, (c) elasticity tested by killing a rank manually.
• Stagger checkpoints across runs: if 100 training runs all checkpoint at the same wall-clock instant, the parallel file system thrashes. Production schedulers stagger writes.
• Pre-bake data shards: the dataloader's hot path is dominated by tokenization and shuffling. Pre-tokenize, pre-shuffle, store as .bin + index files. The pre-bake cost is a one-time CPU job; the speed-up to throughput is permanent.
• Run a chaos-engineering pass: deliberately kill a random rank every hour during a soak test. If the run does not recover, fix the recovery, not the killer.

59.5.8 Determinism and Reproducibility

A reproducible training run lets you bisect a regression to the offending commit; an irreproducible run does not. The cost of full bit-for-bit determinism is real (10-20% throughput loss on H100s), so most teams take a middle path: scientific reproducibility, where two runs with the same seed produce loss curves that agree to within $10^{-3}$, but individual gradients may differ in the last bit.

The discipline that achieves this:

• Pin every seed: PyTorch (torch.manual_seed), NumPy, Python's random, and the dataloader's shuffle seed. Each rank gets seed $\text{base\_seed} + \text{rank}$ so they sample independently.
• Use deterministic CUDA kernels for the critical path: torch.use_deterministic_algorithms(True) selects deterministic implementations of conv, attention, and reductions. This is the throughput tax; most teams disable it during normal training and enable it only when bisecting.
• Capture the full training config in the checkpoint: optimizer state, RNG state, data-loader's current epoch, shard index, position within the shard. Skip any of these and you cannot resume to the same trajectory.
• Log every divergence-related metric: per-rank gradient norm, per-rank loss, learning rate, and the value of any clipping or skip-step decision the trainer made.

59.5.9 The Data Pipeline at Scale

A frequently-overlooked source of MFU loss: the data pipeline. At 50% MFU, the training step on a 70B model on 1000 H100s consumes roughly $10^{14}$ tokens-equivalent FLOPs per second. To stay fed, the data pipeline must deliver, say, $4 \times 10^7$ tokens per second. That is 80 MB/s per rank for BPE-tokenized text; modest, but only if the data is pre-tokenized and pre-shuffled.

The 2026 practice: tokenize once, write to a WebDataset or MosaicML StreamingDataset format (sharded .tar archives with index files), and use random sampling within shards rather than full-corpus shuffling. The dataloader runs on a separate process pool (one per rank) and prefetches several shards ahead. Pin memory and use persistent_workers=True; the per-shard load cost is amortized across thousands of batches.

Failure modes to watch for: data ordering correlated across ranks (a bug in the seeding of the data-loader can give every rank the same first batch; the loss looks fine but the effective batch is $N\times$ smaller than expected), and shard-level imbalance (some shards have longer documents and slow down their host rank, becoming the straggler on every collective).

59.5.10 Cost Attribution and Billing

For an internal team running on a shared cluster, the question "did this experiment cost me three days or three weeks of GPU-time?" is harder to answer than it sounds. Most modern training stacks instrument GPU-hour accounting at the rank level: each rank logs its allocation period to a per-experiment ledger, and the cluster scheduler totals the cost. For external cost (cloud GPU rentals), the same accounting plus a price multiplier produces dollar values.

The economic implication: if your 30-day training run on 1024 H100s costs roughly $$3$M on a major cloud, every percentage point of MFU lost to bad checkpoint cadence, dataloader stalls, or NCCL fragmentation is $$30$k of waste. Frontier labs hire dedicated training-systems engineers whose KPI is MFU; their first quarter typically pays for itself.

pip install torch transformers accelerate datasets bitsandbytes

Chapter 60 (Edge and On-Device LLMs) is the inverse of Chapter 59. Where Chapter 59 fights for every megabyte of HBM and every percent of MFU on 16,000-GPU clusters, Chapter 60 fights for every milliwatt and every kilobyte on phones, laptops, embedded NPUs, and microcontrollers. The 4-bit quantization, knowledge distillation, MoE-routing-pruning, and structured sparsity techniques in Chapter 60 are the precision-and-pruning counterpart to Chapter 59's parallelism: both are answers to the same question (how do you fit a useful LLM into a budget?), but the budgets are at opposite ends of the size spectrum.