Platforms

Training is the chapter of the book where compute is the protagonist. The platforms in this section are not Python imports; they are GPU rentals, distributed-training fabrics, and experiment-tracking SaaS dashboards.

19.1.1 Compute providers for training

For Part IV's training-scale workloads, the same provider list from Section 10.6 applies, but the price-performance ratio matters more (and interconnect, multi-node, and persistent storage become first-class concerns). Lambda, RunPod, and Vast.ai dominate the bare-metal-GPU rental market. CoreWeave, Crusoe, and Nebius serve larger clusters with NVLink and InfiniBand interconnect. Modal and Together AI wrap GPU rental in serverless Python APIs that are friendlier for small teams.

Two 2024-25 entrants reshaped the multi-GPU spot market specifically for training. SF Compute is a multi-GPU spot marketplace that became the leading auction-priced training option in 2024-25. Prime Intellect is a decentralized H100 marketplace plus an open-pretraining research org, relevant for academic teams who can tolerate non-collocated GPUs. Hyperbolic Labs and Salad Cloud cover the 4090-class tier as alternatives to vast.ai.

For full pretraining, the hyperscalers (AWS, GCP, Azure) and TPU pods still win on raw scale. AWS Trainium2 (2024) and Google Trillium TPU (2024) are the two serious non-NVIDIA training accelerators in 2026. NVIDIA DGX Cloud (boosted by the 2025 Lepton AI acquisition) covers the enterprise tier. For fine-tuning, the specialists are almost always cheaper.

Two patterns dominate the 2025-26 distributed-training workflow regardless of provider. The first is spot-friendly checkpointing via fsspec + s5cmd (S3-compatible parallel upload). The second is distributed training over commodity Internet via OpenDiLoCo (2024), the open implementation of Google's DiLoCo, which lets academic teams pretrain across geographically separated nodes without InfiniBand.

19.1.2 Distributed-training fabrics

• Hugging Face Accelerate (Hugging Face, 2021) is the thin abstraction layer that sits between your training script and the underlying parallelism backend (DDP, FSDP, DeepSpeed, TPU). Its objective is to let you write a single-GPU PyTorch loop and have it scale to 8 GPUs or 8 nodes by changing a config file, which matters when you do not want to rewrite your training code for every cluster topology. The core concept is the Accelerator object that wraps your model, optimizer, and dataloader, transparently sharding gradients or parameters depending on the chosen backend; think of it as torch.distributed with the boilerplate amortized. Pick it when you want the path of least resistance into multi-GPU training and intend to use TRL or the official recipes.
• DeepSpeed (Microsoft, 2020) is the original sharded-data-parallel framework, built around the ZeRO family of optimizer-state and gradient partitioning algorithms. Its objective is to fit much larger models in the same GPU memory by removing the redundant copies of optimizer state, gradients, and parameters that vanilla DDP keeps on every rank; this matters when a 70B model with Adam state would otherwise need a half-terabyte of HBM. The core concept is ZeRO stages 1, 2, and 3, which progressively shard more state at the cost of more communication; ZeRO-Infinity additionally offloads to CPU or NVMe. Pick DeepSpeed when you need the most aggressive memory reduction and are willing to deal with its quirkier configuration; avoid for small jobs where FSDP2 is now simpler.
• PyTorch FSDP (Meta, 2022; FSDP2 in 2024) is PyTorch's native fully-sharded data parallel implementation, conceptually equivalent to ZeRO-3 but integrated into the core framework. Its objective is to provide the same memory savings as DeepSpeed without an external dependency, while composing cleanly with torch.compile and tensor parallelism, which matters when you want your training stack to upgrade in lockstep with PyTorch itself. The core concept is per-parameter sharding with on-demand all-gather for the forward and backward passes; FSDP2 reworked this around DTensor and removed many sharp edges. Pick FSDP2 for new training code in 2026; DeepSpeed remains the default only for legacy configs.
• Megatron-LM (NVIDIA, 2019) is the canonical tensor-parallel and pipeline-parallel pretraining framework, written by the team that ships the hardware. Its objective is maximum throughput on NVLink and InfiniBand clusters for from-scratch pretraining of multi-hundred-billion-parameter models, which matters when you are spending tens of millions on a compute run and every percent of utilization is real money. The core concept is the 3D parallelism mesh (tensor, pipeline, data) with hand-tuned kernels and overlap; think of it as a Formula-1 engine compared with FSDP's daily driver. Pick Megatron only for serious pretraining at multi-node scale; avoid for fine-tuning, where its configuration overhead is wasted.
• PyTorch Lightning (Lightning AI, 2019) is a higher-level training framework that abstracts the entire loop into a LightningModule with hooks for training, validation, and optimizer steps. Its objective is to remove training-loop boilerplate so research code stays focused on the model and the loss, which matters when you are iterating on architectures rather than infrastructure. The core concept is callback-driven separation of concerns: the model defines what; the Trainer decides where and how often. Pick it for academic labs or smaller research projects; in production fine-tuning, Hugging Face Trainer + Accelerate has won mindshare.

19.1.3 Experiment tracking

• Weights & Biases (W&B, 2017) is the de facto experiment-tracking SaaS for the ML community, logging metrics, hyperparameters, model artifacts, and system telemetry to a dashboard. Its objective is to make every training run reproducible and comparable across teammates without anyone manually copying numbers into a spreadsheet, which matters the moment you have more than one run in flight. The core concept is a Run object that you initialize once and stream key-value updates to; everything else (hyperparam sweeps, artifact versioning, alerts) is built on top. Pick it for any team-scale ML work where convenience beats self-hosting; avoid only when data-residency or air-gap requirements force MLflow.
• MLflow (Databricks, 2018) is the open-source experiment-tracking and model-registry framework you can self-host on a single VM. Its objective is to give you W&B-shaped functionality without sending experiment data outside your network, which matters for regulated industries (finance, healthcare, defense). The core concept is the same Run abstraction plus a Model Registry that stages model versions through "Staging", "Production", and "Archived". Pick it when self-hosting is non-negotiable; avoid when you would rather pay for hosted W&B and skip the ops burden.
• Neptune.ai (Neptune Labs, 2019) is a research-focused tracker with stronger emphasis on reproducibility primitives like git-sha logging, dataset versioning, and per-run lineage. Its objective is to make academic experiments reviewable months later, which matters when reviewers ask for an exact reproduction of a number from a paper. The core concept is a hierarchical namespace for run metadata that you can query like a database. Pick it for research labs running long-horizon experiments; for production teams, W&B's polish typically wins.
• TensorBoard (Google, 2015) is the original training-curve dashboard, originally tied to TensorFlow but now consumed by every framework through the SummaryWriter protocol. Its objective is to give a free, local, zero-setup way to plot losses and watch them in real time, which matters when you are iterating on a single GPU at your desk. The core concept is event files written to disk that the TensorBoard server tails. Pick it for local debugging; for team-scale or long-horizon tracking, layer W&B or MLflow on top.

# Install the Databricks CLI
pip install databricks-cli

# Configure authentication with a personal access token
databricks configure --token
# Host: https://your-workspace.cloud.databricks.com
# Token: dapi1234567890abcdef

# List available clusters
databricks clusters list --output JSON

# Create a GPU cluster for ML training
databricks clusters create --json '{
  "cluster_name": "llm-training-cluster",
  "spark_version": "14.3.x-gpu-ml-scala2.12",
  "node_type_id": "Standard_NC24ads_A100_v4",
  "num_workers": 4,
  "autoscale": {"min_workers": 2, "max_workers": 8},
  "spark_conf": {
    "spark.databricks.delta.preview.enabled": "true"
  }
}'

# Databricks notebook cell: install dependencies
%pip install transformers datasets accelerate
# Cell 2: Load training data from Unity Catalog
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
# Read a Delta table managed by Unity Catalog
training_df = spark.read.table("ml_catalog.llm_data.instruction_pairs")
print(f"Training samples: {training_df.count():,}")
training_df.show(5, truncate=80)
# Cell 3: Convert Spark DataFrame to HuggingFace Dataset
import pandas as pd
from datasets import Dataset
pdf = training_df.select("instruction", "response").toPandas()
hf_dataset = Dataset.from_pandas(pdf)
print(hf_dataset)

# Create parameterized widgets for training configuration
dbutils.widgets.dropdown("model_name", "meta-llama/Llama-3.1-8B",
    ["meta-llama/Llama-3.1-8B", "mistralai/Mistral-7B-v0.3"])
dbutils.widgets.text("learning_rate", "2e-5")
dbutils.widgets.text("num_epochs", "3")

# Retrieve widget values
model_name = dbutils.widgets.get("model_name")
lr = float(dbutils.widgets.get("learning_rate"))
epochs = int(dbutils.widgets.get("num_epochs"))

print(f"Training {model_name} with lr={lr}, epochs={epochs}")

-- Create a catalog and schema for LLM training assets
CREATE CATALOG IF NOT EXISTS ml_catalog;
USE CATALOG ml_catalog;

CREATE SCHEMA IF NOT EXISTS llm_data
  COMMENT 'Training data for LLM fine-tuning projects';

-- Create a managed Delta table for instruction-tuning pairs
CREATE TABLE IF NOT EXISTS llm_data.instruction_pairs (
  id BIGINT GENERATED ALWAYS AS IDENTITY,
  instruction STRING NOT NULL,
  response STRING NOT NULL,
  source STRING,
  quality_score FLOAT,
  created_at TIMESTAMP DEFAULT current_timestamp()
)
USING DELTA
COMMENT 'Curated instruction-response pairs for SFT training';

-- Grant read access to the ML engineering team
GRANT SELECT ON TABLE llm_data.instruction_pairs
  TO `ml-engineers@company.com`;

import mlflow
from mlflow.models import infer_signature

# Set the MLflow experiment (auto-created if it does not exist)
mlflow.set_experiment("/Users/you@company.com/llm-fine-tuning")

with mlflow.start_run(run_name="llama-sft-v1") as run:
    # Log training hyperparameters
    mlflow.log_params({
        "model_name": "meta-llama/Llama-3.1-8B",
        "learning_rate": 2e-5,
        "batch_size": 8,
        "num_epochs": 3,
        "lora_rank": 16,
        "dataset_version": "v2.1",
    })

    # ... training loop here ...

    # Log metrics at each epoch
    for epoch in range(3):
        mlflow.log_metrics({
            "train_loss": 1.2 - epoch * 0.3,
            "eval_loss": 1.1 - epoch * 0.25,
        }, step=epoch)

    # Register the model in Unity Catalog
    mlflow.transformers.log_model(
        transformers_model={"model": model, "tokenizer": tokenizer},
        artifact_path="llm",
        registered_model_name="ml_catalog.llm_models.llama_sft",
    )

    print(f"Run ID: {run.info.run_id}")

from databricks.sdk import WorkspaceClient
from databricks.sdk.service.serving import (
    EndpointCoreConfigInput,
    ServedEntityInput,
)

w = WorkspaceClient()

# Create a GPU-backed serving endpoint
w.serving_endpoints.create_and_wait(
    name="llama-sft-endpoint",
    config=EndpointCoreConfigInput(
        served_entities=[
            ServedEntityInput(
                entity_name="ml_catalog.llm_models.llama_sft",
                entity_version="1",
                workload_size="Small",      # Small / Medium / Large
                scale_to_zero_enabled=True,
                workload_type="GPU_MEDIUM",
            )
        ]
    ),
)

# Query the endpoint
import requests
response = requests.post(
    "https://your-workspace.cloud.databricks.com/serving-endpoints/llama-sft-endpoint/invocations",
    headers={"Authorization": "Bearer dapi..."},
    json={"inputs": {"instruction": "Explain gradient descent."}},
)
print(response.json())