"The carbon cost of training is a one-time bill; the carbon cost of inference compounds with every user."
Compass, Carbon-Aware AI Agent
Chapter 54 covered transparency; this chapter covers a different kind of disclosure: environmental impact. We work through training-time vs inference-time energy, regional grid carbon intensity, water use, and the small efficiency choices that compound across millions of inferences.
Chapter Overview
Training a frontier LLM emits tons of CO2 equivalent, but inference at scale emits more. This chapter covers the energy and carbon footprint of LLM systems, the measurement frameworks that quantify it, and the engineering choices (model size, hardware, batching, regional energy mix) that reduce environmental impact without trading away product quality.
The macro-level questions: environmental cost and governance. The first half covers Green AI (training and inference energy, water, hardware lifecycle, carbon accounting for ML workloads). The second half covers AI governance frameworks (EU AI Act, NIST RMF, ISO 42001), open governance problems (international coordination, frontier risk, capability evaluations), and where the regulatory landscape is heading.
- Account for training and inference energy, water, and carbon footprint at the workload level.
- Apply Green-AI techniques (model size choice, quantization, batching, regional energy mix) to a target deployment.
- Compare hardware lifecycle impacts across NVIDIA, AMD, and emerging accelerator silicon.
- Architect a carbon-accounting workflow for an enterprise LLM program.
- Evaluate the trade-off between model quality and environmental impact for a specific product.
Prerequisites
Sections
- 55.1 Quantifying the Environmental Cost First-principles formulas for training and inference carbon footprints, standardized efficiency metrics (FLOPs/token, MFU, tokens/kWh), and the inference-aware scaling argument that explains why the modern 7B-class checkpoint dominates. Intermediate
- 55.2 Reducing the Footprint Architectural levers (MoE, distillation, quantization), region selection, and the carbon-tracking tooling stack (CodeCarbon, Climatiq, Boavizta, ML.energy Leaderboard) that turn each mitigation into an auditable number, plus the rebound effect that erases gains if usage scales faster than efficiency. Advanced
- 55.3 Operating Under Compliance Experiment-level energy profiling (tokens/joule), green inference strategies, EU AI Act Article 53 GPAI environmental disclosure obligations and 2026 timelines, and a 10-point Green-AI checklist for production deployments. Intermediate
What's Next?
This chapter begins with Section 55.1: Quantifying the Environmental Cost and builds through mitigation (55.2) to compliance and operations (55.3). Each section builds on the previous one, so we recommend reading them in order.