"Kubernetes does not understand GPUs by default; you teach it."
Deploy, Container-Disciplined AI Agent
Chapter 64 orchestrated workflows; this chapter packages the workers. Containers, Kubernetes, KServe, KubeAI, GPU operators, node autoscalers, and the production-grade serving stack that GPU-aware Kubernetes has become.
Production LLM serving runs on containers, and at scale, on Kubernetes. This chapter covers Docker fundamentals, writing Dockerfiles for ML workloads, Docker Compose for multi-service apps, containerizing inference servers (vLLM, TGI, Triton), and Kubernetes-native patterns for GPU scheduling and serving.
Chapter Overview
Containers and Kubernetes are the universal packaging and scheduling layer for production LLM stacks. This chapter walks the full progression: Docker fundamentals (images, containers, volumes), Dockerfile patterns for ML and LLM projects, Docker Compose for multi-service AI applications, containerizing LLM inference servers (vLLM, TGI, Ollama), and Kubernetes-native LLM operations (GPU scheduling, model serving, autoscaling, GPU partitioning).
Container and Kubernetes tooling has stabilized enough that even early-stage products should ship on it. This chapter is the practitioner's syllabus for getting there.
- Apply Docker fundamentals (images, containers, volumes) to an ML or LLM project.
- Write a Dockerfile that minimizes size, layer churn, and security surface for LLM workloads.
- Architect a Docker Compose stack for a multi-service AI application.
- Containerize vLLM, TGI, or Ollama for production serving.
- Design Kubernetes-native LLM operations with GPU scheduling, autoscaling, and MIG or time-slicing.
Sections in This Chapter
Prerequisites
- Production engineering from Chapter 62
- Scale tools (vLLM, TensorRT-LLM) from Chapter 61
- Working knowledge of Docker and Kubernetes
- 65.1 Docker Fundamentals: Images, Containers, and Volumes Docker packages applications and their dependencies into lightweight, portable units called containers. Intermediate
- 65.2 Writing Dockerfiles for ML and LLM Projects A Dockerfile is the recipe that defines how to build a Docker image. Intermediate
- 65.3 Docker Compose for Multi-Service AI Applications Real-world AI applications rarely consist of a single container. Intermediate
- 65.4 Containerizing LLM Inference Servers LLM inference servers like vLLM, Text Generation Inference (TGI), and Ollama are designed to run inside containers. Advanced
- 65.5 Kubernetes-Native LLM Operations: Scheduling, Serving, and GPU Management GPU scheduling, Kubeflow training operators, production LLM serving on Kubernetes, and GPU sharing/isolation strategies. Advanced
- 65.5a Autoscaling, Networking & Storage for K8s LLMs Autoscaling Kubernetes LLM services with HPA and KEDA, plus networking and storage considerations for production LLM workloads. Advanced
What's Next?
This chapter begins with Section 65.1: Docker Fundamentals: Images, Containers, and Volumes. Each section builds on the previous one, so we recommend reading them in order.