Kubernetes-Native LLM Operations: Scheduling, Serving, and GPU Management

"Kubernetes does not care that your model has 70 billion parameters. It cares about resource requests, health checks, and pod scheduling. The gap between those two worlds is where production LLM engineering lives."

Deploy, Container-Wrangling AI Agent

1. GPU Scheduling for LLM Training

Standard Kubernetes scheduling treats GPUs as simple countable resources: a pod requests N GPUs, and the scheduler finds a node with N available. This works for single-node inference but fails for distributed LLM training, which has requirements that the default scheduler cannot express:

• Gang scheduling: A distributed training job needs all its pods to start simultaneously. If only 7 of 8 pods can be placed, the job cannot proceed, and those 7 pods waste GPU resources while waiting for the 8th.
• Topology awareness: Pods in the same tensor-parallel group must be placed on GPUs within the same node (connected by NVLink), while pipeline-parallel groups benefit from being on nodes in the same rack (lower network latency).
• Fair sharing: Multiple teams sharing a GPU cluster need quota management to prevent a single large training job from monopolizing all resources.
• Preemption: High-priority training jobs (nearing deadlines) should be able to preempt lower-priority jobs, which checkpoint and resume later.

Figure 31.9.1: The Kubernetes-native LLM operations stack. Training workloads flow through batch scheduling tools (Kueue for quotas, Volcano for gang scheduling, Training Operator for PyTorchJobs). Inference workloads use KServe for model routing and custom HPA for autoscaling. Both layers rely on the NVIDIA GPU Operator for driver lifecycle management and MIG partitioning.

1.1 Kueue: Admission Control and Quotas

Kueue is the Kubernetes-native job queueing system (part of the Kubernetes SIGs ecosystem) designed for batch and ML workloads. It provides admission control, fair-sharing quotas, and priority-based scheduling for GPU jobs. Kueue does not replace the Kubernetes scheduler; instead, it controls which jobs are admitted to the cluster and when.

Key Kueue concepts for LLM workloads:

• ClusterQueue: Defines the total GPU resources available to a group of users, with borrowing limits and preemption policies.
• LocalQueue: A namespace-scoped queue that teams submit jobs to. Each LocalQueue is bound to a ClusterQueue.
• ResourceFlavor: Distinguishes between GPU types (A100 vs H100), node topologies, or availability zones.
• Workload: Kueue's abstraction of a job. It computes the total resource requirements and decides whether to admit the workload based on quota availability.

# Code Fragment 31.9.5: Kueue ClusterQueue for a shared GPU training cluster
apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
 name: h100-80gb
spec:
 nodeLabels:
 nvidia.com/gpu.product: "NVIDIA-H100-80GB-HBM3"
 topology.kubernetes.io/zone: "us-central1-a"
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
 name: gpu-training-queue
spec:
 namespaceSelector: {} # Accept jobs from all namespaces
 preemption:
 reclaimWithinCohort: Any
 withinClusterQueue: LowerPriority
 resourceGroups:
 - coveredResources: ["cpu", "memory", "nvidia.com/gpu"]
 flavors:
 - name: h100-80gb
 resources:
 - name: "nvidia.com/gpu"
 nominalQuota: 128 # 128 H100 GPUs total
 borrowingLimit: 32 # Can borrow up to 32 from other queues
 lendingLimit: 64 # Can lend up to 64 to other queues
 - name: "cpu"
 nominalQuota: 1024
 - name: "memory"
 nominalQuota: "8Ti"
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
 namespace: llm-team-alpha
 name: training-queue
spec:
 clusterQueue: gpu-training-queue

1.2 Volcano: Batch Scheduling with Gang Semantics

Volcano is a CNCF project that provides a batch scheduling framework for Kubernetes, with native support for gang scheduling, fair-share policies, and topology-aware placement. For distributed LLM training, Volcano ensures that all pods in a training job are scheduled simultaneously or not at all, preventing resource waste from partial scheduling.

# Code Fragment 31.9.5: Volcano Job for multi-node LLM training with gang scheduling
apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
 name: llama-70b-pretraining
 namespace: llm-training
spec:
 minAvailable: 8 # Gang scheduling: all 8 pods must be schedulable
 schedulerName: volcano
 plugins:
 svc: [] # Create a headless service for pod discovery
 ssh: [] # Enable SSH between pods for NCCL
 queue: default
 policies:
 - event: PodEvicted
 action: RestartJob # Restart entire job if any pod is evicted
 - event: TaskCompleted
 action: CompleteJob
 tasks:
 - replicas: 8 # 8 nodes, each with 8 GPUs
 name: trainer
 template:
 spec:
 containers:
 - name: trainer
 image: nvcr.io/nvidia/pytorch:24.07-py3
 command:
 - torchrun
 - --nproc_per_node=8
 - --nnodes=8
 - --rdzv_backend=c10d
 - --rdzv_endpoint=$(MASTER_ADDR):29400
 - train_llm.py
 resources:
 requests:
 nvidia.com/gpu: 8
 cpu: "96"
 memory: "1500Gi"
 limits:
 nvidia.com/gpu: 8
 env:
 - name: NCCL_IB_DISABLE
 value: "0" # Enable InfiniBand for NCCL
 - name: NCCL_DEBUG
 value: "INFO"
 schedulerName: volcano
 # Topology-aware placement: prefer same rack
 affinity:
 podAffinity:
 preferredDuringSchedulingIgnoredDuringExecution:
 - weight: 100
 podAffinityTerm:
 labelSelector:
 matchExpressions:
 - key: volcano.sh/job-name
 operator: In
 values: ["llama-70b-pretraining"]
 topologyKey: topology.kubernetes.io/rack

2. Kubeflow Training Operator

The Kubeflow Training Operator provides Kubernetes Custom Resource Definitions (CRDs) for managing distributed training jobs. The PyTorchJob CRD is the most commonly used resource for LLM training, handling the coordination of multi-node PyTorch distributed training including master election, worker discovery, and environment setup.

# Code Fragment 31.9.5: PyTorchJob CRD for distributed LLM fine-tuning
apiVersion: kubeflow.org/v1
kind: PyTorchJob
metadata:
 name: llama-8b-finetune
 namespace: llm-training
spec:
 elasticPolicy:
 rdzvBackend: c10d
 minReplicas: 2
 maxReplicas: 4 # Elastic: can scale between 2 and 4 nodes
 maxRestarts: 3
 pytorchReplicaSpecs:
 Master:
 replicas: 1
 restartPolicy: OnFailure
 template:
 spec:
 containers:
 - name: pytorch
 image: ghcr.io/my-org/llm-trainer:v2.3
 command:
 - python
 - -m
 - torch.distributed.run
 - --nproc_per_node=8
 - --rdzv_backend=c10d
 - finetune.py
 - --model_name=meta-llama/Llama-3.1-8B
 - --dataset=my-org/instruction-data
 - --output_dir=/shared/checkpoints/llama-8b-ft
 - --per_device_train_batch_size=2
 - --gradient_accumulation_steps=4
 - --bf16
 - --lora_r=16
 resources:
 requests:
 nvidia.com/gpu: 8
 cpu: "48"
 memory: "256Gi"
 limits:
 nvidia.com/gpu: 8
 volumeMounts:
 - name: shared-storage
 mountPath: /shared
 - name: hf-cache
 mountPath: /root/.cache/huggingface
 volumes:
 - name: shared-storage
 persistentVolumeClaim:
 claimName: training-pvc
 - name: hf-cache
 persistentVolumeClaim:
 claimName: hf-cache-pvc
 Worker:
 replicas: 3 # 3 additional workers (4 total with master)
 restartPolicy: OnFailure
 template:
 spec:
 containers:
 - name: pytorch
 image: ghcr.io/my-org/llm-trainer:v2.3
 command:
 - python
 - -m
 - torch.distributed.run
 - --nproc_per_node=8
 - --rdzv_backend=c10d
 - finetune.py
 - --model_name=meta-llama/Llama-3.1-8B
 - --dataset=my-org/instruction-data
 - --output_dir=/shared/checkpoints/llama-8b-ft
 - --per_device_train_batch_size=2
 - --gradient_accumulation_steps=4
 - --bf16
 - --lora_r=16
 resources:
 requests:
 nvidia.com/gpu: 8
 cpu: "48"
 memory: "256Gi"
 limits:
 nvidia.com/gpu: 8
 volumeMounts:
 - name: shared-storage
 mountPath: /shared
 - name: hf-cache
 mountPath: /root/.cache/huggingface
 volumes:
 - name: shared-storage
 persistentVolumeClaim:
 claimName: training-pvc
 - name: hf-cache
 persistentVolumeClaim:
 claimName: hf-cache-pvc

3. Production LLM Serving on Kubernetes

Serving LLMs in production on Kubernetes requires more than a standard Deployment with a container running vLLM. Production serving needs health checks that understand model loading state, canary rollouts that compare latency and quality metrics between model versions, and resource management that accounts for the unique memory profile of LLM inference (large static model weights plus dynamic KV cache).

3.1 KServe with vLLM and TGI Runtimes

KServe (formerly KFServing) is the Kubernetes-native model serving framework that provides a standardized inference protocol, autoscaling, canary rollouts, and multi-model serving. For LLM workloads, KServe integrates with vLLM and Text Generation Inference (TGI) as serving runtimes.

# Code Fragment 31.9.5: KServe InferenceService with vLLM runtime
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
 name: llama-3-1-70b
 namespace: llm-serving
 annotations:
 # Canary rollout: send 10% of traffic to the new version
 serving.kserve.io/canaryTrafficPercent: "10"
spec:
 predictor:
 minReplicas: 2 # Minimum replicas (never scale to zero)
 maxReplicas: 8
 scaleTarget: 5 # Target concurrent requests per replica
 scaleMetric: concurrency
 containers:
 - name: kserve-container
 image: vllm/vllm-openai:v0.6.4
 args:
 - --model=meta-llama/Llama-3.1-70B-Instruct
 - --tensor-parallel-size=4 # 4 GPUs per replica
 - --max-model-len=8192
 - --gpu-memory-utilization=0.90
 - --enable-chunked-prefill
 - --max-num-seqs=128
 - --port=8080
 resources:
 requests:
 nvidia.com/gpu: 4
 cpu: "24"
 memory: "200Gi"
 limits:
 nvidia.com/gpu: 4
 ports:
 - containerPort: 8080
 protocol: TCP
 # Custom health checks for LLM model loading
 readinessProbe:
 httpGet:
 path: /health
 port: 8080
 initialDelaySeconds: 120 # Model loading takes ~2 minutes
 periodSeconds: 10
 failureThreshold: 3
 livenessProbe:
 httpGet:
 path: /health
 port: 8080
 initialDelaySeconds: 180
 periodSeconds: 30
 failureThreshold: 5
 startupProbe:
 httpGet:
 path: /health
 port: 8080
 initialDelaySeconds: 60
 periodSeconds: 10
 failureThreshold: 30 # Allow up to 5 minutes for startup

3.2 Canary Rollouts for Model Updates

Updating a production LLM (new model version, updated weights, or configuration changes) carries risk. A canary rollout directs a small percentage of traffic to the new version while monitoring key metrics. KServe supports this natively through the canaryTrafficPercent annotation.

A typical canary rollout for an LLM model update follows these stages:

1. Deploy canary (5-10% traffic): Deploy the new model version alongside the current one. Route 5-10% of traffic to the canary.
2. Monitor metrics (1-4 hours): Compare TTFT (Time to First Token), TPOT (Time Per Output Token), P99 latency, error rate, and (if available) quality metrics between canary and production.
3. Promote or rollback: If metrics are within acceptable bounds (typically within 10% of baseline on latency, no increase in error rate), gradually increase canary traffic to 25%, 50%, then 100%. If any metric degrades, roll back immediately.

4. GPU Sharing and Isolation

A single high-end GPU (H100 with 80 GB HBM3) is often more than what a small LLM inference workload needs. Serving a 7B model in FP16 requires approximately 14 GB of GPU memory, leaving 66 GB unused on an H100. GPU sharing techniques allow multiple models or workloads to share a single physical GPU, dramatically improving cluster utilization.

4.1 NVIDIA GPU Operator

The NVIDIA GPU Operator automates the management of all NVIDIA software components needed for GPU workloads on Kubernetes: drivers, container toolkit, device plugin, DCGM (Data Center GPU Manager), and MIG (Multi-Instance GPU) configuration. It runs as a set of DaemonSets that ensure every GPU node has the correct software stack.

4.2 MIG Partitioning for Multi-Model Serving

Multi-Instance GPU (MIG) partitions a single physical GPU into isolated instances, each with its own compute cores, memory, and memory bandwidth. On an H100, MIG supports up to 7 instances, each with its own slice of the GPU's SM (Streaming Multiprocessor) array and HBM.

# Code Fragment 31.9.5: MIG configuration via NVIDIA GPU Operator
apiVersion: nvidia.com/v1alpha1
kind: MIGConfig
metadata:
 name: llm-serving-mig
spec:
 mig-config:
 # Partition each H100 into one 3g.40gb + two 2g.20gb instances
 # Suitable for: one 7B FP16 model + two quantized small models
 - devices: all
 mig-enabled: true
 mig-devices:
 "3g.40gb": 1 # For a 7B FP16 model (primary)
 "2g.20gb": 2 # For two INT4 quantized models (secondary)
---
# Pods request specific MIG profiles
apiVersion: v1
kind: Pod
metadata:
 name: llama-7b-serving
spec:
 containers:
 - name: vllm
 image: vllm/vllm-openai:v0.6.4
 resources:
 limits:
 nvidia.com/mig-3g.40gb: 1 # Request one 3g.40gb MIG instance

4.3 MPS for Time-Slicing

NVIDIA Multi-Process Service (MPS) provides time-slicing of a GPU among multiple CUDA contexts. Unlike MIG, which creates hardware-isolated partitions, MPS shares the GPU's compute and memory across processes with software-level scheduling. MPS is simpler to configure than MIG but offers weaker isolation: a misbehaving process can affect others sharing the GPU.

For LLM serving, MPS is useful when multiple small models (embeddings, classifiers, rerankers) need GPU access but none individually justifies a full GPU or a MIG partition. The NVIDIA device plugin supports MPS-based GPU sharing through the nvidia.com/gpu resource with the sharing.timeSlicing configuration in the GPU Operator's ClusterPolicy.

5. Autoscaling LLM Services

Standard Kubernetes HPA (Horizontal Pod Autoscaler) scales based on CPU or memory utilization. For LLM serving, these metrics are poor proxies for load because GPU utilization and request queue depth are the actual bottlenecks. Effective LLM autoscaling requires custom metrics and strategies tailored to inference workload patterns.

5.1 Custom Metrics for LLM Autoscaling

The most useful autoscaling signals for LLM inference workloads are:

# Code Fragment 31.9.5: HPA with custom metrics for LLM serving
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
 name: llama-70b-hpa
 namespace: llm-serving
spec:
 scaleTargetRef:
 apiVersion: apps/v1
 kind: Deployment
 name: llama-70b-vllm
 minReplicas: 2
 maxReplicas: 12
 behavior:
 scaleUp:
 stabilizationWindowSeconds: 60 # Wait 60s before scaling up
 policies:
 - type: Pods
 value: 2 # Add at most 2 pods at a time
 periodSeconds: 120
 scaleDown:
 stabilizationWindowSeconds: 300 # Wait 5 min before scaling down
 policies:
 - type: Pods
 value: 1 # Remove at most 1 pod at a time
 periodSeconds: 300 # Every 5 minutes
 metrics:
 # Primary metric: request queue depth from vLLM
 - type: Pods
 pods:
 metric:
 name: vllm_num_requests_waiting
 target:
 type: AverageValue
 averageValue: "5" # Scale up if >5 waiting requests
 # Secondary metric: TTFT P95 latency
 - type: Pods
 pods:
 metric:
 name: vllm_time_to_first_token_p95
 target:
 type: AverageValue
 averageValue: "500" # Scale up if TTFT P95 > 500ms

5.2 Scale-to-Zero with Warm Pools

For LLM services that serve intermittent traffic (internal tools, development environments, low-traffic endpoints), scaling to zero when idle saves significant GPU cost. However, LLM cold starts are slow: loading a 70B model from disk to GPU takes 2-5 minutes, which is unacceptable for user-facing services.

Warm pool strategies mitigate cold starts by maintaining a pool of pre-loaded model replicas that are suspended (GPU memory retained but not actively serving) rather than fully terminated. When traffic arrives, a warm replica can begin serving within seconds rather than minutes. KServe supports scale-to-zero with configurable grace periods:

# Scale-to-zero configuration in KServe
spec:
 predictor:
 minReplicas: 0 # Allow scale to zero
 scaleTarget: 1
 annotations:
 # Grace period before scaling to zero (10 minutes of no traffic)
 autoscaling.knative.dev/scale-to-zero-pod-retention-period: "10m"
 # Window to observe traffic before scaling decision
 autoscaling.knative.dev/window: "120s"

5.3 Cold-Start Mitigation Strategies

For services that cannot tolerate cold-start latency, several strategies reduce or eliminate startup time:

• Model preloading via init containers: An init container downloads the model weights to a local SSD (hostPath or emptyDir) before the serving container starts, parallelizing model download with pod scheduling.
• Shared model cache (ReadWriteMany PVC): Store model weights on a shared file system (Lustre, GCS FUSE, EFS) so that new replicas read from cache rather than downloading from the model registry.
• Predictive scaling: Use historical traffic patterns to pre-scale before expected demand spikes (e.g., scale up at 8 AM before the workday begins).
• CUDA memory snapshots: Experimental approaches that snapshot the GPU state of a loaded model and restore it on a new GPU, reducing cold start to seconds.

6. Networking and Storage Considerations

Distributed LLM training on Kubernetes has unique networking requirements. NCCL (NVIDIA Collective Communications Library) needs high-bandwidth, low-latency communication between GPU pods, typically over InfiniBand or RoCE (RDMA over Converged Ethernet). Standard Kubernetes networking (overlay networks, kube-proxy) adds unacceptable overhead for collective operations.

6.1 Network Configuration for NCCL

• Host networking: The simplest approach is to use hostNetwork: true on training pods, bypassing the Kubernetes overlay network entirely. This gives pods direct access to the InfiniBand or RDMA interfaces.
• SR-IOV device plugin: For clusters that require network isolation, the SR-IOV device plugin exposes virtual functions (VFs) from the InfiniBand HCA as Kubernetes resources that pods can request.
• RDMA shared device plugin: Allows multiple pods on the same node to share an RDMA interface without SR-IOV, useful for development and smaller-scale training.

6.2 Storage for Checkpoints and Model Artifacts

Distributed checkpointing (covered in Section 6.8) requires a parallel file system or object storage that all training pods can write to simultaneously. Common options on Kubernetes include:

• Lustre or GPFS (via CSI driver): High-performance parallel file systems that provide the bandwidth needed for large checkpoint writes. Best for clusters with dedicated storage infrastructure.
• GCS FUSE / S3 FUSE: Mount cloud object storage as a POSIX file system. Simpler to set up but lower write throughput; suitable for async checkpoint staging where the GPU-to-CPU copy is the blocking step.
• Local NVMe (hostPath/emptyDir): Fastest for temporary storage (local checkpoint staging, model cache) but not shared across nodes and not persistent across pod restarts.

pip install vllm locust requests

# Start vLLM in a separate terminal
python -m vllm.entrypoints.openai.api_server \
 --model TinyLlama/TinyLlama-1.1B-Chat-v1.0 \
 --port 8000 \
 --max-model-len 2048

import requests
import time
import statistics

def measure_request(prompt, max_tokens=50):
 """Send a single completion request and measure timing."""
 start = time.perf_counter()
 response = requests.post(
 "http://localhost:8000/v1/completions",
 json={
 "model": "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
 "prompt": prompt,
 "max_tokens": max_tokens,
 "temperature": 0.0,
 },
 )
 end = time.perf_counter()

 data = response.json()
 latency_ms = (end - start) * 1000
 tokens_generated = data["usage"]["completion_tokens"]
 tokens_per_sec = tokens_generated / (latency_ms / 1000) if latency_ms > 0 else 0

 return {
 "latency_ms": latency_ms,
 "tokens": tokens_generated,
 "tokens_per_sec": tokens_per_sec,
 }

# Run 10 sequential requests for baseline measurement
prompt = "Explain the concept of attention in neural networks in three sentences."
results = [measure_request(prompt) for _ in range(10)]

latencies = [r["latency_ms"] for r in results]
throughputs = [r["tokens_per_sec"] for r in results]

print("Manual Benchmark (10 sequential requests):")
print(f" Median latency: {statistics.median(latencies):.1f} ms")
print(f" P95 latency: {sorted(latencies)[8]:.1f} ms")
print(f" Mean throughput: {statistics.mean(throughputs):.1f} tokens/sec")
print(f" Tokens per req: {results[0]['tokens']}")

# Save this as locustfile.py
from locust import HttpUser, task, between
import json

class LLMUser(HttpUser):
 """Simulates a user sending completion requests to the LLM."""
 wait_time = between(0.5, 2.0)
 host = "http://localhost:8000"

 prompts = [
 "Summarize the key ideas of transformer architecture.",
 "What are the benefits of quantization for LLM deployment?",
 "Explain continuous batching in three sentences.",
 "Describe the difference between prefill and decode phases.",
 "What is KV cache and why does it matter for inference?",
 ]

 @task
 def generate_completion(self):
 import random
 prompt = random.choice(self.prompts)
 payload = {
 "model": "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
 "prompt": prompt,
 "max_tokens": 100,
 "temperature": 0.7,
 }
 with self.client.post(
 "/v1/completions",
 json=payload,
 catch_response=True,
 ) as response:
 if response.status_code == 200:
 data = response.json()
 tokens = data["usage"]["completion_tokens"]
 response.success()
 else:
 response.failure(f"Status {response.status_code}")

# Run Locust in headless mode with 10 concurrent users
locust -f locustfile.py --headless \
 --users 10 --spawn-rate 2 --run-time 60s \
 --csv results/load_test

# For the web UI (interactive), omit --headless:
# locust -f locustfile.py

import pandas as pd

# Analyze the Locust CSV output
stats = pd.read_csv("results/load_test_stats.csv")
print(stats[["Name", "Request Count", "Median Response Time",
 "95% Response Time", "99% Response Time",
 "Average Response Time", "Requests/s"]].to_string(index=False))

# Compare with the manual baseline
print(f"\nManual baseline median: {statistics.median(latencies):.1f} ms")
print(f"Load test median: {stats['Median Response Time'].iloc[-1]:.1f} ms")
print(f"Latency increase: "
 f"{stats['Median Response Time'].iloc[-1] / statistics.median(latencies):.1f}x")