Autoscaling, Networking & Storage for K8s LLMs

Autoscaling was easy. Networking was tedious. Storage was a debugging nightmare. Putting all three together in one Kubernetes cluster was the most expensive way I have ever rediscovered the value of monoliths.

Deploy, Cluster-Weary AI Agent

65.5.5 Autoscaling LLM Services

Standard Kubernetes HPA (Horizontal Pod Autoscaler) scales based on CPU or memory utilization. For LLM serving, these metrics are poor proxies for load because GPU utilization and request queue depth are the actual bottlenecks. Effective LLM autoscaling requires custom metrics and strategies tailored to inference workload patterns.

65.5.5.1 Custom Metrics for LLM Autoscaling

The most useful autoscaling signals for LLM inference workloads are:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: llama-70b-hpa
namespace: llm-serving
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: llama-70b-vllm
minReplicas: 2
maxReplicas: 12
behavior:
scaleUp:
stabilizationWindowSeconds: 60 # Wait 60s before scaling up
policies:
- type: Pods
value: 2 # Add at most 2 pods at a time
periodSeconds: 120
scaleDown:
stabilizationWindowSeconds: 300 # Wait 5 min before scaling down
policies:
- type: Pods
value: 1 # Remove at most 1 pod at a time
periodSeconds: 300 # Every 5 minutes
metrics:
# Primary metric: request queue depth from vLLM
- type: Pods
pods:
metric:
name: vllm_num_requests_waiting
target:
type: AverageValue
averageValue: "5" # Scale up if >5 waiting requests
# Secondary metric: TTFT P95 latency
- type: Pods
pods:
metric:
name: vllm_time_to_first_token_p95
target:
type: AverageValue
averageValue: "500" # Scale up if TTFT P95 > 500ms

65.5.5.2 Scale-to-Zero with Warm Pools

For LLM services that serve intermittent traffic (internal tools, development environments, low-traffic endpoints), scaling to zero when idle saves significant GPU cost. However, LLM cold starts are slow: loading a 70B model from disk to GPU takes 2-5 minutes, which is unacceptable for user-facing services.

Warm pool strategies mitigate cold starts by maintaining a pool of pre-loaded model replicas that are suspended (GPU memory retained but not actively serving) rather than fully terminated. When traffic arrives, a warm replica can begin serving within seconds rather than minutes. KServe supports scale-to-zero with configurable grace periods:

65.5.5.3 Cold-Start Mitigation Strategies

For services that cannot tolerate cold-start latency, several strategies reduce or eliminate startup time:

• Model preloading via init containers: An init container downloads the model weights to a local SSD (hostPath or emptyDir) before the serving container starts, parallelizing model download with pod scheduling.
• Shared model cache (ReadWriteMany PVC): Store model weights on a shared file system (Lustre, GCS FUSE, EFS) so that new replicas read from cache rather than downloading from the model registry.
• Predictive scaling: Use historical traffic patterns to pre-scale before expected demand spikes (e.g., scale up at 8 AM before the workday begins).
• CUDA memory snapshots: Experimental approaches that snapshot the GPU state of a loaded model and restore it on a new GPU, reducing cold start to seconds.

Fun Fact

Google's internal LLM serving infrastructure reportedly keeps "standby" TPU pods with models pre-loaded in memory, consuming significant resources even when idle. The cost of this approach is justified by the even larger cost of lost revenue when a user-facing LLM service takes 3 minutes to respond to the first request after a scale-up event. The engineering trade-off between idle resource cost and cold-start penalty is one of the defining tensions in production LLM serving economics.

65.5.6 Networking and Storage Considerations

Distributed LLM training on Kubernetes has unique networking requirements. NCCL (NVIDIA Collective Communications Library) needs high-bandwidth, low-latency communication between GPU pods, typically over InfiniBand or RoCE (RDMA over Converged Ethernet). Standard Kubernetes networking (overlay networks, kube-proxy) adds unacceptable overhead for collective operations.

65.5.6.1 Network Configuration for NCCL

• Host networking: The simplest approach is to use hostNetwork: true on training pods, bypassing the Kubernetes overlay network entirely. This gives pods direct access to the InfiniBand or RDMA interfaces.
• SR-IOV device plugin: For clusters that require network isolation, the SR-IOV device plugin exposes virtual functions (VFs) from the InfiniBand HCA as Kubernetes resources that pods can request.
• RDMA shared device plugin: Allows multiple pods on the same node to share an RDMA interface without SR-IOV, useful for development and smaller-scale training.

65.5.6.2 Storage for Checkpoints and Model Artifacts

Distributed checkpointing (covered in Section 6.8) requires a parallel file system or object storage that all training pods can write to simultaneously. Common options on Kubernetes include:

• Lustre or GPFS (via CSI driver): High-performance parallel file systems that provide the bandwidth needed for large checkpoint writes. Best for clusters with dedicated storage infrastructure.
• GCS FUSE / S3 FUSE: Mount cloud object storage as a POSIX file system. Simpler to set up but lower write throughput; suitable for async checkpoint staging where the GPU-to-CPU copy is the blocking step.
• Local NVMe (hostPath/emptyDir): Fastest for temporary storage (local checkpoint staging, model cache) but not shared across nodes and not persistent across pod restarts.

# Start vLLM in a separate terminal
python -m vllm.entrypoints.openai.api_server \
 --model TinyLlama/TinyLlama-1.1B-Chat-v1.0 \
 --port 8000 \
 --max-model-len 2048

import requests
import time
import statistics
def measure_request(prompt, max_tokens=50):
    """Send a single completion request and measure timing."""
    start = time.perf_counter()
    # Production code should wrap this in try/except for network errors.
    response = requests.post(
        "http://localhost:8000/v1/completions",
        json={
        "model": "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
        "prompt": prompt,
        "max_tokens": max_tokens,
        "temperature": 0.0,
        },
    )
    end = time.perf_counter()
    data = response.json()
    latency_ms = (end - start) * 1000
    tokens_generated = data["usage"]["completion_tokens"]
    tokens_per_sec = tokens_generated / (latency_ms / 1000) if latency_ms > 0 else 0
    return {
        "latency_ms": latency_ms,
        "tokens": tokens_generated,
        "tokens_per_sec": tokens_per_sec,
    }
# Run 10 sequential requests for baseline measurement
prompt = "Explain the concept of attention in neural networks in three sentences."
results = [measure_request(prompt) for _ in range(10)]
latencies = [result["latency_ms"] for result in results]
throughputs = [result["tokens_per_sec"] for result in results]
print("Manual Benchmark (10 sequential requests):")
print(f" Median latency: {statistics.median(latencies):.1f} ms")
print(f" P95 latency: {sorted(latencies)[8]:.1f} ms")
print(f" Mean throughput: {statistics.mean(throughputs):.1f} tokens/sec")
print(f" Tokens per req: {results[0]['tokens']}")

# Save this as locustfile.py
from locust import HttpUser, task, between
import json
class LLMUser(HttpUser):
    """Simulates a user sending completion requests to the LLM."""
    wait_time = between(0.5, 2.0)
    host = "http://localhost:8000"
    prompts = [
        "Summarize the key ideas of transformer architecture.",
        "What are the benefits of quantization for LLM deployment?",
        "Explain continuous batching in three sentences.",
        "Describe the difference between prefill and decode phases.",
        "What is KV cache and why does it matter for inference?",
        ]
    @task
    def generate_completion(self):
        import random
        prompt = random.choice(self.prompts)
        payload = {
            "model": "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
            "prompt": prompt,
            "max_tokens": 100,
            "temperature": 0.7,
            }
        with self.client.post(
            "/v1/completions",
            json=payload,
            catch_response=True,
            ) as response:
            if response.status_code == 200:
                data = response.json()
                tokens = data["usage"]["completion_tokens"]
                response.success()
            else:
                response.failure(f"Status {response.status_code}")

# Run Locust in headless mode with 10 concurrent users
locust -f locustfile.py --headless \
 --users 10 --spawn-rate 2 --run-time 60s \
 --csv results/load_test

# For the web UI (interactive), omit --headless:
# locust -f locustfile.py

import pandas as pd
# Analyze the Locust CSV output
stats = pd.read_csv("results/load_test_stats.csv")
print(stats[["Name", "Request Count", "Median Response Time",
    "95% Response Time", "99% Response Time",
    "Average Response Time", "Requests/s"]].to_string(index=False))
# Compare with the manual baseline
print(f"\nManual baseline median: {statistics.median(latencies):.1f} ms")
print(f"Load test median: {stats['Median Response Time'].iloc[-1]:.1f} ms")
print(f"Latency increase: "
    f"{stats['Median Response Time'].iloc[-1] / statistics.median(latencies):.1f}x")