Edge and On-Device LLM Deployment

The fastest API call is the one you never make. The most private data is the data that never leaves the device.

Every engineer who has waited for a cloud inference round trip on a spotty connection

1. Why Edge Deployment Matters

Not every LLM inference should travel to the cloud. When a physician uses an AI assistant in a hospital without reliable internet, when a mobile app needs sub-100ms autocomplete without per-query API costs, or when a defense contractor cannot send data to a third-party server, on-device inference is not a nice-to-have; it is a requirement. Edge deployment moves the model to the user's hardware, eliminating network latency, removing per-query API costs, enabling offline operation, and keeping sensitive data on the device.

The economics are compelling at scale. An application serving 10 million daily queries at $0.002 per query spends $20,000 per day on API costs. If a quantized 3B-parameter model running on the user's device can handle 80% of those queries with acceptable quality, the savings are substantial. The trade-off is clear: smaller models with lower quality versus larger cloud models with higher quality, and the art of edge deployment is finding the right balance for your use case.

Use Case Matrix

2. llama.cpp: Universal C/C++ Inference

llama.cpp, created by Georgi Gerganov, is the foundational project for running LLMs on consumer hardware. Written in pure C/C++ with no Python dependencies, it compiles and runs on virtually any platform: Linux, macOS, Windows, Android, iOS, and even Raspberry Pi. The project introduced the GGUF (GPT-Generated Unified Format) quantization format, which has become the standard for distributing quantized models. llama.cpp supports dozens of model architectures (Llama, Mistral, Phi, Qwen, Gemma, and many others) and provides both a CLI interface and a built-in HTTP server compatible with the OpenAI API format.

# Build llama.cpp from source
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON # For NVIDIA GPUs; omit for CPU-only
cmake --build build --config Release -j $(nproc)

# Download a GGUF model (example: Llama 3.2 3B at Q4_K_M quantization)
# Models are available on Hugging Face in GGUF format
wget https://huggingface.co/bartowski/Llama-3.2-3B-Instruct-GGUF/resolve/main/Llama-3.2-3B-Instruct-Q4_K_M.gguf

# Run interactive chat
./build/bin/llama-cli \
 -m Llama-3.2-3B-Instruct-Q4_K_M.gguf \
 --chat-template llama3 \
 -c 4096 \
 -ngl 99 # Offload all layers to GPU

# Start an OpenAI-compatible API server
./build/bin/llama-server \
 -m Llama-3.2-3B-Instruct-Q4_K_M.gguf \
 --host 0.0.0.0 --port 8080 \
 -c 4096 -ngl 99

GGUF Quantization Levels

GGUF models come in various quantization levels that trade quality for memory and speed. The naming convention encodes the bit width and quantization method. Understanding these trade-offs is essential for choosing the right model variant for your hardware constraints.

3. Ollama: Developer-Friendly Local Model Management

Ollama wraps llama.cpp (and other backends) in a user-friendly interface inspired by Docker. Instead of downloading GGUF files manually and managing command-line flags, Ollama provides a pull/run workflow that handles model downloads, GPU detection, memory management, and API serving automatically. It supports macOS, Linux, and Windows, and exposes an OpenAI-compatible API by default on port 11434.

# Install Ollama (macOS/Linux)
curl -fsSL https://ollama.com/install.sh | sh

# Pull and run a model
ollama pull llama3.2:3b
ollama run llama3.2:3b "Explain edge deployment in one paragraph."

# List available models
ollama list

# Run with a specific quantization
ollama pull llama3.2:3b-instruct-q5_K_M
ollama run llama3.2:3b-instruct-q5_K_M

Custom Modelfiles

Ollama's Modelfile system allows you to create custom model configurations with specific system prompts, parameters, and templates. This is useful for packaging a fine-tuned or customized model as a reusable unit that team members can pull and run identically.

# Modelfile: a custom medical assistant configuration
FROM llama3.2:3b-instruct-q5_K_M

PARAMETER temperature 0.3
PARAMETER top_p 0.9
PARAMETER num_ctx 4096
PARAMETER stop "<|eot_id|>"

SYSTEM """You are a medical terminology assistant running on a hospital
workstation. You help clinicians look up drug interactions, medical
terminology, and clinical guidelines. You always include a disclaimer
that your outputs are for reference only and do not constitute medical
advice. All data stays on this device; no information is sent externally."""

TEMPLATE """{{ if .System }}<|start_header_id|>system<|end_header_id|>

{{ .System }}<|eot_id|>{{ end }}{{ if .Prompt }}<|start_header_id|>user<|end_header_id|>

{{ .Prompt }}<|eot_id|>{{ end }}<|start_header_id|>assistant<|end_header_id|>

{{ .Response }}<|eot_id|>"""

# Build and run the custom model
ollama create medical-assistant -f Modelfile
ollama run medical-assistant "What are the contraindications for metformin?"

Programmatic Access

This snippet shows how to query benchmark results programmatically through the API.

"""Using Ollama's API from Python (OpenAI-compatible endpoint)."""

from openai import OpenAI

# Ollama exposes an OpenAI-compatible API on localhost:11434
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

response = client.chat.completions.create(
 model="llama3.2:3b",
 messages=[
 {"role": "system", "content": "You are a helpful assistant."},
 {"role": "user", "content": "What is quantization in LLMs?"},
 ],
 temperature=0.7,
 max_tokens=500,
)
print(response.choices[0].message.content)

# Streaming works identically to the OpenAI API
stream = client.chat.completions.create(
 model="llama3.2:3b",
 messages=[{"role": "user", "content": "Explain edge deployment."}],
 stream=True,
)
for chunk in stream:
 if chunk.choices[0].delta.content:
 print(chunk.choices[0].delta.content, end="", flush=True)

4. MLX: Optimized Inference on Apple Silicon

Apple's MLX framework is designed specifically for Apple Silicon (M1 through M4 chips), exploiting the unified memory architecture that allows the CPU, GPU, and Neural Engine to share the same memory without copying. For Mac users, MLX often delivers better performance than llama.cpp because it uses Metal shaders optimized for Apple's GPU architecture. The companion library mlx-lm provides a high-level interface for text generation with Hugging Face model compatibility.

# Install MLX and mlx-lm
pip install mlx mlx-lm

# Run a model directly from Hugging Face
mlx_lm.generate \
 --model mlx-community/Llama-3.2-3B-Instruct-4bit \
 --prompt "Explain the benefits of on-device inference." \
 --max-tokens 200

# Convert a Hugging Face model to MLX format with quantization
mlx_lm.convert \
 --hf-path meta-llama/Llama-3.2-3B-Instruct \
 --mlx-path ./llama-3.2-3b-mlx-4bit \
 --quantize \
 --q-bits 4 \
 --q-group-size 64

"""MLX text generation with streaming."""

from mlx_lm import load, generate

# Load a quantized model (downloads from HF if not cached)
model, tokenizer = load("mlx-community/Llama-3.2-3B-Instruct-4bit")

# Generate text
prompt = tokenizer.apply_chat_template(
 [{"role": "user", "content": "What are the advantages of MLX?"}],
 tokenize=False,
 add_generation_prompt=True,
)

response = generate(
 model,
 tokenizer,
 prompt=prompt,
 max_tokens=300,
 temp=0.7,
 verbose=True, # Prints tokens/sec performance
)
print(response)

MLX's unified memory model means that a MacBook Pro with 36GB of RAM can run a 4-bit quantized 30B model entirely in memory without the CPU-to-GPU transfer bottleneck that limits performance on discrete GPU systems. For development and prototyping workflows on Apple hardware, MLX provides the fastest path from model selection to running inference.

5. ExecuTorch: PyTorch Models on Mobile and Edge

Meta's ExecuTorch is the production runtime for deploying PyTorch models on mobile phones, IoT devices, and other resource-constrained hardware. Unlike llama.cpp (which requires models in GGUF format) or MLX (which targets Apple Silicon), ExecuTorch takes standard PyTorch models and exports them to an optimized format (.pte) that runs on Android, iOS, and embedded Linux with hardware-specific acceleration. ExecuTorch supports Qualcomm Hexagon DSPs, Apple Core ML, MediaTek APUs, and ARM CPU backends.

"""Export a model for ExecuTorch deployment (simplified workflow)."""

import torch
from executorch.exir import to_edge
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load a small model suitable for mobile
model_name = "microsoft/phi-2" # 2.7B parameters
model = AutoModelForCausalLM.from_pretrained(
 model_name, torch_dtype=torch.float16
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Trace the model for export
example_input = tokenizer("Hello", return_tensors="pt")
traced = torch.export.export(
 model,
 (example_input["input_ids"],),
 dynamic_shapes={
 "input_ids": {1: torch.export.Dim("seq_len", min=1, max=512)}
 },
)

# Convert to ExecuTorch edge format
edge_program = to_edge(traced)

# Delegate to hardware-specific backends
# For Qualcomm: edge_program = edge_program.to_backend(QnnBackend())
# For CoreML: edge_program = edge_program.to_backend(CoreMLBackend())

# Export the final .pte file
et_program = edge_program.to_executorch()
with open("phi2_mobile.pte", "wb") as f:
 f.write(et_program.buffer)

print("Exported model size:", len(et_program.buffer) / 1e6, "MB")

ExecuTorch's primary advantage over llama.cpp for mobile deployment is its integration with the PyTorch ecosystem. If your model is already in PyTorch (as most research models are), ExecuTorch provides a direct export path without format conversion. It also supports hardware-specific optimizations through a delegate system, where computation-heavy operations are offloaded to specialized accelerators (NPUs, DSPs) that llama.cpp cannot access.

6. Battery and Thermal Constraints on Mobile

Running LLM inference on a mobile device introduces constraints that do not exist in server environments. Battery drain is the most visible: sustained LLM inference can consume 3 to 5 watts on a modern smartphone, draining the battery at a rate of roughly 1% per minute of continuous generation. Thermal throttling is equally important; most mobile SoCs reduce clock speeds after 30 to 60 seconds of sustained compute to prevent overheating, which degrades generation speed mid-response.

Practical mitigations include: (1) using speculative decoding with a tiny draft model to reduce the number of full-model forward passes, (2) capping generation length to prevent extended inference sessions, (3) batching requests when possible to amortize model loading overhead, (4) monitoring device temperature and gracefully degrading to shorter responses or cloud fallback when thermal limits approach, and (5) using the smallest model variant that meets quality requirements. The quality difference between a Q4_K_M and Q5_K_M quantization on a 3B model is often imperceptible to users, but the memory and power savings can extend battery life by 15 to 20%.

Lab: Quantization Quality vs. Latency Benchmark

# Ensure Ollama is installed and running
ollama --version

# Pull two quantization levels of the same model
ollama pull llama3.2:3b-instruct-q4_K_M
ollama pull llama3.2:3b-instruct-q8_0

"""Benchmark two quantization levels on the same prompts."""

import time
import json
from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

PROMPTS = [
 "Explain the concept of quantization in neural networks in 3 sentences.",
 "Write a Python function that computes the Fibonacci sequence iteratively.",
 "Summarize the key differences between TCP and UDP protocols.",
 "What are the main causes of the French Revolution? List 5 factors.",
 "Translate this to formal English: 'gonna grab some food brb'",
 "Write a SQL query to find the top 10 customers by total order value.",
 "Explain photosynthesis to a 10-year-old in simple terms.",
 "What are three common logical fallacies? Give an example of each.",
 "Write a bash one-liner to count the number of .py files recursively.",
 "Compare and contrast microservices and monolithic architectures.",
]

MODELS = ["llama3.2:3b-instruct-q4_K_M", "llama3.2:3b-instruct-q8_0"]

def benchmark_model(model_name: str, prompts: list[str]) -> dict:
 results = []
 for prompt in prompts:
 start = time.perf_counter()
 response = client.chat.completions.create(
 model=model_name,
 messages=[{"role": "user", "content": prompt}],
 max_tokens=300,
 temperature=0.0, # Deterministic for comparison
 )
 elapsed = time.perf_counter() - start
 output = response.choices[0].message.content
 token_count = response.usage.completion_tokens

 results.append({
 "prompt": prompt[:60],
 "output": output,
 "tokens": token_count,
 "time_s": round(elapsed, 2),
 "tok_per_s": round(token_count / elapsed, 1),
 })

 avg_tps = sum(r["tok_per_s"] for r in results) / len(results)
 return {
 "model": model_name,
 "avg_tokens_per_sec": avg_tps,
 "results": results,
 }

# Run benchmarks
for model in MODELS:
 print(f"\nBenchmarking {model}...")
 report = benchmark_model(model, PROMPTS)
 print(f" Average: {report['avg_tokens_per_sec']:.1f} tokens/sec")
 with open(f"benchmark_{model.replace(':', '_')}.json", "w") as f:
 json.dump(report, f, indent=2)