Hardware Constraints

"Cloud GPUs are billed by the hour; phone GPUs are billed by the degree Celsius. The first run finishes; the tenth one throttles. Plan for the tenth."

Quant, Edge-Squeezing AI Agent

Fun Fact

Your phone NPU is faster at INT4 than your laptop GPU at FP16

The 2025-era Snapdragon 8 Elite Hexagon NPU pushes about 45 TOPS at INT4, which beats a discrete RTX 3060 (~12 TFLOPS FP16) on quantized 7B-class inference for short prompts. The catch: only if the model fits in 8 GB unified memory, only if you tolerate a smaller context window, and only if you accept the thermal throttling that kicks in after ~30 seconds of sustained generation. Edge AI is faster than you think and shorter-lived than you hope.

60.3.1 Battery Budgets

Running LLM inference on a mobile device introduces constraints that do not exist in server environments. Battery drain is the most visible: sustained LLM inference can consume 3 to 5 watts on a modern smartphone, draining the battery at a rate of roughly 1% per minute of continuous generation. For a 4000 mAh phone battery at 3.85 V (about 15.4 Wh of usable energy), 4 W of inference burns through the entire battery in roughly four hours of continuous use, which is a useful upper bound but not a realistic workload pattern. The realistic worry is shorter: a five-minute chat session that drops the battery by 5%, repeated a dozen times a day, becomes the dominant battery drain on the device.

The tokens-per-watt metric is the right efficiency target for mobile. On a 2026 Snapdragon 8-class SoC running a 3B-parameter Q4_K_M model, GPU inference delivers roughly 30 to 40 tokens per watt, while the Hexagon NPU delivers 80 to 120 tokens per watt on the same workload because it operates at lower precision (int8) and lower voltage. The NPU path is therefore the right default for any sustained-use feature; the GPU path is acceptable for burst use (a single chat turn) but should not host an "always on" feature.

60.3.2 Thermal Throttling

Thermal throttling is equally important; most mobile SoCs reduce clock speeds after 30 to 60 seconds of sustained compute to prevent overheating, which degrades generation speed mid-response. The throttling curve is not a soft taper but a step function in most devices: the SoC monitors a junction-temperature sensor, and when it crosses a vendor-specific threshold (typically 85 to 95 degrees Celsius), the firmware halves the GPU clock, which halves the tokens-per-second of any in-flight generation. A user who starts a generation at 40 tokens per second and finds it slowing to 20 tokens per second mid-response is experiencing thermal throttling, not a model bug.

Five practical mitigations stack to keep generation responsive without melting the device:

1. Use speculative decoding with a tiny draft model to reduce the number of full-model forward passes.
2. Cap generation length to prevent extended inference sessions.
3. Batch requests when possible to amortize model loading overhead.
4. Monitor device temperature and gracefully degrade to shorter responses or cloud fallback when thermal limits approach.
5. Use the smallest model variant that meets quality requirements.

60.3.3 Memory and Quantization Tradeoffs

Memory is the third constraint and the one that decides which models can even be loaded. A 2026 mid-range Android phone ships with 8 GB of RAM; a flagship phone, 12 to 16 GB; a base-model MacBook Air, 16 to 24 GB unified. The OS, the browser, and the foreground app already claim 4 to 6 GB before the LLM ever loads, so the practical model budget on a phone is 2 to 4 GB, and on a laptop 6 to 12 GB. That puts a hard ceiling on the model size: a 3B model in Q4_K_M (about 1.8 GB of weights plus 0.5 GB of KV cache at typical context lengths) fits comfortably on a phone; an 8B Q4_K_M (about 4.6 GB) does not, even before counting the KV cache.

Those two verdicts are not assertions to take on faith; they fall straight out of a working-set sum. The RAM a model needs on-device is three terms added together: the quantized weights, the KV cache for the context you intend to support, and a scratch budget for activations and runtime overhead. With $P$ parameters quantized to $b$ bits each, a context of $T$ tokens, and the same KV-cache shape used throughout this part (reused from the server-side sizing in Section 57.1), the total is

$$M_{\text{total}} = \underbrace{P \cdot \tfrac{b}{8}}_{\text{weights}} \;+\; \underbrace{2 \cdot L \cdot H_{kv} \cdot d_{\text{head}} \cdot T \cdot 2}_{\text{KV cache}} \;+\; M_{\text{scratch}} ,$$

where the weight term converts bits to bytes by dividing by 8, the KV term keeps the fp16 cache at 2 bytes per element (most runtimes do not quantize the cache on-device), and $M_{\text{scratch}}$ covers the activation buffers, the runtime, and per-layer temporaries (200 to 400 MB is typical for llama.cpp-class runtimes). The device must satisfy $M_{\text{total}} \le M_{\text{budget}}$, where the budget is the physical RAM minus what the OS and foreground app already hold (the 2 to 4 GB phone budget stated above).

Work the two models at 4-bit ($b = 4$) and a $T = 4096$ context against a phone budget of $M_{\text{budget}} = 3.5$ GB (a 12 GB flagship with roughly 8 GB already claimed by the OS, the launcher, and a foreground app). The 3B model (Phi-3.5-class shape: $L = 32$, $H_{kv} = 8$, $d_{\text{head}} = 96$) gives weights of $3 \times 10^9 \cdot 0.5 = 1.5$ GB, a KV cache of $2 \cdot 32 \cdot 8 \cdot 96 \cdot 4096 \cdot 2 \approx 0.40$ GB, and $M_{\text{scratch}} \approx 0.3$ GB, for $M_{\text{total}} \approx 2.2$ GB. That clears the 3.5 GB budget with margin, so the 3B fits. The 8B model (Llama-3.1-8B shape: $L = 32$, $H_{kv} = 8$, $d_{\text{head}} = 128$) gives weights of $8 \times 10^9 \cdot 0.5 = 4.0$ GB on their own, a KV cache of $2 \cdot 32 \cdot 8 \cdot 128 \cdot 4096 \cdot 2 \approx 0.54$ GB, and the same 0.3 GB scratch, for $M_{\text{total}} \approx 4.8$ GB. The weights alone exceed the budget, and the full working set overshoots it by 37%, so the 8B does not fit. Table 60.3.2 lays the two sums side by side.

The sum also explains the two levers that change the verdict. Dropping to a more aggressive quantization shrinks only the weight term: an 8B at 3-bit would weigh $8 \times 10^9 \cdot 0.375 = 3.0$ GB, bringing the total to about 3.8 GB, still over the 3.5 GB budget, which is why sub-4-bit 8B deployment usually waits for a 16 GB device rather than a 12 GB one. Shrinking the context shrinks only the KV term, which is small here but dominates at long context: the same 8B at a 32K context would carry a 4.3 GB KV cache on top of the weights, putting the working set above 8 GB and out of reach of any current phone. Context length, not just parameter count, is a first-class sizing variable on-device.

The quality difference between a Q4_K_M and Q5_K_M quantization on a 3B model is often imperceptible to users, but the memory and power savings can extend battery life by 15 to 20%.

Canonical "Fits on a Phone" Reference Models for 2026

Three model families dominate the on-device "useful, fits, runs at acceptable speed" envelope as of 2026:

• Phi-3.5-mini (Microsoft, ~3.8B parameters): the successor to Phi-3-mini that retains the same "trained on heavily curated synthetic data" recipe and the same 3.8B-parameter footprint. At Q4_K_M it is about 2.3 GB, fits on every flagship phone with margin, and benchmarks at GPT-3.5-class quality. It is the default reference for chat-class on-device workloads.
• Gemma 3 1B / 4B (Google, March 2025): the current Gemma generation, which superseded the Gemma 2 2B that long held this slot. The 1B variant is roughly half the size of Phi-3.5 with a permissive license; the 4B variant adds vision and a 128K context window. The right pick when you need the model to share RAM with a heavy foreground app, or when you target mid-range phones with 6 GB total RAM.
• Apple Foundation Models (Apple, ~3B parameters): the on-device model that ships in iOS 18.x and powers Apple Intelligence's writing tools, message summarization, and Genmoji. It is not directly callable from third-party apps, but the Foundation Models framework exposes it through guided generation APIs in iOS 18.1+. Apple's design notes (announced at WWDC 2024 and detailed in subsequent Apple ML Research posts) describe an architecture where the on-device model handles short interactive tasks and a privacy-preserving Private Cloud Compute tier handles longer prompts on Apple Silicon servers running attested code. Together they define the privacy-preserving fallback pattern that the rest of the industry is converging on.

# Ensure Ollama is installed and running
ollama --version

# Pull two quantization levels of the same model
ollama pull llama3.2:3b-instruct-q4_K_M
ollama pull llama3.2:3b-instruct-q8_0

"""Benchmark two quantization levels on the same prompts."""

import time
import json
from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

PROMPTS = [
 "Explain the concept of quantization in neural networks in 3 sentences.",
 "Write a Python function that computes the Fibonacci sequence iteratively.",
 "Summarize the key differences between TCP and UDP protocols.",
 "What are the main causes of the French Revolution? List 5 factors.",
 "Translate this to formal English: 'gonna grab some food brb'",
 "Write a SQL query to find the top 10 customers by total order value.",
 "Explain photosynthesis to a 10-year-old in simple terms.",
 "What are three common logical fallacies? Give an example of each.",
 "Write a bash one-liner to count the number of .py files recursively.",
 "Compare and contrast microservices and monolithic architectures.",
]

MODELS = ["llama3.2:3b-instruct-q4_K_M", "llama3.2:3b-instruct-q8_0"]

def benchmark_model(model_name: str, prompts: list[str]) -> dict:
 results = []
 for prompt in prompts:
     start = time.perf_counter()
     response = client.chat.completions.create(
         model=model_name,
         messages=[{"role": "user", "content": prompt}],
         max_tokens=300,
         temperature=0.0, # Deterministic for comparison
     )
     elapsed = time.perf_counter() - start
     output = response.choices[0].message.content
     token_count = response.usage.completion_tokens

     results.append({
         "prompt": prompt[:60],
         "output": output,
         "tokens": token_count,
         "time_s": round(elapsed, 2),
         "tok_per_s": round(token_count / elapsed, 1),
     })

     avg_tps = sum(r["tok_per_s"] for r in results) / len(results)
     return {
         "model": model_name,
         "avg_tokens_per_sec": avg_tps,
         "results": results,
     }

# Run benchmarks
for model in MODELS:
 print(f"\nBenchmarking {model}...")
 report = benchmark_model(model, PROMPTS)
 print(f" Average: {report['avg_tokens_per_sec']:.1f} tokens/sec")
 with open(f"benchmark_{model.replace(':', '_')}.json", "w") as f:
     json.dump(report, f, indent=2)

git clone https://github.com/ggerganov/llama.cpp && cd llama.cpp && make LLAMA_METAL=1