The fastest API call is the one you never make. The most private data is the data that never leaves the device.
Quant, Edge-Squeezing AI Agent
Not every LLM inference should travel to the cloud. Privacy constraints, unreliable connectivity, latency requirements, and per-query API costs at scale all create demand for on-device inference. This section establishes the four drivers (privacy, latency, cost, data sovereignty) that decide when edge wins over cloud, the economics that justify a tiered architecture, and the use-case matrix you can use to size your own deployment. The framework landscape that implements these patterns is the subject of Section 60.2, and the hardware constraints that bound them are covered in Section 60.3.
Prerequisites
This section builds on inference optimization and quantization basics from Section 9.1, plus the PEFT and quantization material from Chapter 17. Deployment-architecture patterns are covered in detail later in the book.
60.1.1 The Four Drivers: Privacy, Latency, Cost, Sovereignty
Not every LLM inference should travel to the cloud. When a physician uses an AI assistant in a hospital without reliable internet, when a mobile app needs sub-100ms autocomplete without per-query API costs, or when a defense contractor cannot send data to a third-party server, on-device inference is not a nice-to-have; it is a requirement. Edge deployment moves the model to the user's hardware, eliminating network latency, removing per-query API costs, enabling offline operation, and keeping sensitive data on the device.
The four drivers fall into a clear hierarchy. Privacy is the strongest forcing function: if the regulatory regime (HIPAA, GDPR Article 9, FedRAMP High) or the contract forbids the data from leaving the device, no amount of cloud price reduction makes a cloud architecture legal. Latency is the second: any interaction tighter than a network round-trip (autocomplete, live transcription correction, IME suggestions) cannot tolerate the 50 to 200 ms of WAN time even on a good day, and on a bad day the tail is much worse. Cost is the third: per-query API fees compound at scale, and any workload that runs millions of queries per day eventually finds an on-device tier that pays for itself in months. Data sovereignty is the fourth and increasingly the most strategic: when a competitor or geopolitical adversary operates the frontier model API, the question "what does our usage tell them about our business" becomes a board-level concern.
Apple's on-device language model for iOS 18 runs a 3B-parameter model that fits in 1.5 GB of memory after quantization. It handles autocomplete, message summarization, and notification prioritization without sending a single token to the cloud. Your phone is now running a language model that would have been considered state-of-the-art in 2021, and it does so while you are checking your email in airplane mode.
60.1.2 When Edge Wins vs. Cloud: The Economics
The economics are compelling at scale. An application serving 10 million daily queries at $0.002 per query spends $20,000 per day on API costs. If a quantized 3B-parameter model running on the user's device can handle 80% of those queries with acceptable quality, the savings are substantial. The trade-off is clear: smaller models with lower quality versus larger cloud models with higher quality, and the art of edge deployment is finding the right balance for your use case.
Edge deployment is not about replacing cloud models. The most effective production architectures use a tiered approach: a small on-device model handles simple queries (autocomplete, classification, formatting) with zero latency and zero cost, while complex queries (multi-step reasoning, long-context synthesis) are routed to cloud models. The on-device model also serves as a fallback when the network is unavailable, providing degraded but functional service rather than a blank screen.
The break-even calculation is straightforward. Suppose the on-device tier handles a fraction $f$ of queries at quality $q_e$ (typically 0.7 to 0.9 of the cloud quality), while the cloud tier handles the remaining $1 - f$ at quality $q_c \approx 1$. The blended quality is $q = f \cdot q_e + (1-f) \cdot q_c$ and the blended cost is $C = (1-f) \cdot c_{\text{cloud}}$. At $f = 0.8$ and $q_e = 0.85$ the blended quality is $0.88 \cdot q_c$ at $0.2 \cdot c_{\text{cloud}}$, a 5x cost reduction for a 12% quality discount on the queries that did not need the bigger model. The cost saving compounds with scale; the quality discount does not.
60.1.3 Use Case Matrix
The right edge configuration depends on which of the four drivers dominates and on the hardware the user already owns. The following matrix maps common use cases to their primary driver, the typical model-size envelope that delivers acceptable quality, and the target hardware class.
| Use Case | Primary Driver | Typical Model Size | Target Hardware |
|---|---|---|---|
| Medical records assistant | Privacy (HIPAA) | 3B to 8B | Workstation GPU |
| Mobile keyboard autocomplete | Latency, cost | 0.5B to 1B | Phone NPU/GPU |
| Offline field assistant | No connectivity | 1B to 3B | Laptop CPU |
| Smart home device | Latency, privacy | 0.1B to 0.5B | ARM SoC |
| Enterprise document processing | Data sovereignty | 8B to 70B | On-premise GPU cluster |
The model-size column above is not a style preference; it is the largest model whose working set fits the hardware class. The minimum device tier for a target model follows directly from the on-device working-set sum derived in Section 60.3: a device can host a model only when its usable RAM budget $M_{\text{budget}}$ (physical RAM minus the 2 to 6 GB the OS and foreground app hold) covers the weights, the KV cache for the intended context, and a scratch term, that is $P \cdot \tfrac{b}{8} + 2 L H_{kv} d_{\text{head}} T \cdot 2 + M_{\text{scratch}} \le M_{\text{budget}}$. For the autocomplete row, a phone NPU with a roughly 2 GB usable budget caps the model near 1B at 4-bit; for the medical-records row, a workstation GPU's 24 GB budget is what lets an 8B model run at full context. Read the matrix as the break-even output of that inequality: pick the smallest model that meets the quality bar, then the inequality tells you the cheapest device that can hold it.
- Four drivers push inference to the edge: privacy, latency, cost, and data sovereignty. Privacy is the hardest forcing function; sovereignty is increasingly strategic.
- Edge is rarely a full replacement for cloud. The dominant pattern is a tiered architecture where a small on-device model handles the bulk of queries and the cloud handles the long tail of hard ones.
- Sizing follows the use case: 0.1 to 0.5B on smart-home ARM SoCs, 0.5 to 1B on phones, 1 to 3B on laptops, 3 to 8B on workstations, 8 to 70B on on-premise clusters.
- The economics compound with scale: at millions of queries per day, the on-device tier pays for its development in weeks; the quality discount on the easy queries is small.
With the why and the when of edge deployment established, Section 60.2 surveys the framework landscape that implements these patterns: llama.cpp, Ollama, MLX, ExecuTorch, WebGPU/WebLLM, and Qualcomm AI Hub. Section 60.3 then covers the hardware constraints (battery, thermal, memory) that bound any production deployment.