When to Use LLM vs. Classical ML

"When all you have is a 175-billion-parameter hammer, everything looks like a nail. Sometimes the nail is actually a screw, and a logistic regression is the screwdriver."

Label, Hammer-Skeptical AI Agent

1. The Decision Framework

Choosing between an LLM and a classical ML model is not a binary decision. It is a multi-dimensional optimization across four axes: accuracy, latency, cost, and interpretability. The right choice depends on the specific requirements of your production system, the volume of requests you need to handle, and the tolerance your users have for errors, delays, and unexplainable outputs.

1.1 The Four Decision Axes

Each axis carries different weight depending on the application. A fraud detection system prioritizes accuracy and interpretability (regulators demand explanations). A customer chatbot prioritizes latency and naturalness. A batch document processing pipeline prioritizes cost per document. Understanding which axes matter most for your use case is the first step in making a good decision.

• Accuracy: How correct does the system need to be? Is 95% acceptable, or do you need 99.9%? Does the task have a clear ground truth, or is quality subjective?
• Latency: What is the acceptable response time? Real-time applications need sub-100ms responses. Interactive applications tolerate 1 to 3 seconds. Batch processing has no latency constraint.
• Cost: What is the per-query cost at your expected volume? A $0.01 per query cost is negligible at 100 queries per day but devastating at 10 million queries per day.
• Interpretability: Do you need to explain individual predictions? Regulatory requirements, debugging needs, and user trust all factor into this axis.
• Data Privacy: Can you send data to an external API? Organizations subject to GDPR, HIPAA, or financial regulations may be prohibited from transmitting sensitive data to third-party LLM providers. This constraint often tips the decision toward classical models or locally hosted open-source LLMs. Always verify your data governance policy before choosing an API-based approach.

Figure 12.1.1 summarizes this decision framework, showing how data structure, complexity, and volume guide the choice between classical ML, LLM, and hybrid approaches.

1.2 When Classical ML Wins

Classical machine learning models dominate when the data is structured, the task is well-defined, labeled data is available, and latency or cost constraints are tight. Here are the primary scenarios where classical ML is the better choice:

• Tabular prediction: For structured data with numeric and categorical features (credit scoring, churn prediction, demand forecasting), gradient-boosted trees (XGBoost, LightGBM) consistently outperform LLMs. LLMs cannot natively process tabular data without serializing it to text, which introduces noise and destroys the efficient representations that tree models exploit.
• High-volume classification with labeled data: When you have thousands of labeled examples for a text classification task, a TF-IDF plus logistic regression pipeline achieves competitive accuracy at a fraction of the cost. At 10 million queries per day, the cost difference between a $0.00001 per query classical model and a $0.01 per query LLM is the difference between $100 and $100,000 per day.
• Latency-critical applications: Classical models run inference in microseconds to low milliseconds on CPU. LLMs require tens of milliseconds to seconds depending on output length. For real-time bidding, fraud detection, or recommendation ranking where sub-10ms latency is required, classical ML is the only viable option.
• Deterministic extraction: When patterns are regular and well-defined (phone numbers, email addresses, dates, currency amounts), regex and rule-based systems are faster, cheaper, and more reliable than LLMs. They never hallucinate a phone number that does not exist in the input text.

1.3 When LLMs Win

LLMs excel in scenarios that require understanding natural language semantics, handling ambiguity, generalizing from few examples, or producing fluent text output. The pretraining and scaling that goes into these models is what gives them such broad generalization:

• Zero-shot and few-shot tasks: When labeled data is scarce or unavailable, LLMs can perform classification, extraction, and summarization using only a task description and a handful of examples in the prompt (see Section 11.1 for few-shot prompting techniques).
• Complex reasoning over text: Tasks that require multi-step reasoning, combining information from different parts of a document, or understanding implicit context are natural fits for LLMs.
• Open-ended generation: Producing emails, summaries, creative text, or conversational responses requires the generative capabilities of LLMs, which rely on the decoding strategies covered earlier. Classical models cannot generate coherent paragraphs.
• Ambiguous or subjective tasks: Sentiment analysis with sarcasm, intent detection with ambiguous phrasing, and content moderation with nuanced policy violations all benefit from the broad world knowledge and contextual understanding of LLMs.

2. Empirical Benchmarks: Classification

Why benchmarking matters more than intuition. Many teams skip benchmarking and make the LLM-vs-classical decision based on assumptions. A common mistake is assuming an LLM will always outperform a classical model on text tasks. In practice, a TF-IDF + logistic regression classifier trained on 5,000 labeled examples frequently matches or beats zero-shot LLM classification, at 1/10,000th the cost per prediction. Conversely, teams sometimes avoid LLMs for tasks where a few-shot prompt would outperform a classical model trained on a small, noisy dataset. The only way to know is to test both approaches on your actual data using consistent evaluation metrics.

The best way to make the decision is to benchmark. Let us compare four approaches on a common text classification task: classifying customer support tickets into categories (billing, technical, account, shipping, general). Code Fragment 12.1.3 shows this approach in practice.

2.1 TF-IDF + Logistic Regression Baseline

This snippet trains a TF-IDF plus logistic regression baseline for text classification.

# Train a TF-IDF + Logistic Regression baseline classifier
# This lightweight pipeline runs in milliseconds and costs nothing per prediction
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report
import time
import numpy as np

# Sample labeled data (in production: thousands of examples)
texts = [
 "I was charged twice for my subscription",
 "My app keeps crashing on login",
 "How do I change my email address",
 "Package hasn't arrived after 10 days",
 "What are your business hours",
 "Refund not showing in my account",
 "Error 500 when uploading files",
 "Reset my password please",
 "Tracking number not working",
 "Do you offer student discounts",
] * 100 # Simulate larger dataset

labels = [
 "billing", "technical", "account", "shipping", "general",
 "billing", "technical", "account", "shipping", "general",
] * 100

X_train, X_test, y_train, y_test = train_test_split(
 texts, labels, test_size=0.2, random_state=42
)

# Train TF-IDF + Logistic Regression
vectorizer = TfidfVectorizer(max_features=5000, ngram_range=(1, 2))
X_train_tfidf = vectorizer.fit_transform(X_train)
X_test_tfidf = vectorizer.transform(X_test)

model = LogisticRegression(max_iter=1000, C=1.0)
model.fit(X_train_tfidf, y_train)

# Benchmark inference
start = time.perf_counter()
for _ in range(1000):
 vectorizer.transform(["I was charged twice"])
 model.predict(vectorizer.transform(["I was charged twice"]))
elapsed = time.perf_counter() - start

print(f"Accuracy: {model.score(X_test_tfidf, y_test):.3f}")
print(f"Avg latency: {elapsed / 1000 * 1000:.2f} ms")
print(f"Cost per query: ~$0.000001 (CPU inference)")

# Zero-shot classification with Hugging Face: no labeled training data needed.
# A natural language inference model scores text against arbitrary candidate labels.
from transformers import pipeline

classifier = pipeline("zero-shot-classification", model="facebook/bart-large-mnli")

result = classifier(
 "My app keeps crashing on login",
 candidate_labels=["billing", "technical", "account", "shipping", "general"]
)

print(result["labels"][0]) # "technical"
print(result["scores"][0]) # 0.87 (confidence)

# Batch classification
texts = ["I was charged twice", "Package hasn't arrived", "Reset my password"]
results = classifier(texts, candidate_labels=["billing", "technical", "account", "shipping"])
for text, r in zip(texts, results):
 print(f"{text:40s} -> {r['labels'][0]} ({r['scores'][0]:.2f})")

For structured extraction tasks, regular expressions offer even faster, deterministic results. The following snippet demonstrates regex-based entity extraction for common patterns such as emails, phone numbers, and monetary amounts.

# Build a regex-based extractor for structured patterns like dates and emails
# Pattern matching handles deterministic extraction with zero inference cost
import re
import time

text = """
Contact us at support@example.com or call +1 (555) 123-4567.
Invoice total: $1,234.56 due by 2025-03-15.
Reach Jane Smith at jane.smith@company.org for details.
"""

# Regex extraction: deterministic, fast, perfect for regular patterns
patterns = {
 "emails": r'[\w.+-]+@[\w-]+\.[\w.-]+',
 "phones": r'\+?1?\s*\(?\d{3}\)?[\s.-]?\d{3}[\s.-]?\d{4}',
 "amounts": r'\$[\d,]+\.\d{2}',
 "dates": r'\d{4}-\d{2}-\d{2}',
}

start = time.perf_{counter}()
for _ in range(10000):
 results = {k: re.findall(v, text) for k, v in patterns.items()}
elapsed = time.perf_{counter}() - start

for entity_{type}, matches in results.items():
 print(f" {entity_{type}}: {matches}")

print(f"\nAvg latency: {elapsed / 10000 * 1000:.4f} ms")
print(f"False positives: 0 (deterministic)")
print(f"Cost: $0.00 (no API call)")

5. Cost Modeling at Scale

The per-query cost difference between approaches becomes dramatic at scale. A cost model must account for not just API or inference costs, but also the engineering time to build and maintain each approach, infrastructure costs for self-hosted models, and the opportunity cost of latency.

6. The Decision Matrix

Pulling together all the factors above, here is a practical decision matrix. For any new ML task, walk through these questions in order:

1. Is the pattern regular and well-defined? Use regex or rules. Do not overthink it.
2. Is the data tabular (structured rows and columns)? Use XGBoost or LightGBM. LLMs are not competitive on tabular data.
3. Do you have thousands of labeled examples? Train a classical model (TF-IDF + LR for text, BERT for complex text). It will be cheaper and faster.
4. Is the task zero-shot or few-shot? Use an LLM. It is the only option that works without labeled data.
5. Does the task require generation or complex reasoning? Use an LLM. Classical models cannot generate coherent text or reason over long documents.
6. Is volume extremely high (millions per day) and cost matters? Consider fine-tuning a smaller model (BERT or a small LLM) to replace the large LLM, or use a hybrid approach where a classifier handles the easy cases.
7. None of the above clearly applies? Start with an LLM for rapid prototyping, then evaluate whether a cheaper model can match its performance once you have collected enough labeled data from the LLM's outputs.