LLM as Feature Extractor

"Why replace the whole engine when you can just upgrade the fuel? Embeddings are the premium octane that makes your trusty XGBoost purr."

Label, Fuel-Injected AI Agent

1. Embeddings as Features

Every LLM (and many smaller language models) can produce embeddings: dense vector representations that encode the semantic meaning of text. These embeddings serve as drop-in replacements for hand-crafted features like TF-IDF or bag-of-words, and they consistently outperform them on tasks that require understanding meaning rather than just matching keywords.

1.1 Why Embeddings Beat TF-IDF

Why this pattern is so powerful: The LLM-as-feature-extractor pattern gives you the best of both worlds: the deep semantic understanding of a large language model combined with the speed, cost, and interpretability of a classical classifier. The key insight is that LLM embeddings are a one-time computation. You pay the LLM cost once to generate the embedding, then the resulting vector can be used for any number of downstream tasks (classification, clustering, similarity search, anomaly detection) at essentially zero marginal cost. This amortization is why embedding-based pipelines can achieve LLM-quality understanding at 1/100th the per-prediction cost of calling an LLM for each query. The embedding approach also connects directly to the vector databases and RAG systems covered in Part 5.

TF-IDF represents text as sparse vectors based on word frequencies. It captures lexical overlap but completely misses semantic similarity. The sentences "The car is fast" and "The automobile has high velocity" share zero TF-IDF features despite being semantically identical. Embeddings from a language model map both sentences to nearby points in a dense vector space, because the model has learned that "car" and "automobile," "fast" and "high velocity" are semantically related. We explore this vector space geometry in more detail in Chapter 19, where embeddings power retrieval systems.

The tradeoff is computational cost. Computing a TF-IDF vector requires only a dictionary lookup and some arithmetic. Computing an embedding requires a forward pass through a neural network. However, this cost is paid only once: you can precompute embeddings for your entire dataset and then use the resulting vectors with any classical model at near-zero marginal cost. Figure 12.2.1 shows this end-to-end pipeline.

1.2 Generating Embeddings with OpenAI

The following snippet demonstrates how to generate embeddings for a batch of text inputs using OpenAI's embedding API, then inspects the output shape and cost. Code Fragment 12.2.4 shows this approach in practice.

# Generate text embeddings via OpenAI's API for downstream ML tasks.
# Batches multiple texts in a single call for efficiency.
import openai
import numpy as np

# Initialize OpenAI client (reads OPENAI_API_KEY from env)
client = openai.OpenAI()

def get_embeddings(texts: list[str], model: str = "text-embedding-3-small") -> np.ndarray:
 """Get embeddings for a batch of texts.

 For large datasets, split into chunks of ~2000 texts
 to stay within the API's per-request token limit.
 """
 response = client.embeddings.create(
 input=texts,
 model=model
 )
 return np.array([item.embedding for item in response.data])

# Example: embed customer support tickets
tickets = [
 "I was charged twice for my subscription last month",
 "The app crashes every time I try to upload a photo",
 "How do I update my billing address?",
 "My package was delivered to the wrong address",
 "Can you explain your enterprise pricing plans?",
]

embeddings = get_embeddings(tickets)

print(f"Embedding shape: {embeddings.shape}")
print(f"Embedding dtype: {embeddings.dtype}")
print(f"First embedding (first 5 dims): {embeddings[0][:5]}")
print(f"\nCost: ~$0.00002 per text (text-embedding-3-small)")
print(f"Total for 5 texts: ~$0.0001")

For local inference without API costs, Code Fragment 12.2.4 uses the SentenceTransformers library to generate embeddings on your own hardware.

# Generate sentence embeddings for semantic similarity comparison
# The bi-encoder produces fixed-size vectors for fast nearest-neighbor search
from sentence_transformers import SentenceTransformer
import numpy as np
import time

# Load a local embedding model
model = SentenceTransformer('all-MiniLM-L6-v2') # 80MB, fast

texts = [
 "I was charged twice for my subscription",
 "The app crashes every time I try to upload a photo",
 "How do I update my billing address?",
 "My package was delivered to the wrong address",
 "Can you explain your enterprise pricing plans?",
]

# Generate embeddings locally
start = time.perf_counter()
embeddings = model.encode(texts, normalize_embeddings=True)
elapsed = time.perf_counter() - start

print(f"Model: all-MiniLM-L6-v2")
print(f"Embedding shape: {embeddings.shape}")
print(f"Time for 5 texts: {elapsed*1000:.1f} ms")
print(f"Avg per text: {elapsed/len(texts)*1000:.1f} ms")
print(f"Cost: $0.00 (local inference)")

# Compute similarity matrix
similarity = np.dot(embeddings, embeddings.T)
print(f"\nSimilarity between 'charged twice' and 'billing address': "
 f"{similarity[0][2]:.3f}")
print(f"Similarity between 'charged twice' and 'app crashes': "
 f"{similarity[0][1]:.3f}")

4. Combining Embeddings with Structured Features

The most powerful pattern combines LLM embeddings (capturing text semantics) with traditional structured features (capturing numeric and categorical data) in a single model. This is especially effective for tasks where both text and metadata carry predictive signal, such as support ticket prioritization, product recommendation, and content moderation. Code Fragment 12.2.4 shows this approach in practice.

# Feature ablation: compare structured-only, embeddings-only, and combined features.
# Demonstrates that combining both sources outperforms either alone.
import numpy as np
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LogisticRegression
import xgboost as xgb

np.random.seed(42)
n_samples = 1000

# Simulate embeddings (384-dim from local model)
text_embeddings = np.random.randn(n_samples, 384) * 0.1

# Simulate structured features
structured_features = np.column_stack([
 np.random.randint(0, 365, n_samples), # account_age_days
 np.random.randint(0, 50, n_samples), # previous_tickets
 np.random.uniform(0, 5000, n_samples), # account_value
 np.random.choice([0, 1], n_samples), # is_premium
 np.random.uniform(0, 1, n_samples), # sentiment_score
])

# Add predictive signal
labels = (
 (structured_features[:, 3] == 1).astype(float) * 0.3 + # premium
 (structured_features[:, 2] > 2500).astype(float) * 0.3 + # high value
 np.random.randn(n_samples) * 0.2
) > 0.3
labels = labels.astype(int)

# Three feature configurations
configs = {
 "Structured only": structured_features,
 "Embeddings only": text_embeddings,
 "Combined (structured + embeddings)": np.hstack([
 StandardScaler().fit_transform(structured_features),
 text_embeddings
 ]),
}

model = xgb.XGBClassifier(
 n_estimators=100, max_depth=4, learning_rate=0.1,
 eval_metric='logloss'
)

print("Feature ablation study (5-fold CV accuracy):")
print("=" * 55)
for name, features in configs.items():
 scores = cross_val_score(model, features, labels, cv=5)
 print(f" {name:40s} {scores.mean():.3f} (+/- {scores.std():.3f})")

4.1 Semantic Caching as a Hybrid Pattern

One of the most cost-effective hybrid patterns combines embeddings with a cache layer to avoid redundant LLM calls. In a typical application, many user queries are semantically equivalent even though they differ in wording: "What is your return policy?" and "How do I return an item?" should produce the same response. Semantic caching intercepts incoming queries, embeds them, and checks whether a sufficiently similar query has been answered recently. If a cached response exists within a configurable similarity threshold, it is returned immediately, bypassing the LLM entirely.

The architecture is straightforward. An embedding model (local or API-based) converts each incoming query into a vector. This vector is compared against a vector index of recent query embeddings using cosine similarity. If the nearest neighbor exceeds the similarity threshold (typically 0.92 to 0.97), the cached response is returned. Otherwise, the query is forwarded to the LLM, and both the query embedding and the response are stored in the cache for future reuse. Code Fragment 12.2.4 shows this in practice.

Code Fragment 12.2.4 defines a reusable class.

# Semantic cache: avoid redundant LLM calls by matching new queries
# against cached embeddings using cosine similarity.
import numpy as np
from openai import OpenAI

client = OpenAI()

class SemanticCache:
 def __init__(self, threshold=0.95, max_entries=10000):
 self.threshold = threshold
 self.max_entries = max_entries
 self.embeddings = [] # List of numpy arrays
 self.responses = [] # Corresponding cached responses

 def _embed(self, text):
 resp = client.embeddings.create(
 model="text-embedding-3-small", input=text
 )
 return np.array(resp.data[0].embedding)

 def get_or_generate(self, query, generate_fn):
 query_vec = self._embed(query)

 # Check cache for semantic match
 if self.embeddings:
 sims = np.dot(np.array(self.embeddings), query_vec)
 best_idx = np.argmax(sims)
 if sims[best_idx] >= self.threshold:
 return self.responses[best_idx] # Cache hit

 # Cache miss: generate and store
 response = generate_fn(query)
 self.embeddings.append(query_vec)
 self.responses.append(response)

 # Evict oldest entries if cache is full
 if len(self.embeddings) > self.max_entries:
 self.embeddings.pop(0)
 self.responses.pop(0)

 return response

The cost savings from semantic caching can be dramatic. In customer support applications where 30% to 50% of queries are paraphrases of common questions, semantic caching reduces LLM API costs proportionally while cutting median response latency from 1 to 2 seconds (LLM generation) to under 50 milliseconds (vector lookup). The embedding cost for the cache lookup is negligible: a single embedding API call costs roughly 1,000x less than a full LLM generation. For even lower latency, a local embedding model like all-MiniLM-L6-v2 can handle the cache lookup in under 5 milliseconds on CPU.

5. Dimensionality Reduction for Embeddings

High-dimensional embeddings (1536 dimensions from OpenAI, 768 or 1024 from many open models) can cause issues with some classical models. Logistic regression may overfit without strong regularization. Tree models may struggle with the high dimensionality. Dimensionality reduction techniques like PCA or UMAP can compress embeddings while preserving most of the information.