Build vs. Buy Decision Framework & Total Cost of Ownership

"The cheapest model is the one you do not have to retrain every month."

A Perceptive Compass, Frugally Trained AI Agent

1. The Three Deployment Strategies

Before diving into cost analysis, it helps to define the three primary strategies clearly, since the terminology is often used imprecisely:

API-first (Buy). Use a proprietary model through a provider's API (OpenAI, Anthropic, Google, etc.). You pay per token, have no infrastructure to manage, and get access to frontier-quality models. You also accept the provider's terms of service, data handling policies, rate limits, and the risk that the API changes or disappears.

Self-hosted open-weight (Build-light). Deploy an open-weight model (Llama, Mistral, Qwen, etc.) on your own infrastructure. You control the data pipeline entirely, can fine-tune the model (using the techniques from Chapter 15), and face no per-token charges after the infrastructure is provisioned. You also take on GPU procurement, model serving, scaling, monitoring, and security responsibilities.

Custom training (Build-heavy). Train a model from scratch or perform extensive continued pretraining on domain-specific data. This requires the largest upfront investment but produces a model optimized for your exact use case. The pretraining and scaling considerations from Chapter 06 apply here.

The deployment strategy decision also interacts with the safety and governance requirements from Section 32.5. API-based deployments delegate some compliance burden to the provider (they handle model safety, infrastructure security, data encryption at rest), while self-hosted deployments give you full control but also full responsibility for every layer. For regulated industries, the governance implications of each strategy may outweigh the cost implications.

2. Total Cost of Ownership Model

A proper TCO comparison must account for all cost categories, not just the obvious ones. Organizations that compare only "API price per token" versus "GPU rental cost" miss the majority of the true cost. The following framework captures the full picture:

# Total Cost of Ownership calculator for LLM deployments
from dataclasses import dataclass

@dataclass
class CostComponents:
 """All cost components for a 12-month LLM deployment."""

 # Direct compute costs
 api_costs_monthly: float = 0.0 # API token charges
 gpu_rental_monthly: float = 0.0 # Cloud GPU instances
 gpu_purchase_amortized_monthly: float = 0.0 # On-prem GPU depreciation

 # Engineering costs (monthly, loaded with benefits)
 ml_engineers: int = 0 # Full-time ML engineers
 ml_engineer_monthly_cost: float = 18_000 # Avg loaded cost
 devops_engineers: float = 0 # Fractional DevOps allocation
 devops_monthly_cost: float = 16_000

 # Data and evaluation costs
 data_labeling_monthly: float = 0.0 # Annotation, red teaming
 eval_infrastructure_monthly: float = 0.0 # Benchmark, A/B testing

 # Compliance and security
 compliance_one_time: float = 0.0 # Initial compliance setup
 compliance_ongoing_monthly: float = 0.0 # Ongoing audits, documentation
 security_testing_monthly: float = 0.0 # Red teaming (Section 32.8)

 # Opportunity costs
 time_to_production_months: float = 1.0 # Delay before revenue
 monthly_revenue_opportunity: float = 0.0 # Revenue during delay

 @property
 def monthly_compute(self) -> float:
 return (self.api_costs_monthly +
 self.gpu_rental_monthly +
 self.gpu_purchase_amortized_monthly)

 @property
 def monthly_engineering(self) -> float:
 return (self.ml_engineers * self.ml_engineer_monthly_cost +
 self.devops_engineers * self.devops_monthly_cost)

 @property
 def monthly_data_eval(self) -> float:
 return (self.data_labeling_monthly +
 self.eval_infrastructure_monthly)

 @property
 def monthly_compliance(self) -> float:
 return (self.compliance_ongoing_monthly +
 self.security_testing_monthly)

 @property
 def monthly_total(self) -> float:
 return (self.monthly_compute +
 self.monthly_engineering +
 self.monthly_data_eval +
 self.monthly_compliance)

 @property
 def annual_total(self) -> float:
 return (self.monthly_total * 12 +
 self.compliance_one_time +
 self.opportunity_cost)

 @property
 def opportunity_cost(self) -> float:
 return (self.time_to_production_months *
 self.monthly_revenue_opportunity)

 def breakdown(self) -> dict[str, float]:
 annual = self.annual_total
 if annual == 0:
 return {}
 return {
 "Compute": self.monthly_compute * 12 / annual * 100,
 "Engineering": self.monthly_engineering * 12 / annual * 100,
 "Data & Eval": self.monthly_data_eval * 12 / annual * 100,
 "Compliance": (self.monthly_compliance * 12 +
 self.compliance_one_time) / annual * 100,
 "Opportunity": self.opportunity_cost / annual * 100,
 }

# Scenario comparison: Customer support chatbot at 10M tokens/month
api_strategy = CostComponents(
 api_costs_monthly=3_000, # $3/MTok input, $15/MTok output
 ml_engineers=0, # No ML team needed
 devops_engineers=0.1, # Minimal DevOps
 compliance_one_time=5_000,
 compliance_ongoing_monthly=500,
 time_to_production_months=1,
 monthly_revenue_opportunity=20_000,
)

self_hosted_strategy = CostComponents(
 gpu_rental_monthly=4_500, # 2x A100 on cloud
 ml_engineers=1, # One ML engineer for model ops
 devops_engineers=0.25,
 data_labeling_monthly=2_000, # Ongoing fine-tuning data
 eval_infrastructure_monthly=500,
 compliance_one_time=15_000,
 compliance_ongoing_monthly=1_000,
 security_testing_monthly=500,
 time_to_production_months=3,
 monthly_revenue_opportunity=20_000,
)

print("=== API-First Strategy ===")
print(f"Annual TCO: ${api_{strategy}.annual_{total}:,.0f}")
for category, pct in api_{strategy}.breakdown().items():
 print(f" {category}: {pct:.1f}%")

print(f"\n=== Self-Hosted Strategy ===")
print(f"Annual TCO: ${self_hosted_strategy.annual_total:,.0f}")
for category, pct in self_hosted_strategy.breakdown().items():
 print(f" {category}: {pct:.1f}%")

Code 33.6.1: TCO calculator comparing API-first and self-hosted strategies. The engineering and opportunity costs often dominate the comparison, making the "cheaper GPU" argument misleading when considered in isolation.

3. Hidden Costs That Break TCO Models

The most common TCO analysis failure is underestimating hidden costs. The following are the costs most frequently omitted from initial estimates:

Evaluation infrastructure. You cannot improve what you cannot measure. Building and maintaining evaluation suites (covered in Chapter 29) requires ongoing investment: curated test sets, human evaluation sessions, automated metrics pipelines, and A/B testing infrastructure. Budget 10-20% of your ML engineering time for evaluation.

Data curation and labeling. Fine-tuning requires labeled data. Even with synthetic data generation (covered in Chapter 13), you need human review for quality assurance. A typical fine-tuning dataset of 10,000 examples costs $5,000-$50,000 to produce, depending on domain expertise required. This cost recurs with each model update.

Compliance and security. The EU AI Act compliance requirements from Section 32.9 add documentation, testing, and audit costs. Red teaming from Section 32.8 requires either dedicated staff time or external consultants. These costs apply regardless of whether you use an API or self-host, but self-hosting adds infrastructure security costs.

Model degradation and retraining. Models do not stay accurate forever. Distribution shift (the real world changes but the model's training data is frozen), new product features, and evolving user expectations all require periodic model updates. Budget for retraining every 3-6 months for self-hosted models.

Incident response. When the model produces harmful, incorrect, or embarrassing output (it will happen), someone must investigate, diagnose the root cause, implement a fix, and communicate with affected stakeholders. The observability infrastructure from Section 29.5 reduces investigation time, but incident response remains a significant unplanned cost.

4. Vendor Lock-in Risk Assessment

Vendor lock-in is the risk that switching away from your current provider becomes prohibitively expensive or disruptive. For LLM applications, lock-in occurs at multiple levels:

Model-level lock-in. Your prompts, evaluation suites, and user expectations are calibrated to a specific model's behavior. Switching models requires re-engineering prompts, re-running evaluations, and potentially re-training users. The prompt engineering techniques from Chapter 11 are partially model-specific.

API-level lock-in. Provider-specific features (function calling formats, structured output schemas, vision API interfaces) create code dependencies. The abstraction layers discussed below mitigate this.

Data-level lock-in. If you use a provider's fine-tuning service, your training data and fine-tuned model may be trapped in their ecosystem. Always ensure you retain ownership of training data and can export fine-tuned model weights (or re-create the fine-tune on another platform).

# Multi-provider abstraction layer to reduce lock-in
from abc import ABC, abstractmethod
from dataclasses import dataclass

@dataclass
class LLMResponse:
 """Provider-agnostic response format."""
 content: str
 model: str
 input_tokens: int
 output_tokens: int
 latency_ms: float
 cost_usd: float

class LLMProvider(ABC):
 """Abstract base class for LLM providers."""

 @abstractmethod
 def generate(
 self,
 messages: list[dict],
 temperature: float = 0.7,
 max_tokens: int = 1024,
 ) -> LLMResponse:
 ...

 @abstractmethod
 def supports_tool_calling(self) -> bool:
 ...

class OpenAIProvider(LLMProvider):
 def __init__(self, model: str = "gpt-4o"):
 self.model = model
 self.client = OpenAI()

 def generate(self, messages, temperature=0.7, max_tokens=1024):
 response = self.client.chat.completions.create(
 model=self.model,
 messages=messages,
 temperature=temperature,
 max_tokens=max_tokens,
 )
 choice = response.choices[0]
 usage = response.usage
 return LLMResponse(
 content=choice.message.content,
 model=self.model,
 input_tokens=usage.prompt_tokens,
 output_tokens=usage.completion_tokens,
 latency_ms=response.response_ms,
 cost_usd=self._calculate_cost(usage),
 )

 def supports_tool_calling(self) -> bool:
 return True

 def _calculate_cost(self, usage) -> float:
 # Pricing as of early 2026 (illustrative)
 return (usage.prompt_tokens * 2.50 / 1_000_000 +
 usage.completion_tokens * 10.00 / 1_000_000)

class AnthropicProvider(LLMProvider):
 def __init__(self, model: str = "claude-sonnet-4-20250514"):
 self.model = model
 self.client = Anthropic()

 def generate(self, messages, temperature=0.7, max_tokens=1024):
 response = self.client.messages.create(
 model=self.model,
 messages=messages,
 temperature=temperature,
 max_tokens=max_tokens,
 )
 return LLMResponse(
 content=response.content[0].text,
 model=self.model,
 input_tokens=response.usage.input_tokens,
 output_tokens=response.usage.output_tokens,
 latency_ms=0, # Calculate from request timing
 cost_usd=self._calculate_cost(response.usage),
 )

 def supports_tool_calling(self) -> bool:
 return True

 def _calculate_cost(self, usage) -> float:
 return (usage.input_tokens * 3.00 / 1_000_000 +
 usage.output_tokens * 15.00 / 1_000_000)

class ProviderRouter:
 """Route requests across providers with fallback."""

 def __init__(self, primary: LLMProvider, fallback: LLMProvider):
 self.primary = primary
 self.fallback = fallback

 def generate(self, messages, **kwargs) -> LLMResponse:
 try:
 return self.primary.generate(messages, **kwargs)
 except Exception:
 return self.fallback.generate(messages, **kwargs)

# Usage: switch providers by changing one line
router = ProviderRouter(
 primary=AnthropicProvider(),
 fallback=OpenAIProvider(),
)

Code 33.6.2: Multi-provider abstraction layer. The ProviderRouter enables automatic failover and makes provider switching a configuration change rather than a code rewrite.

5. Decision Framework

The following decision tree synthesizes the factors discussed above into a practical guide. Start at the top and follow the path that matches your situation:

Step 1: Data sensitivity. Does your data contain PII, PHI, classified information, or trade secrets that cannot leave your infrastructure under any circumstances? If no API provider offers acceptable data handling terms (BAAs, SOC 2, data residency), self-hosting is the only option. If a provider offers acceptable terms, proceed to Step 2.

Step 2: Customization requirements. Does your use case require capabilities that general-purpose models lack? If prompt engineering (from Chapter 11) and RAG (from Chapter 20) achieve acceptable quality with an API model, use the API. If you need fine-tuning, check whether the API provider offers a fine-tuning service (OpenAI, Anthropic, and others do). If their fine-tuning meets your needs, use it. If you need full control over the training process, self-host.

Step 3: Volume and latency. At very high volumes (hundreds of millions of tokens per month) or very low latency requirements (sub-100ms TTFT), self-hosting may be more economical or technically necessary. Calculate the crossover point using the TCO model from Section 2.

Step 4: Team capability. Self-hosting requires ML engineering, DevOps, and security expertise. If your team lacks these skills, the cost and risk of building them should be included in the TCO comparison. Hiring an ML team takes 3-6 months; this is 3-6 months of delayed time-to-market.

6. Multi-Vendor Strategies

Many production systems use multiple providers simultaneously, routing requests based on task type, cost constraints, or availability. A multi-vendor strategy provides resilience against outages, negotiating leverage with providers, and the ability to use the best model for each sub-task.

Task-based routing. Use a frontier model (GPT-4o, Claude Sonnet) for complex reasoning tasks and a smaller, cheaper model (GPT-4o-mini, Haiku, Llama 3 8B) for simple classification, extraction, and formatting. The cost savings from routing 70-80% of requests to the cheap model often exceed 50% of the total API bill.

Fallback chains. If the primary provider is unavailable or slow, route to a secondary provider automatically. The ProviderRouter in Code 33.6.2 implements this pattern. Monitor fallback rates; if they exceed 5%, investigate the primary provider's reliability.

A/B testing across providers. Run a fraction of traffic through a challenger model to continuously evaluate whether a cheaper or newer model meets your quality bar. The evaluation infrastructure from Chapter 29 supports this comparison.

7. When to Switch Strategies

The build-vs-buy decision is not permanent. As your volume grows, your requirements evolve, and the provider landscape changes, the optimal strategy shifts. Signals that it is time to re-evaluate:

Switch from API to self-hosted when: Your monthly API bill consistently exceeds the TCO of self-hosting (including engineering), you have accumulated the ML engineering talent to manage infrastructure, your customization needs exceed what API fine-tuning can deliver, or a regulatory change requires data sovereignty you cannot achieve through API providers.

Switch from self-hosted to API when: A new API model significantly outperforms your self-hosted model, your ML engineer leaves and you cannot replace them quickly, the compliance burden of self-hosting (security audits, SOC 2, penetration testing) exceeds the cost savings, or an API provider launches a feature (such as fine-tuning or data residency) that addresses your previous blockers.

Switch from single-vendor to multi-vendor when: You experience your first major outage from your sole provider, your volume gives you negotiating leverage, or different tasks in your pipeline have different quality/cost requirements.

Exercises