Platforms

"An API key is a six-month commitment to a vendor's roadmap. Choose the vendor, not the model."

Compass, Vendor-Locked-In-But-Honest AI Agent

"Platform" in Part III means something different from Parts I and II. Where Part II's question was "where do I run a 70-billion-parameter open-weights model when I do not own an H100", Part III's question is the inverse: "where do I call a frontier model I will never run myself, and what stack sits between my code and that inference endpoint". The answer is a small ecosystem of managed-inference providers, and the choice among them has gotten more interesting (not less) as the market matured through 2025.

This section maps that ecosystem the way Sections 6.1 and 12.1 mapped theirs: vendor by vendor, what they sell, when to pick them, when not to. The shape of the 2026 market is roughly two tiers: first-party model providers (OpenAI, Anthropic, Google) who own the underlying models, and aggregator / model-router platforms (AWS Bedrock, Azure OpenAI, Together, Replicate, Fireworks) who resell those plus their own. The tier you pick shapes pricing, latency, and most importantly which models you have access to.

14.1.1 First-party model providers

OpenAI: the default for many production teams. USP: GPT-5 family (released 2025) plus the multimodal o-series reasoning models; the broadest tool-use ecosystem (Assistants, structured outputs, file search); the most mature batch API. Pick OpenAI when you want the broadest feature surface and you can tolerate vendor-lock to their tool-use formats. Anti-recommendation: do not pick OpenAI as your only provider; the 2023 and 2024 service incidents made multi-provider abstraction layers mandatory.

Anthropic: the default for long-context and tool-heavy workloads. USP: the Claude 4 family (Opus 4.5, Sonnet 4.5, Haiku 4.5; the Opus class powers this book's authoring stack) plus extended-thinking on Claude 3.7+; the 1M context window; prompt caching with the most aggressive pricing in the market (90% discount on cache hits, see Anthropic prompt caching docs). Pick Anthropic when you have long-context workloads (codebases, document review) or when cache hit rates dominate cost. Anti-recommendation: skip if your workload is high-frequency low-token chat where Haiku is overkill and a smaller open-weights model would be cheaper.

Google AI Studio and Vertex AI: Google's two front doors to Gemini. AI Studio is the developer-friendly free-tier-included entry; Vertex is the GCP-integrated enterprise version. USP: Gemini 2.5 Pro's native multimodality (image, video, audio in one model) and the cheapest frontier-class long-context tokens by a wide margin in 2025. Pick when you do video understanding, when you are already on GCP, or when long-context cost dominates. Anti-recommendation: avoid Vertex if you are not on GCP; the integration surface assumes you are.

14.1.2 The hyperscaler resellers

AWS Bedrock: a model marketplace inside AWS. USP: every major closed-weights provider (Anthropic, Cohere, Meta, AI21, Mistral, plus Amazon Nova) behind one IAM-controlled API, with VPC-private endpoints and full audit logging. Pick Bedrock when you are an AWS-shop enterprise where data residency, SOC2, or HIPAA controls are non-negotiable. Anti-recommendation: Bedrock pricing is a 10 to 20% markup over direct provider pricing; do not use it for cost-optimization, only for compliance.

Azure OpenAI: OpenAI's models hosted in Azure with private endpoints and enterprise SLAs. USP: same models as OpenAI direct but with Azure's compliance and data-residency story; provisioned-throughput-unit (PTU) pricing offers predictable latency for high-volume workloads. Pick if you are a Microsoft-shop with regulatory constraints. Anti-recommendation: do not use Azure OpenAI for experimentation; the deployment-creation flow is slow and the model catalog lags OpenAI direct by weeks.

14.1.3 The independent inference platforms

Together AI: the largest open-weights inference host. USP: serverless inference for hundreds of open-weights models (Llama 4 family, Qwen3, DeepSeek, Mixtral) at competitive per-token pricing, plus dedicated endpoints for production traffic. Pick Together when you want to serve open-weights models without operating GPUs. Anti-recommendation: not the cheapest for very high volume; at that point dedicated GPUs on RunPod or self-hosted vLLM are better.

Replicate: per-request serverless for any model published to its registry. USP: the easiest way to call image, audio, and video models alongside LLMs from one API; the registry is the most diverse in the industry. Pick Replicate for multi-modal applications where you mix-and-match models and accept warm-start latency. Anti-recommendation: avoid for latency-sensitive chat where the cold-start overhead is unacceptable.

Fireworks AI: open-weights specialist focused on throughput and latency optimization. USP: the fastest open-weights inference benchmarks in 2025 (often 2-3x throughput of Together on Llama-class models) and a quality-controlled fine-tuning service. Pick Fireworks when latency or tokens-per-second matter more than model breadth. Anti-recommendation: smaller model catalog than Together; if you need a very long tail of open-weights options, look elsewhere.

Beyond hosted APIs, three desktop / local-inference clients matter in 2026 even for API-first developers, because they make "let me try the open-weights equivalent" cheap. LM Studio (2024) is the dominant local Mac / Windows GUI, with a one-click MLX backend on Apple Silicon and an OpenAI-compatible local server. Ollama is the terminal-friendly equivalent. Msty (2024-25) is the emerging multi-model desktop client with side-by-side chat and provider routing built in. The pattern in 2026 is to ship API-first but keep a local fallback path for development and offline demos.

14.1.4 The platforms compared

14.1.5 Default recommendations

For Part III's exercises: start with the free tiers of OpenAI, Anthropic, and Google AI Studio in parallel, hidden behind LiteLLM. Switch to AWS Bedrock or Azure OpenAI only when compliance forces it. Reach for Together, Replicate, or Fireworks when an open-weights model is the right tool. Never call only one provider in production code; the abstraction layer is mandatory.

Section 14.2 next turns from the inference endpoint to the client library that calls it: SDKs, agent frameworks, prompt-management tools, and the observability surface that makes any of this debuggable in production.

# .env  (NEVER commit this; add to .gitignore)
OPENAI_API_KEY=sk-proj-abc123...
ANTHROPIC_API_KEY=sk-ant-...
HF_TOKEN=hf_...
WANDB_API_KEY=...

# app.py  (this is fine to commit)
import os
from dotenv import load_dotenv

# Load .env into os.environ; safe to call multiple times.
# By default looks for .env in cwd and walks upward.
load_dotenv()

# Then use the standard environment-variable pattern; most SDKs
# auto-pick these up without explicit configuration.
from openai import OpenAI
from anthropic import Anthropic

openai_client = OpenAI()      # reads OPENAI_API_KEY
anthropic_client = Anthropic() # reads ANTHROPIC_API_KEY

# For HuggingFace, the standard env var is HF_TOKEN (also HUGGING_FACE_HUB_TOKEN);
# huggingface_hub picks it up automatically:
from huggingface_hub import login
# login() with no args uses the env var

# Standard auto-detect pattern: load .env, then construct clients with no args.
from dotenv import load_dotenv
from openai import OpenAI
from anthropic import Anthropic

load_dotenv()                       # reads .env into os.environ
openai_client = OpenAI()            # picks up OPENAI_API_KEY
anthropic_client = Anthropic()      # picks up ANTHROPIC_API_KEY

resp = openai_client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "ping"}],
)
print(resp.choices[0].message.content)

import keyring

# Set once (e.g. from an interactive shell or a setup script)
# keyring.set_password("openai", "default", "sk-proj-abc123...")

# Read at runtime
api_key = keyring.get_password("openai", "default")

import os
os.environ["OPENAI_API_KEY"] = api_key  # then standard SDK auto-pickup works

import boto3

# No access key needed; boto3 picks up the IAM role attached to the
# EC2 instance, ECS task, or Lambda function via the metadata service.
s3 = boto3.client("s3")
response = s3.list_buckets()

# Same pattern works for cross-service: an IAM role can have permission
# to read secrets from Secrets Manager, so you fetch an OpenAI key
# without ever embedding an AWS access key.
secrets = boto3.client("secretsmanager")
openai_key = secrets.get_secret_value(SecretId="openai/prod")["SecretString"]
os.environ["OPENAI_API_KEY"] = openai_key

# .github/workflows/eval.yml
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
      - run: pip install -r requirements.txt
      - run: python run_eval.py
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          HF_TOKEN: ${{ secrets.HF_TOKEN }}

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/Yelp/detect-secrets
    rev: v1.5.0
    hooks:
      - id: detect-secrets
        args: [--baseline, .secrets.baseline]
  - repo: https://github.com/gitleaks/gitleaks
    rev: v8.21.0
    hooks:
      - id: gitleaks