API Landscape & Architecture

Any sufficiently advanced API is indistinguishable from a junior colleague who types really fast.

Pip, Caffeinated AI Agent

1. The LLM API Ecosystem

Why this matters: Every LLM application, from a simple chatbot to a complex multi-agent system, starts with an API call. Understanding the differences between providers is not academic trivia; it directly determines your cost structure, reliability posture, and how quickly you can ship features. Teams that treat the API layer as an afterthought end up locked into one provider, overpaying for tokens, and scrambling during outages.

The modern LLM API ecosystem is built around a surprisingly simple pattern: you send a sequence of messages (a conversation) to an HTTP endpoint, and the model returns a completion. Under the hood, each provider tokenizes your input, runs inference through a transformer model, and decodes the output tokens back into text. Despite this conceptual simplicity, each provider has evolved distinct conventions, capabilities, and pricing models that shape how you build applications.

The major commercial providers include OpenAI, Anthropic, and Google. Each offers hosted inference with usage-based pricing, typically measured in tokens. Beyond these, cloud platforms like AWS Bedrock and Azure OpenAI provide enterprise wrappers that add compliance, networking, and billing features on top of the same underlying models. For open-source models, serving frameworks like vLLM, TGI, and Ollama expose OpenAI-compatible endpoints, creating a de facto standard. Figure 10.1.1 maps this ecosystem and the relationships between providers.

Figure 10.1.2: The LLM API ecosystem. All major providers share a common HTTP + JSON pattern, but differ in message format, parameter names, and special capabilities.

2. OpenAI Chat Completions API

The OpenAI Chat Completions API established the dominant pattern for LLM APIs. You send a list of messages, each with a role (system, user, or assistant) and content, along with generation parameters. The API returns a completion object containing the model's response, token usage counts, and metadata.

2.1 Core Request Structure

The key parameters that control generation behavior are:

• model: The model identifier (e.g., gpt-4o, o4-mini)
• messages: The conversation history as an array of role/content objects
• temperature: Controls randomness (0.0 = deterministic, 2.0 = maximum randomness), as explored in Chapter 05 on decoding strategies
• top_p: Nucleus sampling threshold (alternative to temperature)
• max_tokens: Maximum number of tokens to generate
• frequency_penalty: Penalizes tokens based on how often they have appeared (range: -2.0 to 2.0)
• presence_penalty: Penalizes tokens that have appeared at all (range: -2.0 to 2.0)

Code Fragment 10.1.2 demonstrates the approach described above.

# Send a chat completion request with system and user messages
# Temperature and max_tokens control randomness and response length
from openai import OpenAI

client = OpenAI() # reads OPENAI_API_KEY from environment

# Send chat completion request to the API
response = client.chat.completions.create(
 model="gpt-4o",
 messages=[
 {"role": "system", "content": "You are a helpful coding assistant."},
 {"role": "user", "content": "Explain Python list comprehensions in two sentences."}
 ],
 temperature=0.7,
 max_tokens=150
)

# Extract the generated message from the API response
print(response.choices[0].message.content)
print(f"Tokens used: {response.usage.prompt_tokens} input, "
 f"{response.usage.completion_tokens} output")

2.2 Streaming Responses with Server-Sent Events

For interactive applications, waiting for the entire response to complete before displaying anything creates a poor user experience. Streaming delivers tokens as they are generated using the Server-Sent Events (SSE) protocol. Each chunk arrives as a small JSON object containing the delta (the new content since the last chunk). Code Fragment 10.1.7 shows this in practice.

# Stream a chat completion using Server-Sent Events
# Each chunk delivers incremental tokens as they are generated
from openai import OpenAI

client = OpenAI()

# Send chat completion request to the API
stream = client.chat.completions.create(
 model="gpt-4o",
 messages=[
 {"role": "user", "content": "Write a haiku about APIs."}
 ],
 stream=True
)

collected_content = ""
for chunk in stream:
 delta = chunk.choices[0].delta
 if delta.content:
 collected_content += delta.content
 print(delta.content, end="", flush=True)

print(f"\n\nFull response: {collected_content}")

2.3 The Batch API

OpenAI's Batch API allows you to submit large collections of requests (up to 50,000) that are processed asynchronously within a 24-hour window. The key advantage is a 50% cost reduction compared to synchronous API calls. To put that in concrete terms: if your evaluation pipeline processes 10,000 prompts at $0.005 per request, the Batch API drops the cost from $50 to $25 for the same results, just delivered within 24 hours instead of immediately. This makes it ideal for tasks like dataset labeling, bulk classification, or evaluation pipelines where real-time responses are unnecessary. Code Fragment 10.1.3 shows this approach in practice.

# Set up the OpenAI client and send a chat completion request
# The response object contains generated text and usage metadata
from openai import OpenAI
import json

client = OpenAI()

# Step 1: Create a JSONL file with batch requests
requests = [
 {
 "custom_id": f"review-{i}",
 "method": "POST",
 "url": "/v1/chat/completions",
 "body": {
 "model": "gpt-4o-mini",
 "messages": [
 {"role": "system", "content": "Classify the sentiment as positive, negative, or neutral."},
 {"role": "user", "content": review}
 ],
 "max_tokens": 10
 }
 }
 for i, review in enumerate(["Great product!", "Terrible service.", "It was okay."])
]

with open("batch_input.jsonl", "w") as f:
 for req in requests:
 f.write(json.dumps(req) + "\n")

# Step 2: Upload and submit the batch
batch_file = client.files.create(file=open("batch_input.jsonl", "rb"), purpose="batch")
batch_job = client.batches.create(input_file_id=batch_file.id, endpoint="/v1/chat/completions",
 completion_window="24h")
print(f"Batch submitted: {batch_job.id}, status: {batch_job.status}")

3. Anthropic Messages API

Anthropic's Messages API follows a similar conversational pattern but introduces several distinctive design choices. System prompts are a separate top-level parameter rather than a message role. The API supports prompt caching (which can reduce costs by up to 90% for repeated prefixes), extended thinking for complex reasoning, and a robust tool use system.

3.1 Core Differences from OpenAI

The most notable architectural differences include:

• System prompt: Passed as a top-level system parameter, not inside the messages array
• Max tokens: Required (not optional), specified as max_tokens
• Content blocks: Responses use content blocks (an array of typed objects) rather than a single string
• Prompt caching: Explicit cache control markers let you cache expensive prefixes
• Extended thinking: Built-in chain-of-thought reasoning that exposes the model's thinking process

Code Fragment 10.1.4 demonstrates the approach described above.

# Call Claude using the Anthropic Messages API
# The response includes input/output token counts for cost tracking
import anthropic

client = anthropic.Anthropic() # reads ANTHROPIC_API_KEY from environment

response = client.messages.create(
 model="claude-sonnet-4-20250514",
 max_tokens=200,
 system="You are a helpful coding assistant. Be concise.",
 messages=[
 {"role": "user", "content": "Explain Python list comprehensions in two sentences."}
 ]
)

print(response.content[0].text)
print(f"Tokens: {response.usage.input_tokens} input, "
 f"{response.usage.output_tokens} output")
print(f"Stop reason: {response.stop_reason}")

3.2 Prompt Caching

Why prompt caching matters: In most production applications, the system prompt is identical across thousands of requests. Without caching, you pay full price for those same system prompt tokens every single time. Prompt caching exploits this repetition: the provider stores the tokenized, processed representation of your prefix, so subsequent requests skip the expensive prefill computation. The intuition is similar to how a web browser caches static assets. The system prompt is your "static asset," and caching prevents you from re-downloading it on every page load.

Anthropic's prompt caching feature is particularly valuable when you repeatedly send requests with a shared prefix (for example, a long system prompt or a large document in context). By marking content blocks with cache_control, you tell the API to cache that prefix. Subsequent requests that share the same cached prefix receive a significant discount on input token costs and reduced latency. Code Fragment 10.1.5 shows this approach in practice.

# Configure and execute the API request
# Adjust parameters based on your use case
response = client.messages.create(
 model="claude-sonnet-4-20250514",
 max_tokens=300,
 system=[
 {
 "type": "text",
 "text": "You are an expert on the Python standard library. " * 50, # Long system prompt
 "cache_control": {"type": "ephemeral"} # Cache this prefix
 }
 ],
 messages=[
 {"role": "user", "content": "What is the difference between os.path and pathlib?"}
 ]
)

# Check cache performance
print(f"Cache creation tokens: {response.usage.cache_creation_input_tokens}")
print(f"Cache read tokens: {response.usage.cache_read_input_tokens}")

4. Google Gemini API

Google's Gemini API uses a generateContent endpoint that shares the same conversational pattern but employs different terminology. Messages are called "contents," roles are "user" and "model" (not "assistant"), and the API supports unique features like grounding (connecting to Google Search), code execution, and native multimodal input. Code Fragment 10.1.7 shows this in practice.

# Call Google Gemini using the genai SDK
# Configuration uses GenerateContentConfig for temperature and token limits
from google import genai

client = genai.Client() # reads GOOGLE_API_KEY from environment

response = client.models.generate_content(
 model="gemini-2.5-flash",
 contents="Explain Python list comprehensions in two sentences.",
 config=genai.types.GenerateContentConfig(
 temperature=0.7,
 max_output_tokens=150,
 system_instruction="You are a helpful coding assistant."
 )
)

print(response.text)
print(f"Tokens: {response.usage_metadata.prompt_token_count} input, "
 f"{response.usage_metadata.candidates_token_count} output")

# LiteLLM: unified interface for 100+ LLM providers.
# Swap provider by changing only the model string prefix.
from litellm import completion

r1 = completion(model="gpt-4o", messages=[{"role": "user", "content": "Hi"}])
r2 = completion(model="anthropic/claude-sonnet-4-20250514", messages=[{"role": "user", "content": "Hi"}])
r3 = completion(model="gemini/gemini-2.5-flash", messages=[{"role": "user", "content": "Hi"}])

5. Enterprise Wrappers: AWS Bedrock and Azure OpenAI

Enterprise cloud providers wrap the underlying models with additional infrastructure for security, compliance, and billing. AWS Bedrock provides access to models from Anthropic, Meta, Mistral, and others through a unified API with IAM authentication, VPC endpoints, and consolidated AWS billing. Azure OpenAI deploys the same OpenAI models within Microsoft's cloud, adding features like content filtering, private networking, and regional data residency.

The following example shows how to call Claude on AWS Bedrock using boto3. The key difference from the direct Anthropic API is authentication (IAM credentials instead of an API key) and the endpoint configuration:

# Call Claude on AWS Bedrock using boto3 with IAM authentication
# The request body follows the Anthropic Messages API format
import boto3
import json

# Bedrock uses IAM credentials (configured via AWS CLI or environment variables)
bedrock = boto3.client(
 service_name="bedrock-runtime",
 region_name="us-east-1"
)

body = json.dumps({
 "anthropic_version": "bedrock-2023-05-31",
 "max_tokens": 256,
 "messages": [
 {"role": "user", "content": "Explain what an API gateway does in two sentences."}
 ]
})

response = bedrock.invoke_model(
 modelId="anthropic.claude-sonnet-4-20250514-v1:0",
 body=body,
 contentType="application/json",
 accept="application/json"
)

result = json.loads(response["body"].read())
print(result["content"][0]["text"])

6. Open-Source Serving: The OpenAI-Compatible Pattern

A powerful convention has emerged in the open-source ecosystem: serving frameworks expose an API that mimics the OpenAI Chat Completions format. This means you can point the OpenAI Python SDK at any compatible server by changing the base_url, and your existing code works without modification. Frameworks like vLLM, Text Generation Inference (TGI), and Ollama all support this pattern. Code Fragment 10.1.5 shows this approach in practice.

# Send a chat completion request to the OpenAI API
# The messages list follows the multi-turn conversation format
from openai import OpenAI

# Point the OpenAI client at a local vLLM server
client = OpenAI(
 base_url="http://localhost:8000/v1",
 api_key="not-needed" # Local servers often skip auth
)

# Send chat completion request to the API
response = client.chat.completions.create(
 model="meta-llama/Llama-3.1-8B-Instruct",
 messages=[
 {"role": "user", "content": "Explain Python list comprehensions in two sentences."}
 ],
 temperature=0.7,
 max_tokens=150
)

# Extract the generated message from the API response
print(response.choices[0].message.content)

7. Provider Comparison

The following table summarizes the key differences across the major providers. Understanding these differences helps you choose the right provider for your use case and design portable abstractions.

Despite the naming differences summarized above, all providers follow the same fundamental lifecycle. Figure 10.1.3 illustrates this universal request/response cycle.

Figure 10.1.5: The universal request/response cycle for LLM APIs. Despite naming differences, every provider follows this same five-step pipeline.

8. Authentication and Rate Limits

Why authentication and rate limiting exist: LLM inference is computationally expensive, and providers need to manage shared GPU resources across thousands of customers. Rate limits prevent any single customer from monopolizing capacity, while authentication ties usage to billing accounts and access policies. Understanding these constraints is essential because your application architecture must be designed around them, not as an afterthought. When you deploy to production, rate limit handling becomes a core reliability concern.

Every provider requires authentication, typically via an API key passed in the Authorization header. Enterprise providers like Azure OpenAI also support Azure Active Directory tokens. Rate limits are enforced at multiple levels: requests per minute (RPM), tokens per minute (TPM), and sometimes requests per day (RPD). Exceeding these limits results in HTTP 429 (Too Many Requests) responses.

Rate limit headers returned with each response tell you how much capacity remains. Monitoring these headers allows you to implement proactive throttling rather than waiting for 429 errors:

• x-ratelimit-remaining-requests: Requests remaining in the current window
• x-ratelimit-remaining-tokens: Tokens remaining in the current window
• x-ratelimit-reset-requests: Time until the request limit resets
• x-ratelimit-reset-tokens: Time until the token limit resets

9. Choosing Between Providers

The right provider depends on your specific requirements. Consider these factors when making your decision:

• Model capability: For the most complex reasoning tasks, compare the latest models from each provider on your specific use case. Benchmarks help, but real evaluation on your data is essential.
• Pricing: Input and output tokens are priced differently. Anthropic and Google tend to offer competitive pricing for large-context workloads. OpenAI's Batch API cuts costs by 50% for non-real-time tasks.
• Context window: Gemini supports up to 1M tokens; Anthropic's Claude supports 200K; OpenAI's GPT-4o supports 128K. Longer contexts enable retrieval-free processing of large documents.
• Compliance: Enterprise wrappers (Bedrock, Azure) offer HIPAA, SOC 2, and data residency guarantees that direct API access may not.
• Latency: Test time-to-first-token and total generation time from your deployment region. Geographic proximity to provider data centers matters.

The following snippet shows how LiteLLM unifies provider calls with a single function. For the full treatment of provider routing, fallback, and cost tracking, see Section 10.3.

# pip install litellm
import litellm

# Same function, different providers (prefix determines the backend)
response = litellm.completion(
 model="anthropic/claude-sonnet-4-20250514",
 messages=[{"role": "user", "content": "Explain caching in one sentence."}],
 max_tokens=60
)
print(response.choices[0].message.content)
print(f"Cost: ${litellm.completion_cost(response):.6f}")