Workflow Orchestration and Durable Execution

A workflow that cannot survive a server restart is not a workflow. It is a prayer.

Anonymous infrastructure engineer, after a 3 AM incident

1. Why LLM Agents Need Durable Execution

Consider an agent that performs a 20-step research workflow: it searches the web, reads documents, extracts structured data, calls multiple LLM providers, writes intermediate summaries, and produces a final report. If the process crashes at step 18, a naive implementation loses all prior work. The user waits again, pays again, and hopes the second run completes without interruption. In production, this is unacceptable.

The problem deepens when you consider the types of failures that plague LLM systems. Provider rate limits cause temporary 429 errors. Network timeouts interrupt streaming responses mid-token. Context windows overflow when an agent accumulates too much history. Provider outages take entire model endpoints offline for minutes or hours. Memory pressure on worker nodes causes out-of-memory kills. Deployments roll forward, terminating in-flight requests. Each of these failures is transient, meaning the operation would succeed if retried, but only if the system remembers where it left off.

Durable execution solves this by persisting the state of a workflow at each significant step. When a failure occurs, the system resumes from the last completed checkpoint rather than restarting from scratch. The workflow function itself looks like ordinary sequential code, but the runtime guarantees that each step executes exactly once, even across process restarts, machine failures, or deployment rollouts. This guarantee, sometimes called "exactly-once semantics," transforms the reliability story for long-running agent pipelines.

Figure 31.6.1: Durable execution recovery. When a workflow crashes at step 3, the durable execution framework replays steps 1 and 2 from recorded history (no network calls) and resumes execution at the failure point. The recovered run is indistinguishable from an uninterrupted run.

2. Temporal: Infrastructure-Level Durability

Temporal is an open-source durable execution platform originally developed at Uber (as Cadence) and now maintained by Temporal Technologies. It provides two core abstractions: workflows and activities. A workflow is a function that must be deterministic: given the same inputs and the same history of activity results, it must make the same decisions. An activity is a function that performs non-deterministic work: calling an LLM, querying a database, sending an email, or invoking a tool.

The Temporal server persists every activity result in an event history. When a worker crashes and restarts, the workflow function replays from the beginning, but instead of re-executing activities, it reads their recorded results from history. The replay is fast (no network calls) and produces the same execution state as the original run. Workers are stateless processes that poll task queues for work; you can scale them horizontally and restart them at will without losing workflow progress.

For LLM agents, the deterministic/non-deterministic split maps cleanly onto the workflow/activity boundary. The agent's control loop (deciding which tool to call next, routing based on LLM output, aggregating results) lives in the workflow. Each LLM call, tool invocation, and external API request is wrapped as an activity with its own retry policy, timeout, and heartbeat interval.

import asyncio
from datetime import timedelta
from temporalio import activity, workflow
from temporalio.client import Client
from temporalio.worker import Worker
from dataclasses import dataclass

@dataclass
class ResearchRequest:
 topic: str
 max_sources: int = 5
 budget_usd: float = 2.0

@dataclass
class ResearchResult:
 summary: str
 sources: list[str]
 total_cost: float

# --- Activities: non-deterministic operations ---

@activity.defn
async def search_web(query: str) -> list[str]:
 """Search the web and return relevant URLs."""
 # Calls an external search API (non-deterministic)
 results = await search_client.search(query, max_results=10)
 return [r.url for r in results]

@activity.defn
async def extract_content(url: str) -> str:
 """Fetch and extract main content from a URL."""
 activity.heartbeat() # Signal liveness during long downloads
 return await content_extractor.extract(url)

@activity.defn
async def llm_summarize(texts: list[str], topic: str) -> str:
 """Call an LLM to synthesize a research summary."""
 response = await llm_client.chat.completions.create(
 model="gpt-4o",
 messages=[
 {"role": "system", "content": "Summarize the following sources."},
 {"role": "user", "content": f"Topic: {topic}\n\nSources:\n" +
 "\n---\n".join(texts)},
 ],
 )
 return response.choices[0].message.content

# --- Workflow: deterministic orchestration ---

@workflow.defn
class ResearchWorkflow:
 """A durable research agent that survives crashes at any step."""

 @workflow.run
 async def run(self, request: ResearchRequest) -> ResearchResult:
 # Step 1: Search (retries up to 3 times on failure)
 urls = await workflow.execute_activity(
 search_web,
 request.topic,
 start_to_close_timeout=timedelta(seconds=30),
 retry_policy=RetryPolicy(maximum_attempts=3),
 )

 # Step 2: Extract content from each source (parallel)
 contents = []
 for url in urls[: request.max_sources]:
 content = await workflow.execute_activity(
 extract_content,
 url,
 start_to_close_timeout=timedelta(seconds=60),
 heartbeat_timeout=timedelta(seconds=15),
 retry_policy=RetryPolicy(maximum_attempts=2),
 )
 contents.append(content)

 # Step 3: Synthesize with LLM
 summary = await workflow.execute_activity(
 llm_summarize,
 args=[contents, request.topic],
 start_to_close_timeout=timedelta(seconds=120),
 retry_policy=RetryPolicy(
 maximum_attempts=5,
 backoff_coefficient=2.0,
 initial_interval=timedelta(seconds=2),
 ),
 )

 return ResearchResult(
 summary=summary,
 sources=urls[: request.max_sources],
 total_cost=0.0, # Tracked via gateway
 )

2.1 The OpenAI Agents SDK Integration

The OpenAI Agents SDK (formerly the Swarm framework) provides an activity_as_tool helper that bridges Temporal activities into the SDK's tool-calling interface. This lets you define durable, retryable operations as Temporal activities and expose them to an agent as callable tools. The agent's LLM decides which tools to invoke, and Temporal guarantees that each invocation is persisted, retried on failure, and never duplicated.

from temporalio import activity, workflow
from agents import Agent, Runner
from agents.extensions.temporal import activity_as_tool

# Define activities as durable tool implementations
@activity.defn
async def query_database(sql: str) -> str:
 """Execute a read-only SQL query and return results as JSON."""
 result = await db_pool.fetch(sql)
 return json.dumps([dict(row) for row in result])

@activity.defn
async def send_notification(channel: str, message: str) -> str:
 """Send a Slack notification. Side effect: must not be duplicated."""
 await slack_client.post_message(channel=channel, text=message)
 return f"Notification sent to {channel}"

# Wrap activities as agent tools inside a workflow
@workflow.defn
class AnalystAgentWorkflow:
 @workflow.run
 async def run(self, user_query: str) -> str:
 # Temporal activities become agent tools
 db_tool = activity_as_tool(
 query_database,
 start_to_close_timeout=timedelta(seconds=30),
 )
 notify_tool = activity_as_tool(
 send_notification,
 start_to_close_timeout=timedelta(seconds=10),
 )

 agent = Agent(
 name="data-analyst",
 instructions="You analyze data and report findings.",
 tools=[db_tool, notify_tool],
 )

 result = await Runner.run(agent, user_query)
 return result.final_output

3. Inngest: Event-Driven Durable Functions

Inngest takes a different approach to durable execution. Instead of a separate server and worker infrastructure, Inngest provides a lightweight SDK that runs inside your existing web application (FastAPI, Next.js, Flask). Functions are triggered by events and broken into steps using the step.run() API. Each step is automatically checkpointed, retried on failure, and observable through Inngest's built-in dashboard.

The event-driven model fits naturally into systems where multiple loosely coupled services react to the same events. When a user uploads a document, one function extracts text, another generates embeddings, and a third triggers a summarization agent. Each function runs independently, with its own retry logic and step-level checkpointing. This is choreography (services react to events) rather than orchestration (a central coordinator directs services). Choreography excels when the workflow is a fan-out of independent tasks; orchestration excels when tasks have complex dependencies and conditional branching.

import inngest

client = inngest.Inngest(app_id="llm-pipeline")

@client.create_function(
 fn_id="research-agent",
 trigger=inngest.TriggerEvent(event="research/requested"),
 retries=3,
)
async def research_agent(ctx: inngest.Context, step: inngest.Step):
 topic = ctx.event.data["topic"]

 # Step 1: Search (checkpointed, retried independently)
 urls = await step.run("search-web", lambda: search_web(topic))

 # Step 2: Extract content in parallel fan-out
 contents = []
 for i, url in enumerate(urls[:5]):
 content = await step.run(
 f"extract-{i}",
 lambda u=url: extract_content(u),
 )
 contents.append(content)

 # Step 3: Summarize with LLM (separate retry budget)
 summary = await step.run(
 "llm-summarize",
 lambda: llm_summarize(contents, topic),
 )

 # Step 4: Emit completion event for downstream consumers
 await step.send_event(
 "research/completed",
 data={"topic": topic, "summary": summary},
 )

 return {"summary": summary, "sources": urls[:5]}

4. LangGraph Persistence: Application-Level Checkpointing

LangGraph, the agent framework from LangChain, provides its own persistence layer through checkpointers. After each node in a LangGraph state graph executes, the full graph state is serialized and stored. This enables several powerful capabilities: resuming interrupted agents, time-travel debugging (inspecting the state at any historical node), and human-in-the-loop patterns where the graph pauses at a designated node, waits for human approval, and then continues.

LangGraph checkpointing operates at the application level: the framework manages state persistence as part of its graph execution engine. This contrasts with Temporal, which operates at the infrastructure level: the Temporal server manages durability for any code running inside workflows, regardless of the application framework. The distinction matters for operational complexity. LangGraph persistence requires only a database (PostgreSQL, SQLite, or Redis); Temporal requires a dedicated server cluster with its own database, metrics, and operational runbooks.

from langgraph.graph import StateGraph, START, END
from langgraph.checkpoint.postgres import PostgresSaver
from typing import TypedDict, Annotated
from langgraph.graph.message import add_messages

class ResearchState(TypedDict):
 messages: Annotated[list, add_messages]
 sources: list[str]
 summary: str
 status: str

def search_node(state: ResearchState) -> dict:
 """Search for sources. State is checkpointed after this node."""
 urls = web_search(state["messages"][-1].content)
 return {"sources": urls, "status": "searched"}

def summarize_node(state: ResearchState) -> dict:
 """Summarize sources with an LLM call."""
 response = llm.invoke(
 f"Summarize these sources about the topic:\n" +
 "\n".join(state["sources"])
 )
 return {"summary": response.content, "status": "summarized"}

def review_node(state: ResearchState) -> dict:
 """Human-in-the-loop review point. Graph pauses here."""
 return {"status": "awaiting_review"}

# Build the graph with checkpointing
checkpointer = PostgresSaver.from_conn_string(DATABASE_URL)

graph = StateGraph(ResearchState)
graph.add_node("search", search_node)
graph.add_node("summarize", summarize_node)
graph.add_node("review", review_node)

graph.add_edge(START, "search")
graph.add_edge("search", "summarize")
graph.add_edge("summarize", "review")
graph.add_edge("review", END)

# Compile with checkpointer and interrupt_before for HITL
app = graph.compile(
 checkpointer=checkpointer,
 interrupt_before=["review"], # Pause before human review
)

# Start execution (pauses at review node)
config = {"configurable": {"thread_id": "research-42"}}
result = app.invoke(
 {"messages": [("user", "Research quantum error correction")]},
 config=config,
)
# result["status"] == "summarized"; graph is paused

# Later: human approves, resume from checkpoint
app.invoke(None, config=config) # Continues from review node

The comparison between LangGraph and Temporal persistence comes down to scope and operational overhead. LangGraph checkpointing is tightly integrated with its graph abstraction, making it simple to add persistence to any LangGraph agent. However, it only covers LangGraph workflows; if your system includes non-LangGraph components (data pipelines, email senders, payment processors), those need their own durability story. Temporal provides a universal durability layer for any code, but requires deploying and operating a separate infrastructure component. For teams already running Kubernetes, Temporal's operational cost is manageable. For teams that want minimal infrastructure, LangGraph's built-in checkpointing (or Inngest's managed service) offers a lower barrier to entry.

Figure 31.6.2: Comparing the three major durable execution frameworks. Temporal provides infrastructure-level guarantees with a separate server cluster. Inngest offers event-driven durability inside your existing web application. LangGraph provides application-level checkpointing tightly integrated with its graph execution engine.

5. Retry and Recovery Strategies

Retries are the first line of defense against transient failures, but naive retry logic can cause more harm than good. Retrying too aggressively amplifies load on an already stressed provider. Retrying non-idempotent operations creates duplicate side effects. Retrying indefinitely wastes budget on requests that will never succeed. A production retry strategy must address all three concerns.

Figure 31.6.3: Retry strategy progression. Immediate retry handles network blips but worsens rate limits. Exponential backoff gives providers recovery time but causes thundering herd problems. Jittered backoff adds randomness to prevent synchronized retries and is the recommended default for LLM API calls.

5.1 Retry Taxonomy

Three retry patterns appear in production LLM systems. Immediate retry re-sends the request with no delay; this works for transient network blips but worsens rate-limit situations. Exponential backoff doubles the delay between each attempt (1s, 2s, 4s, 8s), giving the provider time to recover. Jittered backoff adds randomness to the delay (e.g., 1s +/- 0.3s) to prevent the "thundering herd" problem where many clients retry simultaneously after a shared outage. Jittered exponential backoff is the recommended default for LLM API calls.

import random
import asyncio
from typing import TypeVar, Callable, Awaitable

T = TypeVar("T")

async def retry_with_budget(
 fn: Callable[..., Awaitable[T]],
 *args,
 max_attempts: int = 5,
 initial_delay: float = 1.0,
 backoff_factor: float = 2.0,
 jitter: float = 0.3,
 max_cost_usd: float = 1.0,
 cost_tracker: "CostTracker | None" = None,
 **kwargs,
) -> T:
 """Retry with jittered exponential backoff and budget awareness.

 Stops retrying if cumulative cost exceeds the budget threshold,
 preventing runaway spend on requests that consistently fail
 after consuming tokens (e.g., partial streaming responses).
 """
 delay = initial_delay
 last_exception = None

 for attempt in range(1, max_attempts + 1):
 # Budget gate: stop if we have already spent too much
 if cost_tracker and cost_tracker.total_cost > max_cost_usd:
 raise BudgetExceededError(
 f"Retry budget exhausted: ${cost_tracker.total_cost:.4f} "
 f"exceeds ${max_cost_usd:.2f} limit after {attempt - 1} attempts"
 )

 try:
 return await fn(*args, **kwargs)
 except RateLimitError:
 last_exception = e
 # Respect Retry-After header if present
 delay = max(delay, get_retry_after(e))
 except (TimeoutError, ConnectionError) as e:
 last_exception = e
 except ContextWindowOverflowError as e:
 # Not transient: retrying the same input will always fail
 raise

 if attempt < max_attempts:
 jittered_delay = delay * (1 + random.uniform(-jitter, jitter))
 await asyncio.sleep(jittered_delay)
 delay *= backoff_factor

 raise MaxRetriesExceededError(
 f"Failed after {max_attempts} attempts"
 ) from last_exception

5.2 Idempotency in Agent Steps

An operation is idempotent if executing it multiple times produces the same result as executing it once. LLM completions are naturally idempotent (at temperature 0): calling the same model with the same prompt returns the same output and charges the same cost. But many agent actions are not idempotent: sending an email, creating a database record, charging a credit card, or posting to an API. When a durable execution framework retries or replays a step, non-idempotent actions risk duplication.

The standard solution is an idempotency key: a unique identifier attached to each operation. The receiving system checks whether it has already processed a request with that key and returns the cached result if so. For agent workflows, generate the idempotency key from the workflow ID and step number, ensuring that replays produce the same key and are correctly deduplicated.

Figure 31.6.4: The saga (compensation) pattern for multi-step workflows with side effects. Each step that produces a side effect registers an undo action. When a later step fails, compensations execute in reverse order to restore a clean state.

5.3 Compensation Logic for Side Effects

When a multi-step workflow fails partway through, some steps may have already produced side effects that need to be undone. If an agent books a flight in step 3 but fails to book a hotel in step 5, the flight booking may need to be cancelled. This pattern is called compensation (or the "saga pattern" in distributed systems). Each step that produces a side effect registers a compensation function. If the workflow fails, the framework executes compensations in reverse order, cleaning up partial work.

@workflow.defn
class BookingWorkflow:
 """Saga pattern: each step registers a compensation action."""

 @workflow.run
 async def run(self, request: TravelRequest) -> BookingResult:
 compensations: list[tuple[str, dict]] = []

 try:
 # Step 1: Book flight
 flight = await workflow.execute_activity(
 book_flight, request.flight_details,
 start_to_close_timeout=timedelta(seconds=60),
 )
 compensations.append(("cancel_flight", {"id": flight.id}))

 # Step 2: Book hotel
 hotel = await workflow.execute_activity(
 book_hotel, request.hotel_details,
 start_to_close_timeout=timedelta(seconds=60),
 )
 compensations.append(("cancel_hotel", {"id": hotel.id}))

 # Step 3: LLM generates itinerary summary
 itinerary = await workflow.execute_activity(
 generate_itinerary,
 args=[flight, hotel, request.preferences],
 start_to_close_timeout=timedelta(seconds=120),
 )

 return BookingResult(
 flight=flight, hotel=hotel, itinerary=itinerary
 )

 except Exception as e:
 # Execute compensations in reverse order
 for comp_name, comp_args in reversed(compensations):
 await workflow.execute_activity(
 comp_name, comp_args,
 start_to_close_timeout=timedelta(seconds=30),
 )
 raise WorkflowFailedError(f"Booking failed: {e}") from e

6. Observability for Long-Running Workflows

Long-running agent workflows (minutes to hours) require observability patterns beyond what short-lived request/response APIs need. The observability practices from Chapter 30 provide the foundation; this section extends them to durable workflows.

Distributed tracing connects every step in a workflow into a single trace. When a Temporal workflow calls five activities across three workers, or when an Inngest function fans out into parallel steps, the trace ID propagates through all operations. This lets you visualize the entire workflow timeline in tools like Jaeger, Grafana Tempo, or Datadog, identifying bottlenecks (which activity took 45 seconds?) and failure points (which retry finally succeeded?). Temporal provides built-in integration with OpenTelemetry; Inngest and LangGraph offer trace export through their observability APIs.

Stalled workflow detection is essential for catching agents that are stuck rather than failed. A workflow might be waiting for an activity that silently hangs (the LLM provider accepts the request but never responds). Heartbeat timeouts address this at the activity level: if an activity does not call heartbeat() within the configured interval, the framework considers it failed and schedules a retry. At the workflow level, set a "workflow execution timeout" that caps the total wall-clock time for the entire workflow, preventing indefinite hangs.

Cost-per-execution tracking aggregates token usage and provider costs across all steps in a single workflow execution. This metric is critical for understanding the economics of autonomous agents. A research agent might consume $0.12 per execution on average, but outliers (where the agent enters a retry loop or explores an unusually large search space) might cost $5.00 or more. Tracking cost distribution per workflow type, combined with the gateway-level budget enforcement from Section 31.5, enables both real-time cost control and historical cost analysis.

7. Choosing a Framework

The three frameworks covered in this section serve different needs. Temporal is the right choice when you need infrastructure-level durability for complex, long-running workflows with strict exactly-once guarantees. It excels in environments that already run Kubernetes and have platform engineering teams comfortable with operating stateful infrastructure. The OpenAI Agents SDK integration makes it particularly appealing for teams building tool-heavy agents that interact with external systems.

Inngest is the right choice when you want durable execution without operating additional infrastructure. Its event-driven model and step-level checkpointing provide strong durability guarantees with minimal setup. It fits teams that prefer managed services and want to add durability to existing web applications rather than building a separate worker infrastructure.

LangGraph persistence is the right choice when you are already using LangGraph for agent orchestration and want to add checkpointing, time-travel debugging, and human-in-the-loop capabilities with minimal additional complexity. Its durability guarantees are scoped to the LangGraph execution engine, making it simpler but less general than Temporal.

These frameworks are not mutually exclusive. A common production architecture uses LangGraph for agent logic, wrapped inside a Temporal workflow that provides infrastructure-level durability, with an Inngest event bus connecting the workflow to other services. The key principle is to match the durability guarantee to the cost of failure: lightweight summarization tasks may need only LangGraph checkpointing, while a booking agent that charges credit cards needs Temporal's exactly-once semantics.

Exercises