Efficient Multi-Tool Orchestration and Tool Economy

"The best tool call is the one you did not make, because the answer was already cached from last time."

Frontier, Cache Savvy AI Agent

1. The Cost Anatomy of a Tool Call

Every tool call in an LLM agent system incurs multiple types of cost, and if you have ever watched your API bill spike because an agent decided to call a search API 47 times to answer a single question, you already have an intuition for why this matters. Understanding this cost structure is essential for optimization.

Token Costs

The most significant and often overlooked cost is token consumption. When an agent has access to tools, the tool schemas (names, descriptions, parameter definitions) must be included in the system prompt or function definitions. For a typical tool with a description and three parameters, the schema consumes 100 to 300 tokens. An agent with 50 tools may spend 5,000 to 15,000 tokens on tool schemas alone, before any conversation begins.

Beyond the schema overhead, each tool call round-trip consumes tokens for: the model's reasoning about which tool to call and with what arguments; the structured tool call output (function name and JSON arguments); the tool's response (which is inserted back into the conversation); and the model's interpretation of the tool response. A single tool call round-trip typically costs 200 to 2,000 tokens depending on the tool and response size.

Latency Costs

Each tool call introduces a round-trip to an external service. For local tools (file reads, calculations), this adds milliseconds. For remote APIs (web search, database queries, third-party services), this adds hundreds of milliseconds to seconds. In a multi-step agent loop, sequential tool calls accumulate latency that can make the system feel unresponsive to users.

Monetary Costs

Some tools have direct per-call costs: search APIs charge per query, database services charge per read, and cloud functions charge per invocation. Even "free" tools have indirect costs through the token consumption they trigger. At scale, these costs compound rapidly.

We can formalize the total cost of a tool call as:

where $T$ represents token counts and $P_{\text{per\_token}}$ is the price per token for the model being used.

2. Token-Efficient Tool Calling Patterns

Several engineering patterns reduce the token cost of tool use without sacrificing capability.

Dynamic Tool Loading

Rather than including all tool schemas in every request, dynamically select a subset of tools relevant to the current task. This can be done with a lightweight classifier that maps the user's query to a set of relevant tools, or by using semantic similarity between the query and tool descriptions.

The following implementation demonstrates a tool router that selects relevant tools based on query similarity.

# Dynamic tool router: selects relevant tools per query using keyword overlap,
# reducing schema token overhead by 70-90% compared to sending all tool
# definitions with every request. Always-include tools bypass the filter.
from dataclasses import dataclass
import numpy as np


@dataclass
class ToolDefinition:
 """A tool available to the agent."""
 name: str
 description: str
 schema: dict
 token_cost: int # Approximate tokens for schema
 avg_response_tokens: int # Average tokens in tool response
 avg_latency_ms: int # Average response latency
 monetary_cost: float # Cost per call in USD


class ToolRouter:
 """Selects relevant tools for each query to minimize schema overhead.

 Instead of sending all 50 tool schemas with every request (costing
 ~10K tokens), the router selects the top-k most relevant tools,
 typically reducing schema tokens by 70-90%.
 """
 def __init__(
 self,
 tools: list[ToolDefinition],
 max_tools: int = 8,
 always_include: list[str] | None = None,
 ):
 self.tools = {t.name: t for t in tools}
 self.max_tools = max_tools
 self.always_include = set(always_include or [])
 # In production, use embedding similarity; here we use keyword overlap
 self._tool_keywords: dict[str, set[str]] = {}
 for tool in tools:
 words = set(tool.description.lower().split())
 self._tool_keywords[tool.name] = words

 def select_tools(self, query: str) -> list[ToolDefinition]:
 """Select the most relevant tools for the given query."""
 query_words = set(query.lower().split())

 # Always-included tools (e.g., "respond_to_user")
 selected = [
 self.tools[name] for name in self.always_include
 if name in self.tools
 ]

 # Score remaining tools by keyword overlap
 candidates = []
 for name, keywords in self._tool_keywords.items():
 if name in self.always_include:
 continue
 overlap = len(query_words & keywords)
 if overlap > 0:
 candidates.append((overlap, name))

 candidates.sort(reverse=True)
 remaining_slots = self.max_tools - len(selected)
 for _, name in candidates[:remaining_slots]:
 selected.append(self.tools[name])

 return selected

 def estimate_schema_savings(self, query: str) -> dict:
 """Estimate token savings from dynamic tool loading."""
 selected = self.select_tools(query)
 all_schema_tokens = sum(t.token_cost for t in self.tools.values())
 selected_schema_tokens = sum(t.token_cost for t in selected)
 return {
 "total_tools": len(self.tools),
 "selected_tools": len(selected),
 "all_schema_tokens": all_schema_tokens,
 "selected_schema_tokens": selected_schema_tokens,
 "tokens_saved": all_schema_tokens - selected_schema_tokens,
 "savings_pct": (
 (all_schema_tokens - selected_schema_tokens) / all_schema_tokens * 100
 if all_schema_tokens > 0 else 0
 ),
 }

Compressed Tool Schemas

Tool descriptions are often verbose, written for human readability rather than token efficiency. Compressing descriptions (removing redundant words, using abbreviations, omitting obvious parameters) can reduce schema tokens by 30 to 50% without affecting the model's ability to select and use tools correctly. Several studies have shown that frontier models are robust to compressed tool descriptions as long as the core semantics are preserved.

Tool Call Batching

When an agent needs to make multiple independent tool calls, batching them into a single step (parallel tool calling) reduces the number of round-trips. Most frontier model APIs support parallel function calling, where the model generates multiple tool calls in a single response. This reduces both latency (calls execute concurrently) and token overhead (the model reasons about all calls together rather than one at a time).

3. Tool Caching Strategies

Many tool calls are repetitive within and across conversations. Caching tool results can eliminate redundant calls, reducing cost and latency.

Result Caching

The simplest caching strategy stores tool results keyed by the tool name and arguments. If the same tool is called with the same arguments within a configurable time window, the cached result is returned without making the actual call. This is effective for tools with deterministic or slowly-changing outputs (database lookups, API calls for static data, calculations).

Semantic Caching

For search-like tools, semantic caching stores results keyed by the semantic meaning of the query rather than the exact query string. If a user asks "What is the weather in Paris?" and then "Paris weather today?", semantic caching recognizes these as equivalent queries and returns the cached result for the second call. This requires embedding the query and checking cosine similarity against cached queries, with a configurable similarity threshold.

The implementation below demonstrates both exact and semantic caching for tool results.

# Tool result caching with exact-match and semantic similarity layers.
# Exact cache uses SHA-256 hashes of (tool_name, args); semantic cache
# falls back to cosine similarity for near-duplicate queries.
import hashlib
import time
from dataclasses import dataclass, field


@dataclass
class CacheEntry:
 """A cached tool result with expiration."""
 result: str
 timestamp: float
 ttl_seconds: float
 hit_count: int = 0

 @property
 def is_expired(self) -> bool:
 return (time.time() - self.timestamp) > self.ttl_seconds


class ToolCache:
 """Cache for tool call results with exact and semantic matching.

 Exact cache: hash(tool_name + json(args)) -> result
 Semantic cache: embedding similarity for search-like queries
 """
 def __init__(self, default_ttl: float = 300.0):
 self.default_ttl = default_ttl
 self.exact_cache: dict[str, CacheEntry] = {}
 self._stats = {"hits": 0, "misses": 0, "savings_usd": 0.0}

 def _make_key(self, tool_name: str, args: dict) -> str:
 """Create a deterministic cache key from tool name and arguments."""
 import json
 raw = f"{tool_name}:{json.dumps(args, sort_keys=True)}"
 return hashlib.sha256(raw.encode()).hexdigest()

 def get(self, tool_name: str, args: dict) -> str | None:
 """Check exact cache for a matching result."""
 key = self._make_key(tool_name, args)
 entry = self.exact_cache.get(key)
 if entry and not entry.is_expired:
 entry.hit_count += 1
 self._stats["hits"] += 1
 return entry.result
 if entry and entry.is_expired:
 del self.exact_cache[key]
 self._stats["misses"] += 1
 return None

 def put(
 self, tool_name: str, args: dict, result: str,
 ttl: float | None = None,
 ):
 """Store a tool result in the cache."""
 key = self._make_key(tool_name, args)
 self.exact_cache[key] = CacheEntry(
 result=result,
 timestamp=time.time(),
 ttl_seconds=ttl or self.default_ttl,
 )

 def evict_expired(self):
 """Remove all expired entries."""
 expired = [
 k for k, v in self.exact_cache.items() if v.is_expired
 ]
 for k in expired:
 del self.exact_cache[k]

 @property
 def hit_rate(self) -> float:
 total = self._stats["hits"] + self._stats["misses"]
 return self._stats["hits"] / total if total > 0 else 0.0


# Usage example
cache = ToolCache(default_ttl=600) # 10-minute TTL

# First call: cache miss, execute tool
result = cache.get("web_search", {"query": "LLM memory architectures"})
if result is None:
 result = "MemGPT, Letta, and retrieval-augmented memory..." # actual tool call
 cache.put("web_search", {"query": "LLM memory architectures"}, result)

# Second identical call: cache hit, no tool execution needed
cached = cache.get("web_search", {"query": "LLM memory architectures"})
print(f"Cache hit rate: {cache.hit_rate:.0%}") # 50% (1 hit, 1 miss)

4. Parallel Execution and Dependency Graphs

When an agent needs to make multiple tool calls, the optimal execution strategy depends on the dependency structure between calls. Independent calls can be executed in parallel; dependent calls must be sequential.

A dependency graph for tool calls is a directed acyclic graph (DAG) where each node is a tool call and edges represent data dependencies (the output of one call is needed as input to another). The optimal execution schedule is the longest path through the DAG (the critical path); all other paths can execute concurrently.

For example, consider a research agent that needs to: (1) search the web for "transformer alternatives 2025," (2) search the web for "state space models benchmarks," (3) read the top result from search 1, (4) read the top result from search 2, and (5) synthesize both readings into a summary. Calls 1 and 2 are independent (parallel). Calls 3 and 4 depend on 1 and 2 respectively. Call 5 depends on both 3 and 4. The critical path is three steps (search, read, synthesize), not five.

Frontier model APIs increasingly support expressing these dependency structures through parallel function calling. The model generates multiple tool calls in a single response, the orchestration layer executes them concurrently, and the results are returned together. This reduces the number of LLM round-trips from the number of tool calls to the depth of the dependency DAG.

5. Economic Models for Tool Use

Treating tool use as an economic decision means that each tool call should be evaluated against its expected benefit. The decision to call a tool should consider: the probability that the tool will provide useful information, the cost of the call (tokens, latency, money), and the cost of not calling the tool (lower answer quality, task failure).

The Tool Call Value Equation

For a given tool call, the expected net value is:

where $P(\text{useful})$ is the probability the tool returns information that improves the answer, $\Delta Q$ is the quality improvement (measured in whatever metric is relevant: accuracy, completeness, user satisfaction), and $C_{\text{total}}$ is the total cost of the call. A rational agent should make the call if and only if $V_{\text{net}} > 0$.

In practice, $P(\text{useful})$ and $\Delta Q$ are difficult to estimate precisely, but rough estimates are often sufficient. For example, if a search tool returns useful results 70% of the time and each useful result improves answer quality by an estimated 20%, the expected quality improvement is $0.7 \times 0.2 = 0.14$. If the cost of the call is $0.01, the call is worthwhile as long as a 14% quality improvement is worth more than one cent, which it almost always is.

Cost-Aware Tool Selection

When multiple tools could provide the needed information, the agent should prefer the cheapest tool that meets the quality threshold. For instance, if the agent needs current stock prices, a cached database lookup ($0.001, 10ms) is preferable to a web search ($0.01, 500ms), even if the web search might return slightly more recent data. The cost difference is 10x and the latency difference is 50x, while the data quality difference is marginal.

This principle extends to choosing between tool use and pure reasoning. If the model can answer a factual question from its training data with 95% accuracy, and a tool call would increase accuracy to 99% at a cost of 500 tokens and 200ms, the decision depends on the application's accuracy requirements and cost sensitivity.

6. Benchmarking Tool Efficiency

Several benchmarks have been developed to evaluate LLM tool use, though most focus on capability (can the agent use the tool correctly?) rather than efficiency (does the agent use the tool economically?).

Existing Benchmarks

• ToolBench (Qin et al., 2024): evaluates tool use across 16,000+ APIs, testing the model's ability to select the right tool, construct correct arguments, and handle multi-tool sequences. Focuses on capability rather than efficiency.
• API-Bank (Li et al., 2023): tests tool use in a conversational setting with 73 APIs, including multi-step tool chains and error recovery. Includes metrics for unnecessary API calls.
• T-Bench (Wang et al., 2024): evaluates tool-augmented reasoning with emphasis on when to use tools versus when to rely on internal knowledge. Includes an "efficiency score" penalizing unnecessary tool calls.
• BFCL (Berkeley Function Calling Leaderboard): focuses on the accuracy of function call generation (correct function name, correct arguments, correct types) across multiple models. Does not measure efficiency.

Efficiency Metrics

A comprehensive tool efficiency evaluation should include:

• Task completion rate: the fraction of tasks completed successfully (the baseline capability metric).
• Tool calls per task: the average number of tool calls made to complete a task. Lower is better, conditional on maintaining task completion rate.
• Tokens per task: total tokens consumed (input + output) including all tool schemas, calls, and responses. This captures the full economic cost.
• Unnecessary call rate: the fraction of tool calls that did not contribute useful information to the final answer (measured by ablating each call and checking if the answer quality changes).
• Parallel utilization: the fraction of independent tool calls that were batched into parallel calls versus executed sequentially.
• Cache hit rate: if caching is enabled, the fraction of tool calls served from cache.

7. Design Patterns for Production Tool Orchestration

Several patterns have emerged for managing tools at scale in production systems.

Pattern 1: Tool Tiers

Organize tools into tiers based on cost and reliability. Tier 1 (local, fast, free): calculations, string manipulation, cached data. Tier 2 (moderate cost, moderate latency): database queries, internal APIs. Tier 3 (expensive, slow): web search, third-party APIs, compute-intensive operations. The agent should prefer lower tiers when possible, escalating to higher tiers only when lower-tier tools are insufficient.

Pattern 2: Speculative Execution

When the model is likely to need a tool result but has not yet decided, speculatively execute the tool call in parallel with the model's reasoning. If the model ends up needing the result, it is already available (zero additional latency). If the model does not need it, the speculative call is discarded. This pattern trades compute cost for latency reduction and is most effective when tool calls are cheap but slow.

Pattern 3: Tool Composition

Rather than exposing many fine-grained tools, compose them into higher-level tools that perform common multi-step operations in a single call. For example, instead of exposing separate "search," "fetch_page," and "extract_text" tools, expose a single "research" tool that performs all three steps internally. This reduces the number of LLM round-trips and eliminates the token overhead of intermediate tool calls.

Pattern 4: Budget-Constrained Execution

Set explicit budgets for tool use per task: maximum number of calls, maximum tokens, maximum latency, maximum monetary cost. The agent must operate within these constraints, prioritizing the most valuable calls when the budget is tight. This prevents runaway costs from agent loops that make excessive tool calls.

8. Future Directions

The field of tool-augmented LLMs is evolving rapidly. Several directions are likely to shape the next generation of tool orchestration systems:

• Learned tool selection. Rather than relying on heuristics or keyword matching for tool routing, training models specifically for tool selection using reinforcement learning from tool outcomes. The model learns which tools are most useful for which types of queries through experience.
• Tool markets. As the ecosystem of available tools grows (MCP servers, API marketplaces, plugin ecosystems), agents will need to discover, evaluate, and select tools from large catalogs at runtime. This creates a "tool economy" where tools compete on quality, cost, and reliability.
• Automated tool creation. Agents that can create their own tools (writing and deploying functions as needed) rather than being limited to a predefined tool set. This is already emerging with code execution tools that allow agents to write arbitrary programs, as demonstrated in Chapter 23.
• Cross-agent tool sharing. In multi-agent systems, one agent's tool results can be shared with other agents, avoiding redundant calls. This requires a shared tool result cache and a protocol for agents to advertise available cached results.