Foundational Prompt Design

The quality of a question determines the quality of an answer. This has always been true; now we have machines to prove it at scale.

Prompt, Socratic AI Agent

12.1.1 The Anatomy of a Prompt

Before diving into specific techniques, it helps to understand the structural components that every prompt can contain. A well-designed prompt typically includes some combination of the following elements: an instruction that tells the model what to do, context that provides background information, input data that the model should process, output format specification that constrains the response shape, and examples that demonstrate the desired behavior. Not every prompt needs all five components, but being intentional about which components to include (and which to omit) is the first step toward reliable results.

In the chat API format used by most modern LLMs, these components are distributed across different message roles. The system message typically carries the instruction, context, and format specification. The user message carries the input data. And when using few-shot examples, pairs of user and assistant messages demonstrate the expected behavior before the actual query arrives. Figure 12.1.2 maps these structural components to their corresponding chat API roles.

Figure 12.1.2a: Anatomy of a prompt mapped to chat API roles. The five structural components (instruction, context, output format, examples, input data) are distributed across three message roles. The right panel shows how each card becomes one entry in the messages[] array sent to the model. Color carries the mapping: purple = system, blue = few-shot example pairs, green = the actual user query, orange = the model's response.

12.1.2 Zero-Shot Prompting

Zero-shot prompting is the simplest approach: you give the model an instruction and input with no examples. The model relies entirely on its pretrained knowledge to produce the answer. This works surprisingly well for tasks the model has seen during training, such as summarization, translation, classification of common categories, and question answering. The key to effective zero-shot prompting is specificity. Vague instructions produce vague results.

12.1.2.1 The Specificity Principle

Consider the difference between a vague prompt and a specific one for the same task:

Code Fragment 12.1.2b contrasts a vague prompt with a constrained one, showing how specificity transforms the same task from ambiguous to precise.

# Comparing vague vs. specific zero-shot prompts
# Specificity in instructions directly controls output quality
# Vague zero-shot prompt
vague_prompt = "Summarize this article."
# Specific zero-shot prompt
specific_prompt = """Summarize the following article in exactly 3 bullet points.
Each bullet point should be one sentence of at most 20 words.
Focus on the key findings, not background information.
Use present tense throughout.
Article:
{article_text}"""

The specific prompt constrains the output length (3 bullet points), format (one sentence each), content focus (key findings), and style (present tense). These constraints dramatically reduce the space of possible outputs, making the model far more likely to produce exactly what you need. This principle applies universally: the more precisely you define success, the more reliably the model achieves it.

12.1.2.2 Zero-Shot Classification Example

Code Fragment 12.1.4 illustrates a chat completion call.

# Zero-shot sentiment classification via the chat completions API.
# The system prompt defines the task; no examples are provided.
import openai
# Initialize OpenAI client (reads OPENAI_API_KEY from env)
client = openai.OpenAI()
def classify_sentiment(text: str) -> str:
    # Send chat completion request to the API
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
        {"role": "system",
        "content": """Classify the sentiment of the given text.
        Respond with exactly one word: positive, negative, or neutral.
        Do not include any explanation or punctuation."""},
        {"role": "user",
        "content": text}
        ],
        temperature=0.0,
        max_tokens=5
        )
    # Extract the generated message from the API response
    return response.choices[0].message.content.strip().lower()
    # Test examples
    texts = [
        "This product exceeded all my expectations!",
        "The delivery was late and the item arrived damaged.",
        "The package arrived on Tuesday as scheduled."
        ]
    for t in texts:
        print(f"{t[:50]:50s} => {classify_sentiment(t)}")

12.1.3 Few-Shot Prompting

Few-shot prompting provides examples of the desired input-output mapping before the actual query. This technique was popularized by the GPT-3 paper (Brown et al., 2020), which showed that providing as few as two or three examples can dramatically improve performance on tasks the model has not been explicitly fine-tuned for. Few-shot examples serve multiple purposes: they demonstrate the expected output format, clarify ambiguous instructions, establish the decision boundary for classification tasks, and prime the model's internal representations toward the target behavior.

12.1.3.1 Designing Effective Few-Shot Examples

Not all examples are equally useful. Research and practice have identified several principles for selecting few-shot examples:

• Cover the label space: Include at least one example per category. If you have three sentiment labels, show at least one positive, one negative, and one neutral example.
• Include edge cases: Show examples near the decision boundary. A mildly negative review is more informative than an obviously negative one.
• Match the distribution: If 80% of your real inputs are neutral, your examples should reflect that proportion (or slightly oversample rare classes).
• Keep formatting consistent: Every example should follow exactly the same structure. Inconsistency in formatting confuses the model.
• Order matters: Recent research shows that example order affects performance. Placing the most relevant example last (closest to the query) often helps.

12.1.3.2 Few-Shot Entity Extraction

Code Fragment 12.1.2d illustrates a chat completion call.

# Few-shot entity extraction with edge-case examples
# Demonstrates how to include examples with empty result fields
import openai, json
client = openai.OpenAI()
def extract_entities(text: str) -> dict:
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
        {"role": "system",
        "content": """Extract named entities from the text.
        Return a JSON object with keys: persons, organizations, locations.
        Each value is a list of strings. If no entities are found for a
        category, return an empty list."""},
        # Few-shot example 1
        {"role": "user",
        "content": "Tim Cook announced the new iPhone at Apple Park in Cupertino."},
        {"role": "assistant",
        "content": '{"persons": ["Tim Cook"], "organizations": ["Apple"], "locations": ["Apple Park", "Cupertino"]}'},
        # Few-shot example 2 (edge case: no persons)
        {"role": "user",
        "content": "The European Central Bank raised interest rates in Frankfurt."},
        {"role": "assistant",
        "content": '{"persons": [], "organizations": ["European Central Bank"], "locations": ["Frankfurt"]}'},
        # Actual query
        {"role": "user",
        "content": text}
        ],
        temperature=0.0
        )
    return json.loads(response.choices[0].message.content)
    result = extract_entities(
        "Satya Nadella confirmed that Microsoft will open a new office in Tokyo."
        )
    print(json.dumps(result, indent=2))

# Instructor: extract structured entities directly as a Pydantic model.
# Replaces manual JSON parsing with type-safe, validated output.
import instructor
from openai import OpenAI
from pydantic import BaseModel

class Entities(BaseModel):
    persons: list[str]
    organizations: list[str]
    locations: list[str]

def extract_entities(text: str) -> Entities:
    client = instructor.from_openai(OpenAI())
    return client.chat.completions.create(
        model="gpt-4o-mini",
        response_model=Entities,
        messages=[{"role": "user", "content": text}],
    )

entities = extract_entities(
    "Satya Nadella confirmed that Microsoft will open a new office in Tokyo.")
print(entities.persons)       # ['Satya Nadella']
print(entities.organizations) # ['Microsoft']
print(entities.locations)     # ['Tokyo']

Notice how the second few-shot example deliberately shows an edge case where no persons are present. This teaches the model to return an empty list rather than hallucinating a name. Without this example, models sometimes invent a person associated with the European Central Bank.

12.1.4 System Prompts and Role Assignment

The system prompt is the most powerful lever in prompt design. It sets the model's persona, establishes behavioral constraints, defines the output format, and provides persistent context that applies to every turn of the conversation. A well-crafted system prompt transforms a general-purpose model into a specialized tool.

12.1.4.1 Role Prompting

Assigning a role to the model activates domain-specific knowledge and adjusts the style, vocabulary, and reasoning patterns of the response. When you tell a model to act as a "senior data scientist," it tends to provide more technically rigorous answers with appropriate caveats. When you assign the role of "elementary school teacher," it simplifies language and uses analogies. This works because the model has seen text written by people in these roles during pretraining data, and the role assignment shifts the probability distribution toward that style of text.

12.1.4.2 System Prompt Architecture

Production system prompts follow a layered structure. Each layer serves a distinct purpose, and the order matters because models pay more attention to instructions that appear early in the prompt and those that appear at the very end. Keep in mind that every prompt consumes tokens from the model's context window (recall the context window concept from Chapter 3). A system prompt of 2,000 tokens leaves less room for user input, few-shot examples, and the model's response. For models with 4K or 8K context windows, long system prompts can significantly constrain the available space; models with 128K+ windows offer much more headroom, but token cost still scales with prompt length. Code Fragment 12.1.4a demonstrates this layered structure for a medical coding application.

# Layered system prompt architecture for production use
# Sections: Role, Task, Constraints, Output Format, Examples
SYSTEM_PROMPT = """
## Role
You are a medical coding assistant specializing in ICD-10 classification.
You have 15 years of experience in health information management.
## Task
Given a clinical note, extract the primary diagnosis and assign the
correct ICD-10 code. If the note is ambiguous, list the top 3 most
likely codes with confidence scores.
## Constraints
- Only use ICD-10-CM codes (not ICD-10-PCS procedure codes)
- Never fabricate codes; if unsure, say "requires manual review"
- Do not provide medical advice or treatment recommendations
- Flag any note that mentions patient safety concerns
## Output Format
Return a JSON object:
{
 "primary_diagnosis": "description",
 "icd10_code": "X00.0",
 "confidence": 0.95,
 "alternatives": [{"code": "X00.1", "confidence": 0.80}],
 "flags": ["safety_concern"] or []
}
## Examples
[Include 2-3 few-shot examples here]
"""

The diagram below shows how these layers stack together in a production system prompt architecture.

Figure 12.1.3a: A production system prompt follows a layered architecture. Models attend most strongly to early and late content.

Why prompt architecture matters for agents. The layered system prompt structure introduced here becomes critical when building AI agents (Chapter 26). In agent systems, the system prompt must define not only the persona and output format, but also the agent's available tools, decision-making constraints, and safety boundaries. A poorly structured agent prompt leads to tool misuse, hallucinated actions, and safety violations. The layered architecture (role, task, constraints, format, examples) provides a template that scales from simple chatbots to complex multi-tool agents.

12.1.5 Prompt Templates and Variable Injection

In production applications, prompts are rarely static strings. They are templates with placeholders that get filled at runtime with user input, retrieved context, configuration parameters, and dynamic metadata. A prompt template separates the static instruction logic from the dynamic data, making prompts reusable, testable, and version-controllable.

12.1.5.1 Building a Prompt Template System

Code Fragment 12.1.5 defines a prompt template.

# Prompt template system: separate static logic from dynamic data
# Enables version control, testing, and runtime parameterization
from string import Template
from dataclasses import dataclass
from typing import Optional
@dataclass
class PromptTemplate:
    """A reusable prompt template with variable injection."""
    name: str
    version: str
    system_template: str
    user_template: str
    def render(self, **kwargs) -> list[dict]:
        """Render the template with provided variables."""
        return [
            {"role": "system",
            "content": self.system_template.format(**kwargs)},
            {"role": "user",
            "content": self.user_template.format(**kwargs)}
        ]
# Define a reusable classification template
classifier = PromptTemplate(
    name="intent_classifier",
    version="1.2.0",
    system_template="""You are a customer support intent classifier.
    Classify the customer message into one of these categories: {categories}
    Respond with only the category name, nothing else.""",
    user_template="{message}"
)
# Render at runtime
messages = classifier.render(
    categories="billing, technical_support, account, general_inquiry",
    message="I can't log into my account and I need to reset my password"
)
print(messages[0]["content"])
print(messages[1]["content"])

12.1.6 Handling Edge Cases

Even well-designed prompts encounter edge cases. Understanding the common failure modes and building defenses into your prompts is essential for production reliability.

12.1.6.1 Hallucinations

Models sometimes generate plausible-sounding but factually incorrect information. The Mata v. Avianca case (June 2023, S.D.N.Y.) is the canonical cautionary example: ChatGPT fabricated six citations to nonexistent court cases in a brief, and the attorneys were sanctioned $5,000 for filing them. To mitigate this, explicitly instruct the model to acknowledge uncertainty:

• Add instructions like: "If you are not certain, say 'I don't have enough information to answer this.'" The Anthropic prompt-engineering guide (2024) reports that an explicit "if you don't know, say so" line cuts hallucination rate on a closed-book QA task by roughly 30%, with a comparable rise in "I don't know" responses on questions the model would have guessed at.
• Request citations or sources, then verify them independently. The Lin et al. (2022) "TruthfulQA" benchmark showed that even strong frontier models fabricate URLs and paper titles on a substantial fraction of questions outside their training distribution; never trust an LLM citation that has not been resolved against a real index.
• Use constrained output formats (like selection from a fixed list) to prevent open-ended fabrication. Outlines and Guidance (Microsoft, 2023) ship FSM-based constrained decoders that mathematically guarantee the output is one of N enumerated choices, eliminating the entire category of "model made up an option that isn't in your enum."
• Lower the temperature to reduce creative (and potentially inaccurate) responses. Empirically, dropping temperature from 1.0 to 0.0 on a closed-book factual QA task typically reduces unique-hallucination rate by 40-60%, at the cost of less diverse phrasing; for any application where exact answers matter more than rewriting, default to temperature 0.

12.1.6.2 Refusals

Models refuse legitimate queries that they misidentify as harmful. This is the safety-filter overshoot problem, and it is most painful in domains adjacent to regulated topics (medical, legal, security). When building prompts where false refusals are costly, include explicit permission statements in the system prompt. For example: "You are a medical education tool. You may discuss medical conditions, symptoms, and treatments in an educational context." This calibrates the model's safety filters without disabling them.

12.1.6.3 Verbosity Control

Without explicit length constraints, models over-explain. Several techniques control output length:

• Word/sentence limits: "Answer in at most 2 sentences."
• Format constraints: "Respond with only the JSON object, no explanation."
• Negative instructions: "Do not include any caveats, disclaimers, or preamble."
• max_tokens parameter: Set a hard cap at the API level as a safety net.

12.1.7 Iterative Prompt Refinement: A Practical Workflow

Prompt engineering is inherently iterative. The most effective workflow follows a build-measure-improve cycle. Start with a simple prompt, evaluate it against a test set, identify failure patterns, refine the prompt to address those failures, and re-evaluate. Each iteration should change only one aspect of the prompt so you can attribute improvements (or regressions) to specific modifications.

pip install dspy
import dspy
dspy.configure(lm=dspy.LM("openai/gpt-4o-mini"))
class Sentiment(dspy.Signature):
    """Classify text as positive, negative, or neutral."""
    text: str = dspy.InputField()
    label: str = dspy.OutputField()
classify = dspy.Predict(Sentiment)
optimizer = dspy.MIPROv2(metric=lambda ex, pred, **_: pred.label == ex.label)
compiled = optimizer.compile(classify, trainset=trainset)