The Founder's Prototype Loop

"I scaffolded an entire backend in forty minutes. Then I spent four days figuring out why it kept apologizing to the database."

Deploy, Backend Debugging AI Agent

1. Vertical-Slice Prototyping

Most first-time builders make the same mistake: they build every feature at a shallow level before finishing any single feature deeply. The result is a wide, brittle surface that cannot survive contact with real users. Vertical-slice prototyping inverts this pattern. You pick one user flow and build it completely, from the user interface down through the LLM call, retrieval layer, guardrails, and response rendering.

Why one flow? Because a single complete flow exposes every integration point, every latency bottleneck, and every failure mode that matters. A shallow prototype that calls an LLM and displays raw text proves nothing about your product. A vertical slice that retrieves context from a knowledge base, constructs a prompt with guardrails, calls the model, validates the output, and renders it with citations proves that your architecture actually works.

2. The Build-Measure-Steer Loop

The classic Lean Startup mantra is "Build, Measure, Learn." For AI products, we sharpen the third step. "Learn" is too passive. When your AI product underperforms, you do not simply learn about it; you steer the system by adjusting prompts, swapping models, tightening guardrails, or restructuring retrieval. The founder's prototype loop is therefore Build, Measure, Steer.

Phase 1: Build

Use AI coding assistants (Cursor, Claude Code, GitHub Copilot) to scaffold your vertical slice quickly. The goal is not production-quality code; it is a functional prototype that exercises the full stack. Section 3 below covers the verification discipline required when working with generated code.

Phase 2: Measure

Connect your prototype to an evaluation harness from the start. Even five well-chosen test cases are better than zero. Instrument your prototype with the observability patterns from Chapter 30: log every prompt, every retrieved chunk, every model response, and every user reaction. You cannot steer what you cannot see.

Phase 3: Steer

This is where AI products diverge from traditional software iteration. Steering means making targeted adjustments based on evaluation evidence:

• Prompt adjustments. If the model hallucinates, add grounding instructions. If responses are too verbose, add length constraints. Every prompt change gets re-evaluated against your test cases.
• Model swaps. If latency is too high, try a smaller model. If quality is too low, try a more capable one. The API abstraction patterns from Chapter 10 make this swap straightforward.
• Guardrail additions. If the model produces unsafe or off-topic responses, add output validators, content filters, or topic classifiers. Each guardrail is a hypothesis: "this filter will reduce off-topic responses by X%." Measure it.
• Retrieval tuning. If the model lacks context, adjust your chunking strategy, embedding model, or retrieval parameters using the techniques from Chapter 20.

3. AI Coding Assistants: Trust but Verify

AI coding assistants accelerate prototyping dramatically, but they introduce a new failure mode: plausible but incorrect code. A generated function may have correct syntax, reasonable variable names, and sensible-looking logic while containing subtle bugs that only surface under edge cases. The verification discipline below applies whether you use Cursor, Claude Code, GitHub Copilot, or any other assistant.

1. Treat generated code as a draft. Read every line before integrating it. If you cannot explain what a function does, you do not understand your own prototype.
2. Run it immediately. Do not accumulate generated code. Generate a function, run it, verify the output, then move on. Small feedback loops catch errors before they compound.
3. Check the boundaries. Generated code often handles the happy path well and ignores edge cases. Test with empty inputs, oversized inputs, malformed data, and concurrent access.
4. Verify dependencies. AI assistants sometimes import libraries that do not exist, reference deprecated APIs, or use incorrect function signatures. Confirm every import and every API call against current documentation.
5. Own the architecture. Use AI assistants for implementation, not for architectural decisions. You decide the system structure; the assistant fills in the code within that structure.

4. The Prototype Playbook

The Prototype Playbook maps five stages of increasing sophistication to the techniques covered throughout this book. Each stage builds on the previous one. You do not advance to the next stage until the current stage passes its evaluation criteria.

5. A Minimal Prototype Evaluation Harness

The following Python script implements a lightweight evaluation harness that you can wire into any prototype from Stage 1 onward. It loads test cases, runs each one through your prototype function, scores the output on keyword relevance, safety constraints, and latency, then produces a summary report.

# Minimal prototype evaluation harness
# Connects to any prototype via a callable that accepts a query string
import json
import time
from dataclasses import dataclass, field
from pathlib import Path
from typing import Callable


@dataclass
class EvalCase:
 """A single evaluation test case."""
 query: str
 expected_keywords: list[str] = field(default_factory=list)
 max_latency_ms: float = 5000.0
 must_not_contain: list[str] = field(default_factory=list)


@dataclass
class EvalResult:
 """Result of running one evaluation case."""
 query: str
 response: str
 latency_ms: float
 keyword_score: float # fraction of expected keywords found
 safety_pass: bool # True if no forbidden strings appear
 latency_pass: bool # True if within latency budget
 overall_pass: bool = False

 def __post_init__(self):
 self.overall_pass = (
 self.keyword_score >= 0.5
 and self.safety_pass
 and self.latency_pass
 )


def load_cases(path: str) -> list[EvalCase]:
 """Load evaluation cases from a JSON file.

 Expected format:
 [
 {
 "query": "What is the dental coverage for dependents?",
 "expected_keywords": ["dental", "dependents", "coverage"],
 "max_latency_ms": 3000,
 "must_not_contain": ["I don't know"]
 }
 ]
 """
 raw = json.loads(Path(path).read_text(encoding="utf-8"))
 return [EvalCase(**case) for case in raw]


def run_eval(
 prototype_fn: Callable[[str], str],
 cases: list[EvalCase],
) -> list[EvalResult]:
 """Run all evaluation cases and return scored results."""
 results: list[EvalResult] = []

 for case in cases:
 # Time the prototype call
 start = time.perf_counter()
 response = prototype_fn(case.query)
 elapsed_ms = (time.perf_counter() - start) * 1000

 # Score keyword presence
 response_lower = response.lower()
 hits = sum(
 1 for kw in case.expected_keywords
 if kw.lower() in response_lower
 )
 keyword_score = (
 hits / len(case.expected_keywords)
 if case.expected_keywords else 1.0
 )

 # Check safety: none of the forbidden strings should appear
 safety_pass = all(
 forbidden.lower() not in response_lower
 for forbidden in case.must_not_contain
 )

 results.append(EvalResult(
 query=case.query,
 response=response[:200], # truncate for readability
 latency_ms=round(elapsed_ms, 1),
 keyword_score=round(keyword_score, 2),
 safety_pass=safety_pass,
 latency_pass=elapsed_ms <= case.max_latency_ms,
 ))

 return results


def print_report(results: list[EvalResult]) -> None:
 """Print a summary evaluation report to stdout."""
 total = len(results)
 passed = sum(1 for r in results if r.overall_pass)
 print(f"\n{'=' * 60}")
 print(f" Prototype Evaluation Report")
 print(f" {passed}/{total} cases passed "
 f"({100 * passed / total:.0f}%)")
 print(f"{'=' * 60}")

 for i, r in enumerate(results, 1):
 status = "PASS" if r.overall_pass else "FAIL"
 print(f"\n [{status}] Case {i}: {r.query[:50]}")
 print(f" Keyword score : {r.keyword_score}")
 print(f" Safety pass : {r.safety_pass}")
 print(f" Latency : {r.latency_ms} ms "
 f"(limit: {'ok' if r.latency_pass else 'EXCEEDED'})")

 print(f"\n{'=' * 60}\n")


# Example usage with a stub prototype
if __name__ == "__main__":
 # Replace this stub with your actual prototype function
 def my_prototype(query: str) -> str:
 """Stub that simulates a prototype response."""
 return (
 "Based on the benefits handbook, dental coverage "
 "for dependents includes preventive and basic "
 "restorative services under the standard plan."
 )

 # Define inline test cases for quick testing
 cases = [
 EvalCase(
 query="What dental coverage do dependents get?",
 expected_keywords=["dental", "dependents", "coverage"],
 max_latency_ms=3000,
 must_not_contain=["I don't know"],
 ),
 EvalCase(
 query="Is vision included in the basic plan?",
 expected_keywords=["vision", "basic", "plan"],
 max_latency_ms=3000,
 must_not_contain=["I cannot help"],
 ),
 ]

 results = run_eval(my_prototype, cases)
 print_report(results)

6. Putting It All Together: A Day in the Loop

Here is what one iteration of the Build-Measure-Steer loop looks like in practice for a founder prototyping an HR benefits assistant:

1. Morning (Build). Use Claude Code to scaffold a FastAPI endpoint that accepts a question, retrieves relevant chunks from a benefits PDF using a simple vector store, constructs a grounded prompt, and returns the model's response. Total scaffolding time: 45 minutes. Review every generated file, fix two incorrect import paths, and add error handling the assistant omitted.
2. Midday (Measure). Write 10 evaluation cases covering common benefits questions, edge cases (questions about policies that do not exist), and adversarial inputs (requests for salary data the bot should not disclose). Run the harness from Code Fragment 37.2.1. Result: 7/10 pass. Three failures: one hallucinated a policy that does not exist, one exceeded the latency budget, one leaked a system prompt fragment.
3. Afternoon (Steer). For the hallucination: add an instruction to the prompt requiring every claim to cite a retrieved chunk, and add a post-processing validator that checks for citations. For the latency issue: switch from a large model to a smaller one for this simple Q&A task, following the model-selection guidance from Chapter 10. For the prompt leak: add an output filter that strips any text matching the system prompt pattern. Re-run the harness. Result: 9/10 pass. The remaining failure needs better retrieval; note it for tomorrow's iteration.