Vision, Speech-to-Speech, and Voice AI Frameworks

Speech-to-speech skips the text in the middle and saves 200 milliseconds. It also skips the audit log in the middle and saves 200 milliseconds of compliance review. Choose your trade.

Echo, End-to-End-Audio AI Agent

Building on the STT, TTS, and real-time pipeline foundations from Section 39.6, this part extends conversational AI in two directions. First, we add vision so an assistant can see what the user is pointing at. Second, we look at the new wave of native speech-to-speech models that skip the text bottleneck, and we compare the orchestration frameworks (LiveKit, Pipecat, Vocode) that production teams actually deploy.

39.7.1 Vision in Conversations

Multimodal LLMs (GPT-4o, Claude Sonnet 4, Gemini) can process images alongside text, enabling conversational AI systems that can see. Users can share photos, screenshots, documents, or live camera feeds, and the system can discuss what it sees. This capability transforms many use cases: visual troubleshooting ("what is wrong with this error message?"), product identification ("what plant is this?"), accessibility assistance, and interactive tutoring with visual materials. Figure 39.7.2 illustrates the multimodal conversational pipeline.

import base64
from openai import OpenAI
client = OpenAI()
def encode_image_to_base64(image_path: str) -> str:
    """Read an image file and encode it as base64."""
    with open(image_path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")
class MultimodalConversation:
    """Conversational AI with vision capabilities."""
    def __init__(self, system_prompt: str):
        self.system_prompt = system_prompt
        self.history: list[dict] = []
    def send_text(self, user_message: str) -> str:
        """Send a text-only message."""
        self.history.append({
            "role": "user",
            "content": user_message
            })
        return self._get_response()
    def send_image(self, image_path: str,
        question: str = "What do you see?") -> str:
        """Send an image with an optional question."""
        b64_image = encode_image_to_base64(image_path)
        # Determine MIME type from extension
        ext = image_path.rsplit(".", 1)[-1].lower()
        mime_map = {"jpg": "jpeg", "jpeg": "jpeg",
            "png": "png", "gif": "gif", "webp": "webp"}
        mime_type = f"image/{mime_map.get(ext, 'jpeg')}"
        self.history.append({
            "role": "user",
            "content": [
            {"type": "text", "text": question},
            {
            "type": "image_url",
            "image_url": {
            "url": f"data:{mime_type};base64,{b64_image}",
            "detail": "high"
            }
            }
            ]
            })
        return self._get_response()
    def send_image_url(self, url: str, question: str) -> str:
        """Send an image via URL with a question."""
        self.history.append({
            "role": "user",
            "content": [
            {"type": "text", "text": question},
            {
            "type": "image_url",
            "image_url": {"url": url, "detail": "auto"}
            }
            ]
            })
        return self._get_response()
    def _get_response(self) -> str:
        """Get a response from the multimodal LLM."""
        messages = [
            {"role": "system", "content": self.system_prompt},
            *self.history
            ]
        response = client.chat.completions.create(
            model="gpt-4o",
            messages=messages,
            max_tokens=1000
            )
        assistant_msg = response.choices[0].message.content
        self.history.append({
            "role": "assistant", "content": assistant_msg
            })
        return assistant_msg
# Example: Visual troubleshooting assistant
troubleshooter = MultimodalConversation(
    system_prompt=(
    "You are a technical support assistant that can analyze "
    "screenshots and photos to help users troubleshoot issues. "
    "When shown an image, describe what you see and provide "
    "specific, actionable solutions."
    )
    )
# User shares a screenshot of an error
response = troubleshooter.send_image(
    "error_screenshot.png",
    "I keep getting this error when I try to start the application. "
    "What should I do?"
    )
print(response)

Figure 39.7.2a: Multimodal conversational pipeline accepting voice, text, and image inputs, processing through a multimodal LLM, and producing voice or text outputs.

39.7.2 Native Speech-to-Speech Models

A transformative shift in voice AI is the emergence of native speech-to-speech models that process audio directly, without decomposing the task into separate speech-to-text, language model reasoning, and text-to-speech stages. These models represent a fundamental architectural departure from the pipeline approach described above, and they promise lower latency, richer expressiveness, and a more natural conversational experience.

39.7.2.1 The Pipeline Problem

The traditional STT/LLM/TTS pipeline introduces several limitations that stem from its cascaded architecture. First, each stage adds latency: 200 to 500ms for STT transcription, 500 to 2000ms for LLM generation, and 200 to 500ms for TTS synthesis, yielding a total round-trip time of 1 to 3 seconds. Second, the text bottleneck between stages loses crucial information. When speech is transcribed to text, paralinguistic signals (tone, emotion, pacing, emphasis, hesitation) are discarded. The LLM reasons over flat text, unable to perceive that the user sounds frustrated, confused, or sarcastic. Third, the TTS stage must synthesize prosody from scratch, since it has no access to the original audio input. The result is responses that sound fluent but lack the contextual expressiveness of natural human conversation.

39.7.2.2 Architecture of Speech-Native Models

Native speech-to-speech models replace the three-stage pipeline with an end-to-end architecture that maps audio input directly to audio output. The core idea is to train a single model on speech tokens rather than (or in addition to) text tokens. Audio is encoded into discrete tokens using a neural audio codec (such as EnCodec or SoundStorm), producing a sequence of audio tokens that capture both linguistic content and acoustic properties. The model then generates output audio tokens autoregressively, which are decoded back into a waveform by the codec decoder.

Several architectural variants have emerged:

• Audio-in, audio-out softmax: The model's vocabulary includes both text and audio tokens. During training, it learns to map audio token sequences to audio token sequences, with text tokens serving as an intermediate representation that anchors semantic understanding. GPT-4o uses this approach, processing audio natively within the same transformer that handles text and images.
• Dual-stream architectures: Models like Moshi (by Kyutai) use parallel streams for the user's speech and the model's speech, enabling the model to listen and speak simultaneously. This addresses the turn-taking problem by allowing overlap and interruption without explicit endpointing logic. Moshi processes both streams at 12.5 Hz (one audio frame every 80ms), achieving 200ms theoretical latency.
• Codec language models: These models treat audio codec tokens as a "language" and apply standard language modeling techniques. The model generates multiple codebook levels in parallel or hierarchically: the first level captures semantic content, while subsequent levels add acoustic detail (timbre, pitch, room acoustics). This hierarchical generation enables high-quality audio output at interactive speeds.

39.7.2.3 Key Models and Platforms

OpenAI GPT-4o Realtime API. Released in late 2024, the Realtime API provides WebSocket-based access to GPT-4o's native audio capabilities. The model accepts audio input and produces audio output without intermediate text conversion, achieving round-trip latencies of 300 to 500ms (compared to 1 to 3 seconds for the pipeline approach). The API supports function calling during audio conversations, allowing the model to invoke tools while maintaining a natural voice interaction. Developers can configure voice presets, control turn detection sensitivity, and implement interruption handling through the WebSocket protocol.

Kyutai Moshi. Moshi is an open-weight speech-to-speech model that introduced the concept of full-duplex conversation. Unlike turn-based systems that alternate between listening and speaking, Moshi maintains simultaneous input and output audio streams. The architecture combines a text language model (Helium, a 7B parameter model) with an audio codec model (Mimi) that compresses speech to 1.1 kbps. Moshi achieves a theoretical latency of 160ms and a practical latency of 200ms, making it one of the fastest conversational AI systems. The model was released with open weights and can run on consumer GPUs.

Google Gemini Live. Gemini 2.0 Flash incorporates native audio understanding and generation as part of its multimodal architecture. Gemini Live enables real-time conversational interactions where the model can see (via camera), hear (via microphone), and respond with synthesized speech. The model natively understands audio context, including environmental sounds, multiple speakers, and emotional tone. Gemini Live's integration with Google's ecosystem (Search, Maps, Home) enables voice-driven agent interactions that span multiple modalities and data sources.

39.7.2.4 Pipeline vs. Native: Tradeoffs

39.7.2.5 Building with the Realtime API

Working with native speech-to-speech APIs requires a shift from REST-based request/response patterns to persistent WebSocket connections with streaming audio. The developer sends audio chunks as the user speaks, and the model streams audio chunks back as it generates a response. Function calls can be interleaved with audio output, and the model manages turn-taking internally.

Key implementation considerations include: audio format negotiation (PCM 16-bit at 24kHz is common), voice activity detection configuration (sensitivity thresholds, silence duration for turn detection), interruption policy (whether the model should stop speaking when the user starts), and session management (maintaining conversation state across audio turns). Frameworks like LiveKit and Pipecat are adding native support for these APIs, abstracting the WebSocket management and audio buffering into higher-level constructs.

39.7.3 Comparing Voice AI Orchestration Frameworks

pip install livekit livekit-agents livekit-plugins-openai sounddevice