Open-Source Realtime: Moshi, Pipecat, LiveKit Agents

"The proprietary realtime APIs are great. They are also a single vendor with API price increases and a 30-minute session limit. Open source is the hedge."

Echo, Open-Realtime AI Agent

39.5.1 Moshi: A Native Audio-Text Foundation Model

Kyutai's Moshi (Defossez et al., 2024) is the first widely-used open-weights native audio-text model. The technical novelty is full-duplex: the model has two parallel audio streams (one for each side of the conversation) and can speak while listening. This is a step beyond GPT-4o Realtime, which alternates turns.

Architecture highlights:

• Mimi codec: a 12.5 Hz neural codec producing 8 codebook tokens per frame, for ~100 audio tokens/sec/stream. Lower token rate than GPT-4o means cheaper inference.
• Hierarchical transformer: a "temporal" transformer over frames and a "depth" transformer over codebooks. This factorization keeps the per-step cost manageable.
• Inner monologue: the model emits a text stream alongside the audio stream, so you get a real-time transcript for free.
• Open weights: released under permissive license. The 7B variant runs at real-time on a single L4 GPU.

# Moshi inference via the kyutai-labs/moshi package.
# Full-duplex audio: user and model audio are concurrent streams.
from moshi import load_moshi
import torch, sounddevice as sd, numpy as np

model, mimi, tokenizer = load_moshi(
    checkpoint="kyutai/moshiko-pytorch-bf16",
    device="cuda",
)
FRAME = 1920   # 24kHz audio, 12.5Hz codec = 80ms frames

def run():
    with sd.Stream(samplerate=24000, channels=1,
                   blocksize=FRAME) as stream:
        user_tokens = []
        model_tokens = []
        while True:
            # Read 80ms of microphone audio
            audio, _ = stream.read(FRAME)
            # Encode to user codec tokens
            u_tok = mimi.encode(torch.from_numpy(audio).cuda())
            user_tokens.append(u_tok)
            # Model emits one frame of (audio tokens, text token) in response.
            # Both streams advance concurrently; full-duplex.
            m_tok, m_text = model.step(u_tok)
            model_tokens.append(m_tok)
            # Decode and play model audio
            m_audio = mimi.decode(m_tok)
            stream.write(m_audio.cpu().numpy())
            # Real-time text transcript
            if m_text:
                print(tokenizer.decode(m_text), end="", flush=True)

39.5.2 Pipecat: A Pipeline Framework for Voice Agents

Pipecat (Daily.co, 2024) is the most widely-used open-source framework for composing voice agent pipelines. The abstractions: a pipeline is a directed graph of processors that pass frames (audio chunks, transcripts, LLM tokens, TTS audio) through. The framework handles backpressure, error recovery, and observability.

# Pipecat pipeline: Daily transport, Deepgram STT,
# OpenAI GPT-4o-mini, Cartesia TTS. Voice agent in ~40 lines.
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.task import PipelineTask
from pipecat.services.deepgram import DeepgramSTTService
from pipecat.services.openai import OpenAILLMService
from pipecat.services.cartesia import CartesiaTTSService
from pipecat.transports.services.daily import DailyTransport

transport = DailyTransport(room_url=URL, token=TOKEN, bot_name="Agent")
stt = DeepgramSTTService(api_key=DEEPGRAM_KEY)
llm = OpenAILLMService(api_key=OPENAI_KEY, model="gpt-4o-mini")
tts = CartesiaTTSService(api_key=CARTESIA_KEY, voice_id="79a125e8-...")

pipeline = Pipeline([
    transport.input(),          # Mic in from Daily room
    stt,                         # Audio -> partial transcripts
    llm,                         # Transcripts -> LLM tokens
    tts,                         # LLM tokens -> audio
    transport.output(),         # Audio -> Daily room
])

task = PipelineTask(pipeline, params={"enable_metrics": True})
asyncio.run(task.run())

Pipecat ships connectors for most production services: Deepgram, AssemblyAI, OpenAI, Anthropic, Cartesia, ElevenLabs, Replicate, Whisper, plus transport options (Daily, LiveKit, Twilio, WebRTC, WebSocket). Replacing any one piece is a one-line change. This makes Pipecat the right starting point for most open-source voice agent deployments in 2026.

39.5.3 LiveKit Agents

LiveKit Agents (LiveKit, 2024) is a similar framework with deeper WebRTC integration. LiveKit's heritage is real-time communication infrastructure (Zoom-style video rooms); the Agents framework adds AI participants that can join those rooms.

The conceptual model: a LiveKit room is a meeting; an agent is a participant in that room with the same audio I/O affordances as a human participant. The agent uses LiveKit's media servers for low-latency WebRTC transport across geographic regions.

• Strengths: best-in-class WebRTC performance, geographic scale (LiveKit Cloud has global media servers), native support for video as well as audio.
• Weaknesses: more opinionated about transport (LiveKit only); newer, smaller community than Pipecat.

The choice between Pipecat and LiveKit Agents is largely about transport: if you are deploying on a phone call (Twilio) or a custom WebSocket, use Pipecat. If you are building a video-call experience (e.g., a virtual interviewer or a tutoring widget), use LiveKit Agents.

39.5.4 Transport Layer: LiveKit, Daily, Twilio

The transport layer carries audio between the user device and the agent. Three production-grade choices:

39.5.5 Deployment and Scaling

An open-source voice agent typically runs as a Python service in a container, one process per concurrent conversation. Scaling involves three considerations:

1. GPU residency for self-hosted models: if you run Moshi or a self-hosted Whisper, you need GPUs warm and waiting. A single A10G can serve 2 to 4 concurrent Moshi conversations; a single H100 can serve 8 to 12. Use Modal (Section 9.5) or RunPod for elastic GPU pools.
2. Network proximity: deploy near your users. LiveKit Cloud's global media servers handle this transparently; with self-hosted media, you need multi-region deployment.
3. Concurrency limits: each conversation holds open sockets to ASR, LLM, TTS providers. Stay under the per-vendor rate limits or roll out across multiple API keys.