Frontier Omni Models: GPT-4o, Gemini, Llama-4-Omni, Chameleon

"The omni demo is twenty seconds of magic followed by an API price list."

Pixel, Omni-Aspirational AI Agent

22.9.1 GPT-4o and the Realtime API

GPT-4o (omni), released May 2024, was OpenAI's first natively multimodal model. The full architecture was not disclosed, but the public information and behavior suggest:

• A single transformer trained on interleaved text, audio, and image tokens (early fusion).
• Discrete audio tokenization at ~24 Hz frame rate (compatible with low-latency streaming).
• Image inputs as patch tokens (likely SigLIP-style encoder).
• Output image generation through a separate diffusion decoder gated by the model's emit tokens (a hybrid pattern; see Section 22.8).

The capability headline was the Realtime API: an audio-in, audio-out conversational loop with under 320ms time-to-first-audio-token. Voice mode preserves paralinguistic cues (emotion, sarcasm, sighs) that pure-text pipelines lose. Pricing for the Realtime API in 2026 sits around $0.06 per minute of audio input and $0.24 per minute of audio output, with $5 / $20 per million text input/output tokens for the text path.

# GPT-4o Realtime API: WebSocket session for streaming audio.
# See Section 38.2 for the full protocol walkthrough.
import asyncio
import websockets
import json

async def realtime_voice_chat():
    url = "wss://api.openai.com/v1/realtime?model=gpt-4o-realtime-preview"
    headers = {"Authorization": f"Bearer {api_key}", "OpenAI-Beta": "realtime=v1"}

    async with websockets.connect(url, extra_headers=headers) as ws:
        await ws.send(json.dumps({
            "type": "session.update",
            "session": {
                "modalities": ["text", "audio"],
                "voice": "alloy",
                "input_audio_format": "pcm16",
                "output_audio_format": "pcm16",
                "turn_detection": {"type": "server_vad"},
            }
        }))
        # Stream PCM-16 audio chunks of ~100ms each.
        async for chunk in mic_chunks():
            await ws.send(json.dumps({
                "type": "input_audio_buffer.append",
                "audio": b64(chunk),
            }))
        # Server emits response.audio.delta events with TTS chunks.

22.9.2 Gemini 2.0 and 2.5: Native Multimodal at Long Context

Google DeepMind's Gemini family was the first frontier-scale natively multimodal model (Gemini 1.0 in December 2023). Gemini 2.0 (December 2024) and Gemini 2.5 (mid-2025) deepened the capability:

• Long-context multimodal: Gemini 1.5 Pro shipped 2M-token context, including interleaved images, audio, and video frames. A 1-hour video can be passed in full at native temporal resolution. Gemini 2.5 Pro extended this to ~10M tokens with optimized retrieval.
• Native audio generation and understanding: voice and audio streams are tokenized at a higher rate than GPT-4o (sufficient for music understanding, not just speech).
• Tool use and agent capabilities: Gemini 2.0 was the first frontier model to ship with native agent tooling (web search, code execution, mouse-and-keyboard control) as built-in capabilities.
• Video generation: Veo 3 (the Gemini-aligned video model) integrates with the LLM for prompt expansion and post-generation editing.

Pricing in 2026: Gemini 2.5 Pro is $1.25 per million input tokens and $10 per million output tokens for the standard context window, with audio and image tokens billed at modality-specific rates. Gemini 2.0 Flash is roughly 5x cheaper and remains the workhorse for high-volume multimodal applications.

22.9.3 Chameleon and Llama-4-Omni: Open-Source Frontier

Meta's Chameleon (Team, 2024) was a milestone: the first open-weights frontier-scale early-fusion multimodal model. Chameleon-7B and Chameleon-34B handle interleaved text and image input and output, all through a single autoregressive transformer over discrete tokens. The model uses:

• A VQ-VAE-style image tokenizer producing 1024 tokens per 512x512 image.
• BPE text tokenizer with vocabulary extended by the image codebook.
• Standard decoder-only transformer (similar to Llama-2) with QK-norm for training stability.

Chameleon's released image generation capability was limited (the open weights had it disabled at launch for safety reasons), but the architectural blueprint influenced everything that followed.

Llama-4-Omni (Meta, 2025) inherits Chameleon's early-fusion DNA and extends it with audio I/O. The model is positioned as the open-source competitor to GPT-4o, with three sizes (8B, 70B, 400B+ MoE) and weights released under a permissive license for research. Inference cost on commodity hardware: a Llama-4-Omni-8B serves voice conversations at ~$0.01 per minute on a single A10G, an order of magnitude cheaper than GPT-4o Realtime.

22.9.4 Capability and Cost Matrix

22.9.5 Architecture Deep-Dive Notes

Detailed architectures are confidential for the proprietary models, but the open papers and the behavior in production give consistent clues:

• GPT-4o: appears to be early fusion at the input, late fusion for image generation. Audio in/out uses a discrete codec at ~25 Hz. Estimated parameter count: 200B to 400B in MoE form.
• Gemini 2.5: native multimodal training on Pathways infrastructure, with TPU v5p deployment. Long context relies on a learned compression scheme that retrieves relevant chunks from a memory bank, similar in spirit to Section 32.1's RAG patterns but learned end-to-end.
• Llama-4-Omni: early fusion, RVQ audio tokenizer at 50 Hz, image tokenizer at ~1024 tokens per image. MoE for the 400B variant; dense for smaller sizes. Released alongside training code, which makes the architecture inspectable.
• Chameleon: dense decoder-only, fully early fusion, 7B and 34B sizes. Used QK-norm and clamped attention to stabilize training across modalities, an architectural lesson that propagated to most subsequent omni models.

22.9.6.2026 The 2026 Pick List

22.9.7 Evaluation: Beyond the Leaderboards

Benchmark scores (MMLU-Pro, MMMU, MMB, LiveBench) capture capability under standardized conditions but miss two factors that often dominate production decisions:

1. Voice agent tone and persona: GPT-4o Voice's prosody feels noticeably different from Gemini Live's. Audition the models with your actual application prompts before committing.
2. Latency under real load: published TTFT figures assume warm cache, light load. Production load patterns can double the latency on any of these systems.

The recommended evaluation protocol: 100 to 500 real prompts from your domain, run each through every candidate model, score the outputs on a 5-point rubric covering accuracy, helpfulness, tone, and latency. The cost of running such an eval (~$200 in API spend plus a few hours of human time) is dwarfed by the cost of picking the wrong frontier model.

pip install openai anthropic pillow pandas