Unified Multimodal Models & Omni-Architectures

A truly multimodal model does not translate between modalities. It thinks in all of them simultaneously.

Pixel, Polyglot Perceiving AI Agent

1. Pipeline vs. Native Multimodal Architectures

The distinction between pipeline and native multimodal models is the most important architectural concept in this section. A pipeline model chains separate specialist models: CLIP encodes the image, Whisper transcribes the audio, and a language model reasons over the combined text representations. A native multimodal model processes all modalities within a single set of transformer layers, using a shared representation space where image tokens, audio tokens, and text tokens attend to each other directly.

2. Early Fusion vs. Late Fusion

The distinction between early and late fusion describes where in the model different modalities are combined. In early fusion, raw inputs from all modalities are tokenized and concatenated into a single sequence before the first transformer layer. Every attention layer can attend across modalities from the very start. In late fusion, each modality is processed by separate encoder stacks for many layers, and the representations are only combined in the final layers.

Early Fusion (GPT-4o style)

GPT-4o and Gemini 2.0 use early fusion: images are converted to visual tokens by a patch-based tokenizer, audio is converted to audio tokens by a learned encoder, and text uses the standard tokenizer. All tokens are interleaved into a single sequence and fed through the full transformer stack. This allows the model to reason about relationships between modalities from the earliest layers. For example, the model can associate a spoken word with a visual region in the first few layers, rather than having to reconstruct this relationship after separate encoding.

Late Fusion (Flamingo style)

Late fusion models like Flamingo process images through a frozen vision encoder for many layers, producing a compact set of visual features. These features are then injected into the language model via cross-attention layers at specific positions. The advantage is efficiency: the vision encoder can be pretrained independently and frozen, reducing training cost. The disadvantage is that cross-modal interactions are limited to the layers where injection occurs. Figure 27.4.2 compares early fusion, late fusion, and cross-attention fusion architectures side by side. Code Fragment 27.4.2 below puts this into practice.

# Gemini 2.5: native multimodal with early fusion
# Process image + audio + text in a single forward pass for cross-modal reasoning
import google.generativeai as genai
from pathlib import Path

# Gemini 2.5: native multimodal with early fusion
# The model processes image + text + audio in a single forward pass
genai.configure(api_key="YOUR_API_KEY")
model = genai.GenerativeModel("gemini-2.5-flash")

# Upload an image and audio file
image = genai.upload_file(Path("meeting_whiteboard.jpg"))
audio = genai.upload_file(Path("meeting_recording.mp3"))

# Cross-modal query: reason across image and audio simultaneously
response = model.generate_content([
 image,
 audio,
 "Compare the action items discussed in the audio recording with "
 "what was written on the whiteboard. Identify any items mentioned "
 "verbally but missing from the whiteboard."
])

print(response.text)

3. Any-to-Any Generation

The most striking capability of unified multimodal models is any-to-any generation: the ability to take input in any combination of modalities and produce output in any modality. GPT-4o can accept text and produce audio, accept an image and produce text, or accept audio and produce text with an image. This is fundamentally different from pipeline systems where each input-output modality pair requires a separate model chain.

Architecturally, any-to-any generation requires the model to have both encoders and decoders for every supported modality. For text, the standard autoregressive decoder suffices. For images, the model typically includes a diffusion decoder or a discrete visual token decoder that generates images token by token. For audio, a vocoder or audio token decoder converts the model's internal representations into waveforms. The shared transformer backbone provides the cross-modal reasoning, while modality-specific heads handle the final generation step. Code Fragment 27.4.2 below puts this into practice.

# Any-to-any generation: analyze an image, then generate a new one based on the analysis
# Demonstrates cross-modal flow from image understanding to image generation
from openai import OpenAI
import base64

client = OpenAI()

# GPT-4o: native multimodal understanding + generation
# Accept image input, produce text analysis + generate related image

# Step 1: Analyze an image with text
with open("chart.png", "rb") as f:
 image_data = base64.b64encode(f.read()).decode()

analysis = client.chat.completions.create(
 model="gpt-4o",
 messages=[{
 "role": "user",
 "content": [
 {"type": "image_url",
 "image_url": {"url": f"data:image/png;base64,{image_data}"}},
 {"type": "text",
 "text": "Describe the trends in this chart. What are the key insights?"}
 ]
 }]
)
print(analysis.choices[0].message.content)

# Step 2: Generate an improved version of the chart (image generation)
image_response = client.images.generate(
 model="gpt-image-1",
 prompt=(
 f"Create a clean, professional bar chart showing the following data: "
 f"{analysis.choices[0].message.content[:500]}. "
 f"Use a modern color palette with clear labels and a title."
 ),
 size="1024x1024"
)
print(f"Generated image URL: {image_response.data[0].url}")

from transformers import pipeline

captioner = pipeline("image-to-text", model="Salesforce/blip2-opt-2.7b")
result = captioner("chart.png")
print(result[0]["generated_text"]) # "a bar chart showing quarterly revenue growth"

4. Training Unified Multimodal Models

Training a unified multimodal model requires solving several challenges that do not exist in text-only training. The first is tokenization across modalities: images, audio, and text have fundamentally different structures and information densities. A single image can contain as much information as thousands of text tokens, and audio streams have a very different temporal structure than text sequences. The model needs a tokenization strategy that represents each modality efficiently while allowing meaningful cross-modal attention.

Training Stages

Unified multimodal models typically follow a staged training curriculum, where each phase builds on the previous one:

1. Modality-specific pretraining: Train separate encoders and decoders on large unimodal datasets (ImageNet for vision, LibriSpeech for audio, text corpora for language). This gives each modality a strong foundation.
2. Alignment pretraining: Train the model on paired multimodal data (image-caption pairs, audio-transcript pairs) to align representations across modalities in a shared embedding space.
3. Unified fine-tuning: Fine-tune the complete model end-to-end on interleaved multimodal data where inputs and outputs can be in any modality. This stage teaches the model to reason across modalities and generate in any target modality.
4. Instruction tuning: Fine-tune on human-curated instruction-following data that covers multimodal tasks: "describe this image," "transcribe this audio," "generate an image of..." This stage aligns the model with user expectations.

5. Multimodal Benchmarks

Evaluating multimodal models requires benchmarks that test cross-modal reasoning, not just performance on each modality in isolation. A model that achieves high accuracy on ImageNet and high perplexity on text benchmarks might still fail at tasks requiring joint reasoning across images and text. The major multimodal benchmarks as of 2025 include:

6. The Current Landscape: GPT-4o, Gemini, and Beyond

As of early 2025, the unified multimodal model landscape is dominated by three families, each with distinct architectural choices and capabilities.

GPT-4o (OpenAI)

GPT-4o ("omni") was the first widely available native multimodal model, launched in May 2024. It processes text, images, and audio natively in a single model. The "o" in the name reflects the "omni" architecture: rather than routing different modalities through separate models, GPT-4o uses a unified transformer that operates on tokens from all modalities. Its image generation capabilities were significantly expanded in 2025, with GPT-4o gaining the ability to produce high-quality images alongside text responses.

Gemini 2.0/2.5 (Google)

Google's Gemini family uses a natively multimodal architecture trained on interleaved text, image, audio, and video data from the ground up. Gemini 2.0 Flash and Gemini 2.5 Pro support long-context windows (up to 1 million tokens for Gemini 2.5 Pro) that can include mixed-modality inputs, making them particularly strong for tasks involving long documents with embedded images, lengthy videos, or multi-turn conversations with visual context. Gemini 2.5 also introduced "thinking" capabilities (extended reasoning) for multimodal tasks.

Open-Weight Alternatives

On the open-weight side, models like LLaVA-OneVision, InternVL2, and Qwen2-VL have closed much of the gap with proprietary models on vision-language tasks. However, true any-to-any generation (accepting and producing images, audio, and video) remains primarily a capability of proprietary systems due to the enormous compute requirements for end-to-end multimodal training.

Exercises