Early Fusion vs Late Fusion

"Where you mix the modalities determines how deeply they ever talk to each other."

Pixel, Fusion-Curious AI Agent

22.7.1 Three Points on the Spectrum

Fusion is a continuum, but three operating points cover most production systems:

• Early fusion (Chameleon, Llama-4-Omni input side): every modality is tokenized at the input. Text uses BPE tokens, images use VQ-VAE or similar discrete codes, audio uses a neural codec like SoundStream. All tokens are interleaved and processed by the same transformer from layer 1 onward.
• Mid-fusion (LLaVA, Qwen-VL, Idefics2): each modality has its own frozen encoder (CLIP-ViT for images, audio Whisper, etc.) that produces feature vectors. A small projection layer (linear or MLP) maps those features into the LLM's embedding space. They are then prefixed or interleaved with text tokens and fed into a vanilla text LLM.
• Late fusion (CLIP-style retrieval models): modality-specific encoders produce embeddings that meet only at the contrastive head or the very last attention layer. Late fusion is dominant in retrieval (Section 33.1) but rare in generation.

22.7.2 Early Fusion Mechanics: The Chameleon Approach

Meta's Chameleon (Team, 2024) is the cleanest expression of early fusion. The full pipeline:

1. Train a discrete image tokenizer (a VQ-GAN-style encoder) that maps a 256x256 image to 1024 tokens from a 8192-entry codebook.
2. Train a discrete audio tokenizer similarly.
3. Expand the LLM's vocabulary to include the image and audio codebook entries.
4. Train a single autoregressive transformer on interleaved sequences of text, image, and audio tokens, with a unified next-token-prediction objective.

The result is a model that does not distinguish between modalities at the architectural level. Every layer sees every token type and learns whatever cross-modal patterns the data demands. The same approach drives the input side of Llama-4-Omni and the image generation head of GPT-4o.

22.7.3 Mid-Fusion Mechanics: The LLaVA Pattern

LLaVA (Liu et al., 2023) crystallized the mid-fusion recipe that dominates open-source multimodal LLMs in 2024-2026:

1. A frozen vision encoder (CLIP ViT-L/14) extracts patch embeddings from the input image, typically 576 tokens at 24x24 patches.
2. A small projection layer (a 2-layer MLP) maps those 576 tokens from the vision encoder's hidden dim into the LLM's embedding dim.
3. The projected tokens are prepended to the text token sequence and fed into a frozen or fine-tuned LLM (Vicuna, Mistral, Qwen-2).

The forward pass can be written compactly as

where $\mathbf{I}$ is the input image, $\mathbf{T}$ is the text token sequence, $\mathrm{ViT}$ is the frozen vision encoder, and $\phi$ is the trainable 2-layer MLP projector that bridges the vision and language embedding spaces. Only $\phi$ is trained in LLaVA's first stage; the LLM is fine-tuned (often with LoRA) in the second stage.

# Mid-fusion forward pass: vision encoder + projection + LLM.
import torch
import torch.nn as nn

class MidFusionVLM(nn.Module):
    def __init__(self, vision_encoder, llm, hidden_dim=4096):
        super().__init__()
        self.vision = vision_encoder      # CLIP ViT-L, frozen
        self.llm = llm                     # Mistral-7B, fine-tuned
        self.projector = nn.Sequential(
            nn.Linear(vision_encoder.hidden_dim, hidden_dim),
            nn.GELU(),
            nn.Linear(hidden_dim, hidden_dim),
        )

    def forward(self, image, text_token_ids):
        # 1. Encode image as 576 visual tokens of vision_encoder.hidden_dim
        with torch.no_grad():
            visual = self.vision(image).last_hidden_state
        # 2. Project into LLM's embedding space
        visual_proj = self.projector(visual)
        # 3. Embed text tokens
        text_emb = self.llm.embed_tokens(text_token_ids)
        # 4. Concatenate: [visual_tokens, text_tokens]
        combined = torch.cat([visual_proj, text_emb], dim=1)
        # 5. Run through the LLM transformer stack
        return self.llm.transformer(inputs_embeds=combined)

Variants of the basic recipe include:

• Perceiver Resampler (Flamingo, Alayrac et al. 2022; Idefics): a fixed number of learned latent queries (typically 64) cross-attend to the raw vision features and emit a fixed-size summary. Flamingo then injects this summary into a frozen LLM via interleaved gated cross-attention blocks rather than direct token concatenation. The Resampler predates Q-Former and inspired it.
• Q-Former (BLIP-2, Li et al. 2023): a Perceiver-style resampler with learnable query tokens that compresses the vision features to 32 tokens, but additionally trained with image-text matching, image-text contrastive, and image-grounded text generation losses before being attached to the LLM. The compressed tokens are then concatenated into the LLM input (not cross-attended like Flamingo).
• Dynamic resolution (Qwen2-VL): the vision encoder produces a variable number of tokens depending on input resolution, letting the model handle high-res images without truncating detail.

22.7.4 Capability Trade-offs

22.7.5 Fusion and Generation

Fusion choice affects not just understanding but generation. To generate an image, a mid-fusion model must either:

• Emit special "image tokens" that a downstream diffusion model decodes (GILL, MGIE).
• Output a CLIP-aligned latent that a frozen image generator consumes (BLIP-Diffusion).
• Call out to a separate image-generation tool via function-calling (the early ChatGPT + DALL-E 3 architecture).

An early-fusion model can in principle emit image tokens directly through the same autoregressive head that emits text. Chameleon does this; GPT-4o's image generation appears to use a hybrid (early-fusion text+image tokens at the input, with a diffusion model handling the actual pixel synthesis on the output side). The trade-off is the tokenizer fidelity bottleneck mentioned earlier.

22.7.6 Evaluation Pitfalls

Benchmarks for multimodal models often reward modality alignment more than cross-modal reasoning. MMMU and MMBench, for example, can be partially solved by models that just project image features into an LLM's input, with limited cross-modal reasoning. To stress-test fusion depth, look for benchmarks that require:

• Visual grounding (RefCOCO, Visual7W): the model must point at a region described in text.
• Cross-modal arithmetic (ChartQA, MathVista): the answer requires multi-step reasoning over numerical content in an image.
• Compositional generation (T2I-CompBench, GenAI-Bench): the model must produce an image faithful to a complex text prompt.
• Long-form interleaved (Mantis, MMDU): the model must handle dozens of images interleaved with text across thousands of tokens.