Any-to-Any Generation

"A model that can describe a sound effect should be able to make one. A model that can recognize a melody should be able to hum it back."

Pixel, Any-To-Any AI Agent

22.8.1 The Encoder-LLM-Decoder Pattern

The dominant any-to-any recipe, used by NextGPT (Wu et al., 2023), Unified-IO 2 (Lu et al., 2023), and CoDi-2 (Tang et al., 2024), looks like this:

1. Input encoders: each modality has its own encoder (ImageBind, CLAP, BEATs, ViT). Encoders produce embeddings projected into the LLM's embedding space, the mid-fusion pattern from Section 22.7.
2. LLM core: a text LLM (Vicuna, Mistral, Llama) that processes the unified input embedding sequence and emits an output embedding sequence containing both text tokens and special "modality tokens" that signal a generation request.
3. Output decoders: when the LLM emits a modality token, the corresponding decoder (Stable Diffusion for images, AudioLDM for audio, Zeroscope for video) consumes a learned latent the LLM produces and synthesizes the output.

The architecture is intentionally modular: each piece (vision encoder, LLM, image decoder) is pretrained independently, and only thin alignment layers are trained jointly. Training cost is a fraction of a frontier omni model's, which is why this pattern dominated 2023-2024 academic work.

22.8.2 NextGPT and the Alignment Tokens

NextGPT (Wu et al., 2024) introduced the trick of generation alignment tokens. Special tokens like [IMG] and [/IMG] mark spans in the LLM's output that should be interpreted as image-generation requests. The hidden states at those positions are projected into a fixed-size latent that becomes the conditioning input to a frozen Stable Diffusion decoder.

# Schematic of NextGPT-style any-to-any inference.
# The LLM emits special tokens that route generation
# to modality-specific decoders.
def any_to_any_generate(model, prompt):
    output_tokens, output_hidden = model.llm(prompt)
    results = []
    i = 0
    while i < len(output_tokens):
        tok = output_tokens[i]
        if tok == "[IMG]":
            # Find matching close tag; pool the hidden states inside
            j = output_tokens.index("[/IMG]", i)
            latent = model.image_projector(output_hidden[i:j].mean(dim=0))
            img = model.stable_diffusion(cond_latent=latent)
            results.append(("image", img))
            i = j + 1
        elif tok == "[AUD]":
            j = output_tokens.index("[/AUD]", i)
            latent = model.audio_projector(output_hidden[i:j].mean(dim=0))
            aud = model.audio_ldm(cond_latent=latent)
            results.append(("audio", aud))
            i = j + 1
        else:
            results.append(("text", tok))
            i += 1
    return results

22.8.3 AnyGPT: Discrete Tokens Across All Modalities

AnyGPT (Zhan et al., 2024) takes the early-fusion route to its logical end. Every modality is tokenized into a discrete vocabulary:

• Text: byte-pair encoding.
• Image: SEED-Tokenizer, a 256-token codebook per 256x256 image.
• Audio: SpeechTokenizer, a residual-VQ codec with 8 codebooks per frame.
• Music: Encodec, RVQ-style codec.

The model is a single autoregressive transformer trained on interleaved sequences of these discrete tokens. Inputs and outputs use the same vocabulary; the model picks which to emit based on conditioning. Training data is constructed by pairing text descriptions with tokenized media (LAION images, AudioCaps audio, MusicCaps music).

The pure-discrete approach is elegant but lossy: the tokenizers introduce a quality ceiling. AnyGPT compensates by chaining a high-quality continuous decoder after the discrete output, the discrete tokens act as a semantic plan that a downstream diffusion model fleshes out. This is the same pattern that GPT-4o image generation appears to use.

22.8.4 Training Data: The Bottleneck

The reason any-to-any models lag specialist models on every individual modality is data. A specialist image generation model trains on billions of (caption, image) pairs. A specialist audio generation model trains on millions of (caption, audio) pairs. Any-to-any models need cross-modal data: examples where the input and output are different modalities. This data is rare:

• Text-image pairs: LAION-5B and successors. Plentiful.
• Text-audio pairs: AudioCaps, Clotho. Hundreds of thousands of examples.
• Image-audio pairs: VGGSound, AudioSet (with video frames). Millions but noisy.
• Audio-image pairs: rare; usually mined from video.
• Cross-modal instruction data: synthesized by prompting strong models (GPT-4o, Gemini). Quality varies.

The 2025 trick: use frontier any-to-any models to generate synthetic cross-modal pairs that train the next generation. This is the same flywheel that powered text instruction data via Self-Instruct and is the closest thing the field has to a scaling lever.

22.8.5 Quality and the Frontier Gap

Open-source any-to-any models (NextGPT, AnyGPT, CoDi-2) produce output noticeably weaker than dedicated single-modality models on every axis: their images are blurrier than SD3, their audio is less natural than ElevenLabs, their video is jerkier than Veo 3 or Sora 2. The gap closes for frontier proprietary omni models (Gemini 2.0, GPT-4o, Llama-4-Omni) because they get to train on the entire pretraining corpus and have access to internal modality-specific models for distillation.

In 2026, the practical pattern remains: use any-to-any models for their cross-modal reasoning, fall back to specialist models when you need maximum quality on a single modality. The hybrid pattern from Section 22.6 applies recursively, even within an any-to-any pipeline.

22.8.6 Tool-Calling as an Alternative

A pragmatic 2025 alternative to a single any-to-any model is an LLM agent that calls modality-specific tools. The text LLM is the orchestrator; specialized generation models are invoked via function calling. ChatGPT with DALL-E 3, Claude with image-generation tools, and most production "AI assistants" use this pattern. It is the limit of the pipeline approach from Section 22.6, extended to many modalities.

The advantage: each tool is independently best-in-class. The disadvantage: the orchestrator's cross-modal reasoning is limited to whatever passes through the tool-call interface (typically text), so paralinguistic and fine-grained visual cues are lost. This trade-off, as throughout this chapter, depends on whether your product needs cross-modal reasoning or just orchestration.