Music Generation: MusicLM, MusicGen, Suno, and Udio

"In 2023 we were thrilled when a neural model could produce 30 seconds of coherent indie folk. In 2025 Suno v5 ships full three-minute pop singles with verse-chorus structure, intelligible lyrics, and a hook you cannot get out of your head. The state of the art moves in months, not years."

Echo, Melodically-Inclined AI Agent

20.3.1 Audio Codecs: The Tokenizer of Music

Raw 44.1 kHz stereo audio is 1.4 megabits per second; a four-minute song is 40 megabytes. No transformer is going to autoregress over 88,000 floating-point samples per second of audio. The bridge between waveforms and language models is the neural audio codec: a VQ-VAE-style encoder-decoder that compresses audio to a sequence of discrete tokens at a manageable rate (typically 50-75 tokens per second per residual codebook, with 4-32 codebooks stacked).

Think of a codec as the audio analog of subword tokenization for text. A text tokenizer converts "transformers" into something like ["transform", "ers"] at roughly 1 token per 4 characters. An audio codec converts a 16-bit waveform into a sequence of discrete codebook indices at roughly 1 token per 320 input samples. Both replace a high-bitrate raw signal (UTF-8 bytes or PCM samples) with a much shorter sequence of integers drawn from a fixed vocabulary, so an LLM can autoregress over them with manageable context length and quadratic-attention cost.

Three codecs dominate the field, each from a different lab and each with a distinct sweet spot:

• EnCodec (Defossez et al., Meta, 2022) was the first widely used neural codec. It operates at 24 kHz with 8 residual codebooks at 75 Hz, giving 600 tokens per second of audio.
• SoundStream (Zeghidour et al., Google, 2021) preceded EnCodec and uses a similar residual-VQ structure. MusicLM uses SoundStream tokens directly.
• DAC (Descript Audio Codec, Kumar et al., 2023) supports 44.1 kHz stereo and is the codec of choice for music-quality systems. It produces about 86 tokens per second per codebook with 9 codebooks. Suno and Udio are widely believed to use proprietary codecs derived from DAC.

20.3.2 MusicGen: The Open-Source Baseline

MusicGen (Copet et al., Meta, 2023, arXiv:2306.05284) is the reference open-source music generator. It comes in 300M, 1.5B, and 3.3B parameter sizes. The architecture is a single transformer decoder that autoregresses over EnCodec tokens at 32 kHz, conditioned on text via a frozen T5 encoder (or melody via a chromagram for the melody-conditioned variant). The clever piece is the "delay pattern" interleaving: instead of generating one full codebook then the next, MusicGen interleaves codebooks with offset delays so the model never has to predict more than one codebook in a single forward pass, sharply reducing the effective vocabulary at each step and the cost of long-context modeling.

MusicGen at 3.3B can produce 30-second clips that are clearly identifiable as the prompted genre, with discernible chord progressions and rhythm. It struggles with vocals, song structure beyond a single section, and long-term coherence. As a baseline for fine-tuning or as a teaching artifact for "how does this work," it is the model to study; as a finished product, it is two years behind the commercial frontier.

# MusicGen 30-second clip via the AudioCraft library
# pip install audiocraft
from audiocraft.models import MusicGen
from audiocraft.data.audio import audio_write

# 3.3B parameter checkpoint; fits in 12 GB of VRAM at fp16.
model = MusicGen.get_pretrained("facebook/musicgen-large")
model.set_generation_params(
    duration=30,      # 30-second clip
    top_k=250,        # MusicGen-recommended sampling
    top_p=0.0,        # top_k only
    temperature=1.0,
    cfg_coef=3.0,     # classifier-free guidance
)

prompts = [
    "an upbeat 90 BPM lo-fi hip-hop track with a warm Rhodes piano "
    "and a vinyl crackle texture, perfect for late-night studying",

    "a cinematic orchestral piece in C minor with a soaring french horn "
    "melody over pulsing strings, building to a triumphant climax",
]

wavs = model.generate(prompts)     # shape: (batch, channels, samples)
for idx, wav in enumerate(wavs):
    audio_write(f"clip_{idx}", wav.cpu(), model.sample_rate,
                strategy="loudness", loudness_compressor=True)
    print(f"Wrote clip_{idx}.wav at {model.sample_rate} Hz")

20.3.3 MusicLM and the Hierarchical Semantic-Acoustic Pattern

MusicLM (Agostinelli et al., Google, 2023, arXiv:2301.11325) predates MusicGen by a few months and uses a more elaborate hierarchical decomposition. It cascades three models: SoundStream produces acoustic tokens at 600 Hz, w2v-BERT produces "semantic" tokens at 25 Hz representing high-level musical structure, and MuLan (a text-music CLIP-style joint embedding model) produces conditioning tokens from the text prompt. The pipeline generates MuLan tokens (text -> joint embedding), then semantic tokens (high-level structure), then acoustic tokens (audio-quality SoundStream codes), and finally decodes to waveform. The structural decomposition is the same idea Bark uses for speech (Section 20.1), and it shows up again in Suno's reported pipeline.

Google never released MusicLM weights, but the architecture is documented and recipes for hierarchical music generation (Open MuLan, the AudioLM lineage) are public. The hierarchical pattern matters because pure end-to-end token generation over 30 seconds of 44.1 kHz audio requires very long contexts and very expensive training; factoring the problem into semantic + acoustic layers makes both parts tractable.

The hierarchical factoring is best read as a chain rule on the joint probability. Writing $m$ for the MuLan text embedding, $s_{1:T_s}$ for the semantic-token sequence (length $T_s$ at 25 Hz), and $a_{1:T_a}$ for the acoustic-token sequence (length $T_a$ at 600 Hz), MusicLM models

At inference the sum is approximated by a single sampled draw of $s_{1:T_s}$ from $p_\phi$ followed by a sampled draw of $a_{1:T_a}$ from $p_\theta$. The two transformers train independently: $p_\phi$ on (semantic token, MuLan embedding) pairs derived from MuLan's frozen text-music encoder, and $p_\theta$ on (acoustic, semantic) pairs derived from the same training clips. Factoring through the 24x-shorter semantic stream is what makes long-range musical structure (key, tempo, instrument layout) learnable: the acoustic transformer never has to invent that structure from a text prompt directly, it just decorates the structure the semantic transformer already laid down.

import torch, scipy.io.wavfile
from transformers import MusicgenForConditionalGeneration, AutoProcessor

# Google never released MusicLM weights. MusicGen is the open-source replacement
# that uses the same codec-LM recipe (single transformer over EnCodec tokens).
processor = AutoProcessor.from_pretrained("facebook/musicgen-small")
model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-small")
model = model.to("cuda" if torch.cuda.is_available() else "cpu").eval()

prompts = [
    "calm acoustic guitar with light percussion, 90 BPM",
    "upbeat 80s synthwave with arpeggiated bass, 120 BPM",
]
inputs = processor(text=prompts, padding=True, return_tensors="pt").to(model.device)

with torch.no_grad():
    audio = model.generate(**inputs, max_new_tokens=512, do_sample=True,
                           guidance_scale=3.0)
# audio shape: (batch=2, 1, ~10.24 s at 32 kHz)

scipy.io.wavfile.write("guitar.wav",    rate=32_000, data=audio[0, 0].cpu().numpy())
scipy.io.wavfile.write("synthwave.wav", rate=32_000, data=audio[1, 0].cpu().numpy())

20.3.4 Suno, Udio, and AIVA: The Commercial Frontier

Suno is the consumer leader. Suno V3 (March 2024) was the first system to generate full songs with vocals, lyrics, and recognizable verse-chorus structure under user control. V4 (late 2024) added longer outputs (up to 4 minutes) and better lyric intelligibility. V5 (October 2025) ships with stem outputs (separate vocal, drums, bass, and instrument stems), genre interpolation between two reference prompts, and an extend mode that adds a chorus or bridge to existing user-provided audio. Suno's reported architecture is a hierarchical codec LM (semantic + acoustic, plus a separate text-to-lyric model) trained on tens of thousands of hours of licensed music, with proprietary codec improvements.

Udio (April 2024) is the closest competitor. Udio V1 launched within weeks of Suno V3 and made comparable claims; V3 (early 2025) added precise prosody control over lyrics, instrumentals-only mode, and a "remix" feature where the user supplies a melodic motif. Udio's audio quality at the high end is often judged equal to Suno's; the systems differ more in default style biases (Udio leans cleaner and more pop-produced; Suno leans grittier and more genre-promiscuous).

AIVA (Artificial Intelligence Virtual Artist) is the longest-running commercial music AI, dating to 2016. AIVA targets composers and film scorers rather than consumers: it ships symbolic MIDI output that the user can edit in a DAW, multi-instrument scores in classical and cinematic styles, and royalty-free licensing terms designed for media production. AIVA is the right tool when you need editable score output; Suno and Udio are right when you need finished audio.

20.3.5 Conditioning: Text, Melody, and Reference Audio

Music generation systems support four conditioning modalities, ordered from universal to controversial:

• Text prompts are the universal interface. The model is trained with paired text-music data (LP captions, Spotify metadata, expert tags from MusicCaps).
• Melody conditioning takes a chromagram or a humming clip and generates a full arrangement that follows the user's melodic intent. MusicGen-melody, Suno's "extend with this idea" mode, and Udio's remix feature all use this.
• Reference audio conditioning passes a short clip of an existing song and asks the model to produce something stylistically similar. This is the most controversial mode because of the obvious copyright implications, and the commercial APIs add safeguards that prevent verbatim cloning of known recordings.
• Lyric conditioning is unique to vocals-capable systems (Suno, Udio). The user provides text and the model synthesizes vocals that pronounce those lyrics over the generated instrumental.

20.3.6 The Rights and Training Data Question

Every commercial music generator faces the same legal question: what music was it trained on, and what license terms apply to the outputs? The 2024 lawsuits (RIAA suing Suno and Udio, separately) argue that both systems were trained on unlicensed copies of major-label recordings, that the outputs can be made to sound like specific artists, and that this constitutes infringement at training time and at inference time. Suno and Udio's defenses center on fair use of training data and substantial transformation of outputs. The lawsuits are unresolved as of late 2025, and they will set precedent for the entire music AI category.

The technical countermeasures the platforms have adopted fall into four categories:

• Training-data filtering to remove known commercial recordings (claimed by both companies, hard to verify).
• Output-side filtering that blocks generations matching the audio fingerprint of any known recording (Suno added this in V4).
• Opt-out registries for artists who do not want their style imitated (Suno's "Studio" program).
• Revenue-sharing pilots with select labels (Udio's 2025 deal with Beatport for electronic music).

None of these resolves the underlying training-data question, but together they shape the product into something defensible in court.

20.3.7 Production Music and the Mid-Tier Stack

Below the consumer-finished Suno/Udio tier and above the research-baseline MusicGen tier sits the production-music middle: Stability AI's Stable Audio 2.5 (latent diffusion over audio, 44.1 kHz, up to 3 minutes), Splice's CoSo (loop-and-stem generation aimed at producers in DAWs), and Soundraw and Boomy (consumer-facing services that license out-of-the-box royalty-free tracks). Stable Audio in particular is interesting because it uses diffusion rather than codec-LM generation; the underlying autoencoder produces a latent representation at 21 Hz over which a latent DiT denoises. This is closer to image diffusion than to text-to-speech generation, and it leads to qualitatively different outputs (Stable Audio is strong at ambient, electronic, and sound-design textures; weaker at lyric-driven pop).

The mid-tier matters for production because it ships under licenses that commercial users find acceptable (full ownership of outputs, no per-use royalty), and because it integrates with DAWs (Logic Pro, Ableton Live, FL Studio) via plugins. A 2026-era audio professional typically uses several systems in a single project: Suno for first-draft songs, Stable Audio for ambient beds, AIVA for editable orchestral cues, and a human producer to glue it all together.