Audio, Music & Video Generation

"Give me a text prompt and I will compose you a symphony. Give me thirty seconds of your voice and I will speak in your likeness forever."

Pixel, Musically Inclined AI Agent

1. Text-to-Speech (TTS) Systems

Text-to-speech has undergone a revolution in the past few years. Traditional concatenative and parametric systems have been replaced by neural models that produce speech nearly indistinguishable from human recordings. The modern TTS pipeline typically involves a text encoder that converts input text to phonemes or tokens, an acoustic model that generates mel-spectrograms or audio tokens, and a vocoder that converts these representations into audible waveforms.

VITS: End-to-End Speech Synthesis

VITS (Variational Inference with adversarial learning for end-to-end Text-to-Speech) combines a variational autoencoder, normalizing flows, and adversarial training into a single end-to-end model. Unlike earlier two-stage approaches (text to spectrogram, then spectrogram to waveform), VITS generates raw audio directly from text, producing high-quality speech with natural prosody. It remains one of the most efficient architectures for real-time TTS. Code Fragment 27.2.5 below puts this into practice.

# Using Coqui TTS (open-source VITS implementation)
from TTS.api import TTS

# List available models
print(TTS().list_models())

# Load a VITS model for English
tts = TTS(model_name="tts_models/en/ljspeech/vits")

# Generate speech from text
tts.tts_to_file(
 text="Neural text-to-speech has made enormous progress in recent years.",
 file_path="output_vits.wav",
)

# Multi-speaker model with voice cloning
tts_multi = TTS(model_name="tts_models/multilingual/multi-dataset/xtts_v2")
tts_multi.tts_to_file(
 text="Voice cloning requires only a few seconds of reference audio.",
 file_path="cloned_output.wav",
 speaker_wav="reference_voice.wav", # 6+ seconds of target speaker
 language="en",
)

from openai import OpenAI
client = OpenAI()

response = client.audio.speech.create(
 model="tts-1-hd",
 voice="nova",
 input="Neural text-to-speech has made enormous progress in recent years.",
)
response.stream_to_file("output.mp3")

Bark: Generative Audio with Paralinguistics

Bark, developed by Suno, takes a different approach by modeling speech as a sequence of audio tokens using an autoregressive transformer (similar to how GPT models text). This token-based approach naturally handles not just speech but also laughter, music, background noise, and paralinguistic cues. Bark generates semantic tokens from text, converts them to coarse acoustic tokens, then refines them to fine acoustic tokens, with each stage handled by a separate transformer. Code Fragment 27.2.6 below puts this into practice.

# Bark: token-based speech generation with paralinguistic cues
# Supports laughter, emotion, and speaker presets through text markers
from transformers import AutoProcessor, BarkModel
import scipy

# Load Bark model
processor = AutoProcessor.from_pretrained("suno/bark")
model = BarkModel.from_pretrained("suno/bark")
model = model.to("cuda")

# Generate speech with paralinguistic cues
text = "Hello! [laughs] This is an example of Bark generating speech with emotion."

inputs = processor(text, voice_preset="v2/en_speaker_6")
inputs = {k: v.to("cuda") for k, v in inputs.items()}

audio_array = model.generate(**inputs)
audio_array = audio_array.cpu().numpy().squeeze()

# Save the generated audio
sample_rate = model.generation_config.sample_rate
scipy.io.wavfile.write("bark_output.wav", rate=sample_rate, data=audio_array)

F5-TTS and Zero-Shot Voice Cloning

F5-TTS represents the latest generation of TTS models built on flow matching (the same technique behind Flux for image generation). It uses a diffusion transformer (DiT) architecture to generate mel-spectrograms from text, conditioned on a reference speech sample. The flow matching approach enables fast, high-quality generation with natural prosody. F5-TTS achieves remarkable zero-shot voice cloning from as little as 3 seconds of reference audio, raising important safety and ethical questions (Chapter 32) about consent and misuse. It is one of the most accessible voice cloning systems available. Figure 27.2.1 shows the modern TTS architecture.

Real-Time Conversational Audio

GPT-4o introduced native audio input and output, meaning the model can listen, understand, and respond with natural speech in real time. Rather than the traditional pipeline of speech-to-text followed by LLM processing followed by text-to-speech, GPT-4o processes audio tokens directly within the transformer, preserving nuances like intonation, emotion, and speaking pace. This enables sub-200ms latency for conversational applications. Kyutai's Moshi follows a similar approach as an open-source alternative, using a multi-stream architecture that processes both the user's speech and its own generated speech simultaneously, enabling natural turn-taking and even interruption handling.

# Production-ready speech-to-text with faster-whisper
# Uses CTranslate2 backend for 4x speed improvement over original Whisper
from faster_whisper import WhisperModel

# Load the turbo variant for best speed/accuracy trade-off
model = WhisperModel("large-v3-turbo", device="cuda", compute_type="float16")

# Transcribe with VAD filter to skip silence
segments, info = model.transcribe(
 "meeting_recording.mp3",
 vad_filter=True, # Skip silent segments
 vad_parameters={"min_silence_duration_ms": 500},
 beam_size=5,
 word_timestamps=True, # Get per-word timing
)

print(f"Detected language: {info.language} (probability {info.language_probability:.2f})")
for segment in segments:
 print(f"[{segment.start:.1f}s - {segment.end:.1f}s] {segment.text}")

from openai import OpenAI
client = OpenAI()

with open("meeting_recording.mp3", "rb") as audio_file:
 transcript = client.audio.transcriptions.create(
 model="whisper-1",
 file=audio_file,
 response_format="verbose_json",
 timestamp_granularities=["word"],
 )
print(f"Language: {transcript.language}")
for word in transcript.words:
 print(f"[{word.start:.1f}s] {word.word}")

2. Music Generation

Music generation with AI has progressed from simple MIDI patterns to full song production with vocals, instrumentation, and complex arrangements. The core challenge is modeling long-range temporal structure: music has hierarchical patterns from individual notes (milliseconds) to phrases (seconds) to sections (minutes) that must all be coherent.

MusicLM and MusicGen

Google's MusicLM was the first model to generate high-fidelity music from text descriptions at 24kHz. It uses a hierarchical sequence-to-sequence approach: MuLan embeddings (a music-text joint embedding model) condition semantic tokens from w2v-BERT, which then condition acoustic tokens from SoundStream. Meta's MusicGen simplified this into a single autoregressive transformer that generates EnCodec audio tokens directly, conditioned on text or melody. MusicGen introduced efficient codebook interleaving patterns that allow generating multiple codec streams with a single model pass. Code Fragment 27.2.7 below puts this into practice.

# MusicGen: generate music from a text description using Meta's
# single-pass autoregressive transformer over EnCodec audio tokens.
from audiocraft.models import MusicGen
import torchaudio

# Load the medium MusicGen model (1.5B parameters)
model = MusicGen.get_pretrained("facebook/musicgen-medium")
model.set_generation_params(
 duration=10, # seconds of audio to generate
 top_k=250, # nucleus sampling parameter
 temperature=1.0,
)

# Generate from a text description
descriptions = ["upbeat jazz trio with walking bass and brushed drums"]
wav = model.generate(descriptions) # returns tensor [batch, channels, samples]

# Save as WAV file at 32kHz sample rate
torchaudio.save("jazz_trio.wav", wav[0].cpu(), sample_rate=32000)

Suno and Udio: Full Song Generation

Suno and Udio represent the current state of the art in full song generation, producing complete songs with vocals, instrumentation, and lyrics. These commercial systems accept text descriptions of genre, mood, and style, along with optional lyrics, and generate radio-quality tracks of 2 to 4 minutes. While the exact architectures are proprietary, they likely combine text-conditioned audio generation with separate vocal synthesis and mixing stages. The quality has reached a point where generated music is difficult to distinguish from human-produced tracks in blind listening tests.

3. Text-to-Video Generation

Text-to-video is arguably the most challenging generative modality, requiring the model to produce temporally coherent sequences of frames that are individually high quality and collectively tell a consistent visual story. A single second of 24fps 1080p video contains roughly 50 million pixels, compared to about 1 million for a single 1024x1024 image.

The core challenge of video generation is temporal coherence: each frame must look good on its own, but consecutive frames must also form a smooth, physically plausible motion sequence. An image generator can produce a beautiful still of a dog; a video generator must ensure that the dog's legs move correctly, that the background stays consistent as the camera pans, and that objects obey basic physics across frames. This is why video generation is orders of magnitude harder than image generation, and why the field lags behind by roughly 2 to 3 years in quality. The same architectural principles that power image generation (diffusion, transformers, flow matching) apply to video, but with the added dimension of time making everything more expensive and more difficult.

Architecture: Diffusion Transformers (DiT) for Video

Most modern video generation models extend the Diffusion Transformer (DiT) architecture to handle spatiotemporal data. Instead of processing a single image's latent representation, the model processes a 3D latent volume (height, width, time). Attention operates across both spatial and temporal dimensions, either through factored attention (separate spatial and temporal attention layers) or full 3D attention. The latent space comes from a video VAE that compresses frames both spatially and temporally. Figure 27.2.2 illustrates the Video DiT architecture.

Leading Video Generation Models

OpenAI's Sora demonstrated that scaling DiT architectures to video produces remarkable results, generating up to 60-second clips with consistent characters, realistic physics, and cinematic camera movements. Runway's Gen-3 Alpha focuses on controllable commercial video generation with features like motion brush and camera control. Kuaishou's Kling 2 achieves strong temporal consistency using a 3D VAE that compresses video in both space and time. Google's Veo 2 generates high-definition video with excellent prompt adherence and physical realism. Code Fragment 27.2.6 below puts this into practice.

# Using CogVideoX (open-source) via diffusers
import torch
from diffusers import CogVideoXPipeline
from diffusers.utils import export_to_video

pipe = CogVideoXPipeline.from_pretrained(
 "THUDM/CogVideoX-5b",
 torch_dtype=torch.bfloat16,
)
pipe.enable_model_cpu_offload()

prompt = "A golden retriever running through a sunlit meadow, slow motion, cinematic"

video = pipe(
 prompt=prompt,
 num_videos_per_prompt=1,
 num_inference_steps=50,
 num_frames=49,
 guidance_scale=6.0,
).frames[0]

export_to_video(video, "golden_retriever.mp4", fps=8)

4. 3D Generation

3D content generation from text or images is an emerging frontier. Current approaches include score distillation sampling (SDS), which uses a pre-trained 2D diffusion model to optimize a 3D representation (NeRF or Gaussian splatting) so that it looks correct from every viewing angle. DreamFusion pioneered this approach, and subsequent work like Instant3D and LRM (Large Reconstruction Model) has made 3D generation faster and more reliable.

Multimodal Composition Pipelines

Real-world applications often combine multiple generative modalities into a single pipeline. A film production tool might use an LLM to write a script, a video model to generate scenes, a TTS model for dialogue, and a music model for the soundtrack. Orchestrating these components requires careful attention to temporal synchronization, style consistency, and resource management. Frameworks like ComfyUI provide node-based interfaces for building such pipelines, while programmatic approaches use Python to chain models together. Figure 27.2.3 shows the multimodal composition pipeline.

Exercises