Text-to-Speech: VITS, Bark, and F5-TTS

"A voice is a fingerprint with a heartbeat. Neural TTS in 2026 prints that fingerprint from a five-second sample and forges the heartbeat to match."

Echo, Pitch-Perfect AI Agent

20.1.1 From Tacotron to End-to-End Neural TTS

Modern TTS has three jobs: convert text into a phonetic plan, predict an acoustic feature sequence (typically a mel-spectrogram or a stream of audio codec tokens), and turn that feature sequence into a waveform. Before 2021, each job was a separate neural network. Tacotron 2 (Shen et al., 2018) handled text-to-spectrogram with an attention-based seq2seq, and a WaveNet or HiFi-GAN vocoder turned the spectrogram into 16-bit PCM samples at 22 or 24 kHz. The cracks in this pipeline showed at the seams: training data mismatch between modules, exposure bias when the spectrogram predictor saw its own outputs at inference, and a long tail of failure modes where the attention would skip or repeat words.

The end-to-end shift fused these stages. VITS (Kim et al., 2021) trained the text encoder, the duration predictor, the latent z-sequence, and a HiFi-GAN-style decoder jointly using a variational lower bound plus an adversarial loss. The model never explicitly emits a spectrogram at inference; the latent z passes straight into the GAN-trained decoder, and you read waveform out the other end. This eliminated the mismatch problem and pushed the mean opinion score (MOS) within 0.1 of natural recordings on LJSpeech.

20.1.2 VITS: The Flow-Based End-to-End Baseline

VITS (Variational Inference with adversarial learning for end-to-end Text-to-Speech) is the architectural ancestor of most open-source neural TTS in 2026. It treats speech generation as a conditional VAE problem with four core modules: a text encoder that produces a sequence of phoneme-level latents, a stochastic duration predictor that decides how long each phoneme should last, a normalizing flow that transforms the text-conditioned prior into a more expressive distribution over the latent z (so that the prior can match the richer posterior), and a HiFi-GAN decoder that maps z directly to waveform. The clever piece is the monotonic alignment search (MAS): a non-differentiable dynamic-programming routine inside the training loop that finds the best phoneme-to-frame alignment, eliminating the need for an external aligner like Montreal Forced Aligner.

The training objective combines a reconstruction loss on the waveform (mel-spectrogram L1 plus multi-period and multi-scale discriminators), a KL divergence between the prior and the posterior over z, and a duration loss over the phoneme-level durations. The result is a single network that takes characters or phonemes in and produces 24 kHz audio out, trainable on a single GPU in a few days on a dataset like LJSpeech.

The VITS training objective is a conditional variational lower bound augmented with HiFi-GAN adversarial and feature-matching losses. Writing $x$ for the waveform, $c$ for the phoneme sequence, $z$ for the latent acoustic representation, $q(z \mid x)$ for the posterior encoder, and $p(z \mid c)$ for the text-conditional prior (text encoder followed by a normalizing flow), the loss is

where the first term is the HiFi-GAN reconstruction term, the KL pulls the text-conditional prior toward the posterior given the speech, $\mathcal{L}_\mathrm{dur}$ trains the stochastic duration predictor (a normalizing flow on phoneme durations), and the last two are the adversarial and feature-matching losses on the synthesized waveform $\hat{x}$. The KL is computed with respect to the monotonic alignment search (MAS) result, which finds the discrete alignment between phonemes and latent frames that maximizes the prior likelihood given the posterior sample. Because MAS is non-differentiable, the discrete alignment is treated as a stop-gradient hard target and gradients flow only through $q$ and the flow.

from TTS.api import TTS
import torch

# Pretrained VITS on LJSpeech (single English speaker, 22 kHz).
device = "cuda" if torch.cuda.is_available() else "cpu"
tts = TTS(model_name="tts_models/en/ljspeech/vits", progress_bar=False).to(device)

# Direct waveform synthesis: text in, 22 kHz float waveform out.
wav = tts.tts(text="The quick brown fox jumps over the lazy dog.")
print(len(wav), "samples", f"({len(wav) / 22_050:.2f} s at 22 kHz)")
# 71680 samples (3.25 s at 22 kHz)

tts.tts_to_file(text="VITS skips spectrograms entirely.", file_path="out.wav")

# Multi-speaker variant: pick speaker by name from the speaker list.
vctk = TTS("tts_models/en/vctk/vits", progress_bar=False)
vctk.tts_to_file(text="Hello world.", speaker="p225", file_path="p225.wav")

20.1.3 Bark and Codec Language Models

Suno AI's Bark (released April 2023) sits in a different lineage. Instead of a VAE plus flow, it is a stack of transformer decoders that autoregress over audio codec tokens. Bark uses EnCodec at 24 kHz with 8 residual codebooks, producing 75 tokens per second per book, or 600 tokens per second of audio total. The model decomposes this into three stages: a semantic transformer that predicts a coarse stream of semantic tokens (derived from a quantized HuBERT-like model), a coarse-acoustic transformer that predicts the first two EnCodec codebooks conditioned on the semantic stream, and a fine-acoustic transformer that fills in codebooks 3 through 8 conditioned on the first two. The fine model is non-autoregressive across codebooks but autoregressive across time.

This staged decomposition matters because audio is exquisitely high-dimensional. A naive flat autoregression over 600 tokens per second of 24 kHz speech would consume the entire context window of GPT-4 in 13 seconds of audio. Splitting semantics from acoustics, and coarse from fine, lets each stage operate at a manageable rate.

Figure 20.1.3 unrolls the residual chain: each codebook quantizes what the previous codebooks could not capture, so the running residual $r_k = r_{k-1} - e_k$ shrinks from coarse spectral envelope down to fine high-frequency detail across the $K$ stages.

20.1.4 F5-TTS: Flow Matching Meets TTS

F5-TTS (Chen et al., 2024, arXiv:2410.06885) is the 2024 state of the art in zero-shot open-weight TTS. It abandons both VAE and codec-LM lineages in favor of a flow-matching transformer that denoises mel-spectrograms conditioned on text. The "F5" in the title stands for "A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching" (five F-words; the authors lean into the pun). It is a 336M-parameter DiT (Diffusion Transformer; see Section 33.1 for the video DiT variant) trained on the Emilia multilingual dataset (~95k hours).

The architectural simplification is striking. F5-TTS has no phoneme tokenizer, no duration predictor, no alignment search, and no vocoder beyond an off-the-shelf BigVGAN. Text is byte-level. Audio conditioning at inference (the reference voice clip) is concatenated with masked target frames, and the DiT learns to fill in the missing mel frames given the text. Inference is 16 to 32 flow-matching steps, similar in cost to Stable Diffusion 3 at the same parameter count. The model handles zero-shot voice cloning from a 5-second prompt out of the box, code-switches between languages, and runs at roughly 0.15 real-time factor on a single A100.

# Synthesizing speech with Bark (Suno-AI's open-source codec LM)
# pip install bark
from bark import SAMPLE_RATE, generate_audio, preload_models
from scipy.io.wavfile import write as write_wav

# Download and cache the three transformer stages (semantic, coarse, fine).
# First call pulls ~5 GB; subsequent calls are local.
preload_models()

# Bark supports inline tags: [laughs], [sighs], [music], (whispering),
# as well as multilingual text and a small set of curated "history prompts".
prompt = (
    "Hello, my name is Suno. And, uh, I like pizza. [laughs] "
    "But I also like other things, like reading books and watching films."
)

audio_array = generate_audio(prompt, history_prompt="v2/en_speaker_6")
write_wav("bark_output.wav", SAMPLE_RATE, audio_array)
print(f"Generated {len(audio_array) / SAMPLE_RATE:.1f} seconds at {SAMPLE_RATE} Hz")

20.1.5 Prosody, Multi-Speaker, and Emotion Control

Naturalness is no longer the bottleneck; expressivity is. The 2024-2026 generation of TTS competes on prosody (pitch contour, stress, pacing), emotion (calm, angry, sad, excited), and speaker fidelity in multi-speaker contexts. There are four main control mechanisms in production.

The first is speaker embeddings, where a separate speaker-verification network (a wav2vec-XLSR or a similar self-supervised model) produces a 256-to-512-dimensional vector that conditions the TTS decoder. This is how VITS-2 and YourTTS extend single-speaker models to multi-speaker zero-shot synthesis. The second is reference-prompt conditioning, where a clip of target audio is concatenated with the input mel and the model learns to copy the style; F5-TTS, VALL-E, and NaturalSpeech 3 all use this trick. The third is natural-language style prompts: PromptTTS and Audiobox accept text like "an angry old man speaking quickly with a slight British accent" alongside the content text. The fourth is fine-grained controls exposed at inference: ElevenLabs v3 ships an "emotion" slider and a "stability" slider that act as classifier-free guidance weights on internal style channels.

20.1.6 The 2026 TTS Landscape: Open vs. Commercial

The commercial-open gap on raw audio quality is narrower than on most other modalities, but production reliability still favors the API vendors. Figure 20.1.4 summarizes the field as of late 2025.

# Comparison: F5-TTS zero-shot synthesis (open weights, mel + flow matching)
# pip install f5-tts
from f5_tts.api import F5TTS

f5 = F5TTS(
    model_type="F5-TTS",            # or "E2-TTS" for the smaller variant
    vocoder_name="vocos",           # 24 kHz, BigVGAN also supported
)

# Reference clip + its transcript anchor the voice and prosody.
# Five seconds of clean speech is enough; longer hurts more than it helps.
ref_audio = "alice_5sec.wav"
ref_text = "Hello, this is Alice and I am testing the synthesis system."

target_text = (
    "The flow matching denoiser inside F5-TTS predicts the velocity "
    "of the mel-spectrogram, not the spectrogram directly."
)

wav, sr, _ = f5.infer(
    ref_audio=ref_audio,
    ref_text=ref_text,
    gen_text=target_text,
    nfe_step=32,                     # number of flow-matching steps
    cfg_strength=2.0,                # classifier-free guidance scale
    speed=1.0,
)
import soundfile as sf
sf.write("f5_output.wav", wav, sr)
print(f"Generated {len(wav) / sr:.1f}s at sr={sr}")

20.1.7 Prosody from an LLM: The Emerging Pattern

A 2025 architectural trend that will dominate 2026 is using an LLM to generate the prosody plan and only delegating the acoustics to the TTS model. The pipeline looks like: (1) GPT-4o or Claude takes the text and an instruction ("read this as if you are calming a child after a nightmare"), and emits the same text annotated with SSML-like tags for pacing, emphasis, and emotion shifts; (2) a smaller TTS model (Cartesia or F5-TTS) executes that prosody plan. The factorization is appealing because the LLM already knows narrative pacing, sarcasm, and dialogue structure from its pretraining corpus, while the acoustic model has a much narrower job. Audiobox (Meta, 2023) and Voicebox (Meta, 2023) prototyped this with end-to-end joint training, but the production pattern of the API era is to keep the LLM and the TTS as separate services and rely on a structured intermediate representation between them.