Audio Editing: Stems, Style Transfer, and Remixing

"Generation gets the headlines; editing pays the rent. The unsexy work of cleaning, separating, repairing, and remixing audio is where most production AI dollars actually go."

Echo, Audio-Editing AI Agent

20.4.1 Stem Separation: Demucs, MDX-Net, and the Music Source Separation Frontier

Stem separation (or music source separation, MSS) takes a stereo mixed track and outputs separate channels for the constituent sources: typically vocals, drums, bass, and "other" (everything else). Before 2020 this was a heuristic-driven mess (Wiener filtering, REPET, spectral masking); after 2020 it became a neural problem with reliably superhuman quality for the vocal track. The 2024 state of the art is a small set of related architectures: Demucs v4 (Defossez et al., Meta, 2022 with successive refinements), MDX-Net (Kim et al., Sony, 2021-2024, evolution of the model that won MDX 2021), and the BS-RoFormer family (Lu et al., 2024) that recently set new benchmarks on the MUSDB18 test set.

Demucs v4 (the "Hybrid Transformer Demucs" or "htdemucs" variant) is the open-source default. It is a U-Net with a hybrid time-domain/spectrogram representation: the bottom of the U operates on waveforms with 1D convolutions and self-attention, while a parallel branch processes spectrograms with 2D convolutions; the two branches share information at every layer. Training is supervised on MUSDB18 plus large-scale internal data; the model emits four 44.1 kHz stereo stems given a mixed input. On modern hardware, htdemucs separates a 4-minute song in under 30 seconds on a single GPU.

The Separation Objective: SI-SDR

Naming Demucs and BS-RoFormer tells you which models to download but not what they are trained to maximize. A naive choice is mean-squared error between the estimated stem $\hat{s}$ and the reference $s$. That objective has a fatal flaw for audio: it is sensitive to overall gain. An estimate that recovers the vocal perfectly but at half the loudness is musically correct yet incurs a large MSE, so the loss would push the model to match absolute amplitude rather than waveform shape. The field's answer is the scale-invariant signal-to-distortion ratio (SI-SDR), which first projects the estimate onto the reference to absorb any gain mismatch, then measures the residual. Define the projection of the estimate onto the reference and the residual error as

$$ s_{target} = \frac{\langle \hat{s}, s\rangle}{\lVert s\rVert^2}\, s, \qquad e_{noise} = \hat{s} - s_{target}, $$

where $\langle \hat{s}, s\rangle$ is the inner product of the two waveforms. The term $s_{target}$ is the component of the estimate that lies along the reference (the part we want), and $e_{noise}$ is everything orthogonal to it (distortion, interference from other stems, and artifacts). The metric is the log-ratio of their energies:

$$ \text{SI-SDR} = 10\log_{10}\frac{\lVert s_{target}\rVert^2}{\lVert e_{noise}\rVert^2}. $$

Because $s_{target}$ rescales the reference by exactly the factor $\alpha = \langle \hat{s}, s\rangle / \lVert s\rVert^2$ that best matches $\hat{s}$, multiplying $\hat{s}$ by any constant leaves the ratio unchanged: that is the scale invariance, and it is why SI-SDR, not raw SDR or MSE, is the standard both as the reported metric on MUSDB18 and (negated, since training minimizes) as the training loss. Higher is better; a vocal stem in the high teens of decibels is excellent, while single digits are audibly degraded. As a micro-check, if $\hat{s} = 2s$ exactly, then $\alpha = 2$, $s_{target} = 2s = \hat{s}$, $e_{noise} = 0$, and SI-SDR diverges to $+\infty$: a doubled-but-otherwise-perfect estimate scores perfectly, exactly as scale invariance demands.

Mask-Based Separation and Band-Splitting

The Key Insight above framed separation as predicting a mask; here is the operation in symbols. Let $X \in \mathbb{C}^{F \times T}$ be the complex (or magnitude) short-time Fourier transform of the mixture, with $F$ frequency bins and $T$ time frames. For each source the model predicts a time-frequency mask $M$ of the same shape, and the estimated source spectrogram is the elementwise (Hadamard) product

$$ \hat{S} = M \odot X. $$

A binary mask sets $M_{f,t} \in \{0, 1\}$, assigning each cell entirely to one source; this is cheap but introduces musical-noise artifacts at cell boundaries where energy is shared. A soft mask lets $M_{f,t} \in [0, 1]$ (or unbounded complex values for phase-aware masks), so overlapping sources split the energy of a shared cell proportionally, which is what modern separators predict. Inverting $\hat{S}$ back to the time domain through the inverse STFT yields the stem that SI-SDR scores. The band-split idea behind BS-RoFormer and the hybrid Demucs branch refines this: rather than masking the full spectrogram with one network, split the frequency axis into contiguous sub-bands and process each with its own transformer or convolutional stack before recombining. This matches the non-uniform resolution music demands, since a bass line lives in a few low bins while cymbals spread across the entire high end, so narrow low-frequency bands and wide high-frequency bands each get capacity proportioned to their information content.

# Stem separation with Demucs v4 (Hybrid Transformer Demucs)
# pip install -U demucs
from demucs.api import Separator
import torch
import soundfile as sf

# htdemucs_ft is the four-stem fine-tuned model: vocals, drums, bass, other.
separator = Separator(
    model="htdemucs_ft",
    device="cuda" if torch.cuda.is_available() else "cpu",
    shifts=2,          # test-time augmentation; trades latency for quality
    split=True,        # chunk long inputs to control memory
)

# Returns a dict {"vocals": tensor, "drums": tensor, "bass": tensor, "other": tensor}
origin, separated = separator.separate_audio_file("song.wav")

for stem_name, tensor in separated.items():
    # tensor shape: (channels=2, samples)
    sf.write(
        f"stems/{stem_name}.wav",
        tensor.cpu().numpy().T,
        separator.samplerate,
    )
    print(f"  {stem_name:<8} -> stems/{stem_name}.wav  {tensor.shape[1] / separator.samplerate:.1f}s")

20.4.2 Audio Inpainting and Repair

Audio inpainting fills in missing or corrupted segments of a recording. Use cases include click and pop removal (vinyl transfers, old archive recordings), dropout repair (cellular call recordings), removing unwanted spoken words from podcast tracks ("um", profanity, name mentions), and reconstructing damaged historical recordings. Classical DSP handles short gaps (sub-50 ms) acceptably; longer gaps require generative models that fill in plausible content given the surrounding context.

The 2024 state of the art uses one of two recipes. The first is masked codec-LM infilling: encode the audio with EnCodec or DAC, mask the codec tokens covering the corrupted region, and use a bidirectional (BERT-style) or autoregressive transformer to fill them in. AudioLM-derived models and recent VALL-E variants use this pattern. The second is diffusion-based inpainting: the audio is processed through a diffusion model whose conditioning includes the surrounding clean context, and the model denoises the corrupted region back to a plausible reconstruction. Stable Audio Open supports this, as does Adobe's "Generative Audio Fill" feature in Premiere Pro (introduced 2024).

The qualitative line between repair and creative generation is fuzzy. Filling in a 100 ms click is "repair"; filling in a 5-second gap with newly generated music in the same style as the surrounding mix is "extension"; the underlying model is the same. The product framing depends on the gap length and the user's intent.

20.4.3 Style Transfer and Timbre Conversion

Audio style transfer takes a source audio (the "content") and reshapes it to match a target reference (the "style") while preserving the source's musical content. The taxonomy splits into instrument timbre transfer (drums in the style of these other drums; piano sounding like organ), voice timbre transfer (the source's words and prosody, with a different vocal character; Section 20.2 covered the cloning side), and ambient style transfer (a dry recording reshaped to sound like it was recorded in a cathedral, or through a guitar amp).

The 2024 approach that has displaced earlier WaveNet-and-CycleGAN methods is the same one driving image style transfer: a diffusion model conditioned on a style reference embedding. The model encodes the style reference with a pretrained audio embedding model (CLAP, AudioCLIP, or a music-specific equivalent), denoises the source audio guided by both the content reference and the style embedding, and outputs a hybrid that combines both. Stable Audio 2.5's "style transfer" mode and Suno's V5 "remix in genre" feature are productionized versions of this.

20.4.4 Pitch, Time, and Classical DSP with Neural Replacements

The classical audio DSP toolbox (pitch shift, time stretch, equalization, compression, reverb, denoising) is being progressively replaced by neural equivalents that produce subjectively better results, especially at extreme settings. Neural pitch shift (e.g., Audio Toolkit's NeuralPitch, Soundtoys Little AlterBoy) preserves formants and timbre across multi-octave shifts in a way that classical phase vocoders cannot. Neural time stretch (Antares ATR, iZotope Radius) avoids the metallic artifacts of phase-vocoder stretches at 4x or longer factors. Neural denoising (NVIDIA RTX Voice, Krisp, iZotope RX 11 Voice De-noise) removes background hum, hiss, traffic, and room reverb with quality and CPU efficiency that classical Wiener filtering cannot approach.

The training pattern is the same in all cases: collect paired data (clean source, processed target), train a U-Net or small transformer in time-frequency domain, and ship it as a real-time-capable processor. The interesting twist for 2025-2026 is that these models are getting small enough to ship as DAW plugins that run locally on a CPU; iZotope RX 11 ships with several million-parameter models that process in real time on a M2 MacBook.

20.4.5 The Remix Stack: From Stems to Published Track

A common 2026 production workflow takes an existing song, separates it into stems, applies neural processing to one or more stems, and remixes the result. The use cases include: a DJ extracting an acapella for a custom backing track, a producer replacing the drums in a sample with newly programmed drums, a film editor isolating dialogue from a TV broadcast for use in a documentary, or a karaoke service generating instrumentals from songs that never released official karaoke versions.

The legal status of these workflows is jurisdictionally inconsistent. The US considers extracting stems from a copyrighted recording and republishing them a derivative work that requires a master-recording license; the same is true in the UK and EU. Pure personal use is generally tolerated; commercial republication is not. The 2024 rise of consumer-grade stem separation (Moises, LALAL.AI, Spotify's release of stems from a small catalog of licensed tracks) is forcing the industry to develop new licensing models for stem-level distribution, which Suno V5 and Spotify Stem Library (announced 2025) are early productions of.