Audio Classification with CLAP and Supervised Fine-Tuning

"Audio classification has been commoditized. The work is no longer training a model from scratch; it is choosing which pretrained backbone (AST, wav2vec, Whisper, CLAP, HuBERT) and which downstream head fits the question."

Echo, Pitch-Perfect AI Agent

20.0.5.1 The Four Flavors of Audio Classification

Section 20.0 introduced the bipartite taxonomy at the chapter level. This section expands the classification leaf into the four flavors that drive most production deployments.

Content classification assigns broad acoustic categories: music versus speech versus environmental noise, podcast versus advertisement, vocal versus instrumental. The use case is routing: an upstream model decides which downstream pipeline (ASR, music tagger, sound event detector) receives the clip. The standard backbone is AST or a CLAP zero-shot classifier; the label set is small (3 to 10 classes).

Event classification labels short sound events: alarm, fire crackling, glass break, gunfire, doorbell, baby cry. The reference dataset is AudioSet (Gemmeke et al., 2017) with 527 classes drawn from a structured ontology, and the reference backbone is AST pretrained on AudioSet. A smaller domain-specific dataset is ESC-50 (50 environmental classes; Piczak, 2015) which is small enough to fit in memory and is the standard benchmark for evaluating new audio encoders. Production use cases include security monitoring (gunfire and glass break detection), accessibility (alarm transcription for deaf users), and content moderation (detecting inappropriate sounds in user-uploaded video).

Intent classification maps a spoken utterance to a discrete action label. The MINDS-14 dataset (Gerz et al., 2021) is the canonical example: 14 intent classes in a banking domain (pay_bill, transfer, card_issues, address, app_error, ...) across 14 languages. The reference recipe uses a wav2vec 2.0 backbone or its multilingual variant XLS-R, fine-tuned with a classification head. Intent classification is the building block of voice assistants and IVR systems.

Keyword spotting (KWS) detects a small closed vocabulary of trigger words: "stop", "play", "next", "OK Google". Because the vocabulary is bounded (Speech Commands has 35 classes; Warden, 2018) and the audio clips are short (1 second), KWS models can be tiny enough to run continuously on always-on microcontrollers. The reference backbone is AST fine-tuned on Speech Commands; smaller depthwise-separable CNN architectures are used for on-device deployment.

The companion recognition tasks are speaker identification (which person is speaking) and language identification (which language is being spoken). LangID typically uses a Whisper-based head; speaker ID uses WavLM or ECAPA-TDNN (a non-transformer architecture optimized for embedding speakers, popular in production).

20.0.5.2 Pretrained Recipes: One Line Each

For each flavor at least one curated HuggingFace checkpoint exists and the pipeline call is a single line. The reader's job is to pick the right model name; the inference loop is identical.

from datasets import load_dataset
from transformers import pipeline

# Speech Commands dataset; streaming avoids the full 1.5 GB download.
sc = load_dataset("speech_commands", "v0.02", split="validation", streaming=True)
example = next(iter(sc))
print(example["label"], sc.features["label"].names[example["label"]])

# AST fine-tuned on Speech Commands.
kws = pipeline("audio-classification",
               model="MIT/ast-finetuned-speech-commands-v2")
print(kws(example["audio"]["array"], top_k=5))
# [{'score': 0.999, 'label': 'backward'},
#  {'score': 0.0003, 'label': 'happy'},
#  {'score': 0.0002, 'label': 'follow'},
#  {'score': 0.0001, 'label': 'stop'},
#  {'score': 0.0001, 'label': 'up'}]

from datasets import load_dataset, Audio
from transformers import pipeline

minds = load_dataset("PolyAI/minds14", name="en-AU", split="train")
minds = minds.cast_column("audio", Audio(sampling_rate=16_000))
example = minds[0]
print(example["transcription"])
# "I would like to pay my electricity bill using my card can you please assist"

# XLS-R (multilingual wav2vec 2.0) fine-tuned on MINDS-14.
intent = pipeline("audio-classification",
                  model="anton-l/xtreme_s_xlsr_300m_minds14")
print(intent(example["audio"]["array"], top_k=3))
# [{'score': 0.963, 'label': 'pay_bill'},
#  {'score': 0.014, 'label': 'freeze'},
#  {'score': 0.006, 'label': 'card_issues'}]

from datasets import load_dataset
from transformers import pipeline

fleurs = load_dataset("google/fleurs", "all", split="validation", streaming=True)
example = next(iter(fleurs))

# Whisper fine-tuned for language identification on FLEURS.
langid = pipeline("audio-classification",
                  model="sanchit-gandhi/whisper-medium-fleurs-lang-id")
print(langid(example["audio"]["array"], top_k=3))
# [{'score': 0.999, 'label': 'Afrikaans'},
#  {'score': 0.0001, 'label': 'Northern-Sotho'},
#  {'score': 0.00005, 'label': 'Icelandic'}]

20.0.5.3 AST for Content and Event Classification

For audio content and event classification, the default backbone is AST fine-tuned on AudioSet (the 527-class checkpoint introduced in Section 20.0.3.3). The inference recipe at the lower-level processor + model API exposes a useful trick: the id2label mapping on the model config lets the reader convert the integer prediction back to a human-readable class.

from datasets import load_dataset, Audio
from transformers import AutoFeatureExtractor, ASTForAudioClassification
import torch

# The release name embeds the AudioSet mAP: 10x10 patches, 0.4593 mean AP.
checkpoint = "MIT/ast-finetuned-audioset-10-10-0.4593"
extractor = AutoFeatureExtractor.from_pretrained(checkpoint)
model = ASTForAudioClassification.from_pretrained(checkpoint).eval()

ds = load_dataset("ashraq/esc50", split="train")
ds = ds.cast_column("audio", Audio(sampling_rate=extractor.sampling_rate))
example = ds[0]

inputs = extractor(example["audio"]["array"],
                   sampling_rate=extractor.sampling_rate,
                   return_tensors="pt")

with torch.no_grad():
    logits = model(**inputs).logits

predicted_id = torch.argmax(logits, dim=-1).item()
print(f"Predicted AudioSet class: {model.config.id2label[predicted_id]}")
# Predicted AudioSet class: Dog

# AudioSet is multi-label (up to multiple events per clip), so often you want
# all classes above a threshold rather than just argmax:
probs = torch.sigmoid(logits)[0]
top_indices = torch.topk(probs, k=5).indices
for i in top_indices:
    print(f"  {probs[i].item():.3f}  {model.config.id2label[i.item()]}")

20.0.5.4 CLAP: Zero-Shot Audio Classification

The pretrained checkpoints above cover hundreds of classes, but real downstream tasks often need labels that no checkpoint was trained on: "is this a vacuum cleaner?", "does this clip contain a baby crying?", "is this the sound of running water?". CLAP (Contrastive Language-Audio Pretraining; Wu et al., 2023) solves this with the same trick CLIP uses for images: learn a shared embedding space for audio and text such that the audio embedding and its description's text embedding land close together. At inference, embed the audio plus a list of candidate text descriptions ("Sound of a dog", "Sound of vacuum cleaner") and pick the one with the highest cosine similarity. No task-specific fine-tuning required.

Architecture

CLAP has two encoders.

The audio encoder is typically HTSAT (Hierarchical Token-Semantic Audio Transformer) or PaSST, both transformer architectures that consume log-mel spectrograms. The output is a single fixed-length embedding per clip (after mean pooling) projected to a shared embedding dimension (typically 512 or 1024).

The text encoder is a BERT-family transformer (RoBERTa-base in the LAION release) that takes the candidate text description and produces a matching fixed-length embedding in the same shared space.

Training: Symmetric InfoNCE

Training pairs are (audio clip, text description). Common sources include LAION-Audio-630K (the largest open audio-text dataset, with ~630,000 clips), AudioCaps (audio with crowdsourced captions), WavCaps (audio with web-scraped captions), and FreeSound (Creative Commons audio with user-supplied tags). Each minibatch of $N$ pairs $(a_i, t_i)$ gets two contrastive losses, one in each direction:

The first term is audio-to-text matching: out of the $N$ candidate texts in the batch, the model should pick the true one given the audio. The second term is text-to-audio: out of the $N$ candidate audios, pick the true one given the text. The temperature $\tau$ (a learned scalar, typically around 0.07) sharpens the softmax. The form is identical to CLIP's contrastive loss with audio replacing image.

Two Engineering Tricks That Matter

CLAP would not work without two domain-specific tricks that bridge the audio-text gap.

Variable-length audio with chunk-and-fuse. Audio clips vary from one second to several minutes; a single fixed-size patch embedding loses information for long clips. CLAP's audio encoder handles this with a multi-branch input. For clips longer than 10 seconds, the model samples three random 10-second chunks, computes a fourth down-sampled global representation (the full clip resampled to fit a 10-second window), and feeds all four through the audio encoder. The four embeddings are merged through an attention feature fusion module that learns a weighted combination. For clips shorter than 10 seconds, the input is simply repeated and zero-padded to 10 seconds, and the chunk-and-fuse module degenerates to processing the same content four times.

Caption-versus-keyword text handling with T5 keyword-to-sentence augmentation. CLAP's training data mixes two text formats. Some examples have full sentence captions ("A dog is barking in the distance while a car passes by"); others have only keyword labels ("dog bark", "car"). Embedding keywords directly works poorly because the text encoder was pretrained on full sentences. CLAP routes keyword-only examples through a small T5 model fine-tuned for keyword-to-sentence augmentation: "dog bark" becomes "The sound of a dog barking." This synthesized sentence is then fed to the text encoder. The augmentation does not need to be perfect; it just needs to bring the keyword text into the same distribution the text encoder expects.

AudioCLIP: The Related Early Model

Guzhov et al.'s AudioCLIP (Guzhov et al., 2022) was an earlier attempt at the same idea, extending OpenAI's CLIP with an audio branch trained on AudioSet labels treated as text descriptions. AudioCLIP supports three modalities (image, text, audio) and three pairwise alignment directions, useful for cross-modal retrieval. CLAP came after with cleaner audio-text-only training and stronger downstream performance, but AudioCLIP remains the relevant reference for trimodal audio-image-text retrieval scenarios.

from datasets import load_dataset
from transformers import pipeline

esc50 = load_dataset("ashraq/esc50", split="train")
example = esc50[0]

# laion/clap-htsat-unfused is the open-source CLAP release.
zs = pipeline(task="zero-shot-audio-classification",
              model="laion/clap-htsat-unfused")

# Define the candidate label set as natural-language phrases.
labels = [
    "Sound of a dog barking",
    "Sound of a vacuum cleaner",
    "Sound of rain on a window",
    "Sound of a baby crying",
]
result = zs(example["audio"]["array"], candidate_labels=labels)
for r in result:
    print(f"  {r['score']:.3f}  {r['label']}")
# Picks "Sound of a dog barking" with the highest score on a dog clip.

20.0.5.5 Supervised Fine-Tuning: DistilHuBERT on GTZAN

When neither a curated checkpoint nor CLAP's zero-shot capability fits the task, the next step is supervised fine-tuning on labeled data. The canonical recipe uses a small pretrained encoder (DistilHuBERT from Section 20.0.4), the HuggingFace Trainer, and a few hundred to a few thousand labeled examples. The textbook example is music genre classification on GTZAN (Tzanetakis & Cook, 2002): 1000 thirty-second clips across 10 genres (blues, classical, country, disco, hiphop, jazz, metal, pop, reggae, rock), the de-facto benchmark for music genre classifiers since 2002.

import numpy as np
import evaluate
from datasets import load_dataset, Audio
from transformers import (
    AutoFeatureExtractor, AutoModelForAudioClassification,
    TrainingArguments, Trainer,
)

# 1. Load and split GTZAN. 999 records (one rock file is corrupt and dropped).
gtzan = load_dataset("marsyas/gtzan", "all")
gtzan = gtzan["train"].train_test_split(seed=42, shuffle=True, test_size=0.1)
print(gtzan)
# DatasetDict({
#     train: Dataset({ num_rows: 899, features: ['file', 'audio', 'genre'] })
#     test:  Dataset({ num_rows: 100, features: ['file', 'audio', 'genre'] })
# })

# Map integer label IDs to human-readable names (and back).
id2label = {i: gtzan["train"].features["genre"].int2str(i) for i in range(10)}
label2id = {v: k for k, v in id2label.items()}
num_labels = len(id2label)

# 2. Feature extractor (preprocesses raw audio for the model).
model_id = "ntu-spml/distilhubert"
feat = AutoFeatureExtractor.from_pretrained(
    model_id, do_normalize=True, return_attention_mask=True,
)

# Resample to the rate DistilHuBERT expects (16 kHz).
gtzan = gtzan.cast_column("audio", Audio(sampling_rate=feat.sampling_rate))

MAX_DURATION_S = 30.0
def preprocess(batch):
    arrays = [x["array"] for x in batch["audio"]]
    inputs = feat(
        arrays, sampling_rate=feat.sampling_rate,
        max_length=int(feat.sampling_rate * MAX_DURATION_S),
        truncation=True, return_attention_mask=True,
    )
    return inputs

gtzan_encoded = gtzan.map(preprocess, batched=True, batch_size=32,
                          remove_columns=["audio", "file"])
gtzan_encoded = gtzan_encoded.rename_column("genre", "label")

# 3. Model with a 10-class classification head on top of DistilHuBERT.
model = AutoModelForAudioClassification.from_pretrained(
    model_id, num_labels=num_labels, label2id=label2id, id2label=id2label,
)

# 4. Metric: top-1 accuracy.
accuracy = evaluate.load("accuracy")
def compute_metrics(eval_pred):
    predictions = np.argmax(eval_pred.predictions, axis=1)
    return accuracy.compute(predictions=predictions, references=eval_pred.label_ids)

# 5. Trainer.
training_args = TrainingArguments(
    output_dir="distilhubert-gtzan",
    eval_strategy="epoch",
    save_strategy="epoch",
    learning_rate=5e-5,
    per_device_train_batch_size=8,
    gradient_accumulation_steps=4,         # effective batch size 32
    per_device_eval_batch_size=8,
    num_train_epochs=10,
    warmup_ratio=0.1,
    logging_steps=10,
    load_best_model_at_end=True,
    metric_for_best_model="accuracy",
    push_to_hub=False,
    fp16=True,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=gtzan_encoded["train"],
    eval_dataset=gtzan_encoded["test"],
    tokenizer=feat,                  # feature extractor doubles as collator
    compute_metrics=compute_metrics,
)
trainer.train()
# Expect ~85% test accuracy after 10 epochs on a single A100 (about 30 min).

A few production notes that the slide does not cover. First, DistilHuBERT is preferred over the full HuBERT base for this kind of demo because it fine-tunes about 2x faster with minimal accuracy loss; for production deployments where the last 2 to 3 accuracy points matter, swap to facebook/hubert-base-ls960 or larger. Second, gradient_accumulation_steps=4 simulates a batch size of 32 on a GPU that can only fit 8 samples in memory; reader with more VRAM can drop this and increase the per-device batch size. Third, the fp16=True flag enables mixed-precision training, which is essential on consumer GPUs (RTX 4090 and below) to fit the model with reasonable batch sizes.

20.0.5.6 Which Recipe Should the Reader Use?

A short decision tree for any new audio classification task:

1. Does a curated HuggingFace checkpoint already cover the task? (Search the Hub for the dataset name, e.g., speech_commands, minds14, fleurs, audioset, esc50, gtzan.) If yes, the pipeline one-liner from Section 20.0.5.2 is the entire solution.
2. Are the target labels arbitrary text descriptions, with no labeled training data? Use CLAP zero-shot (Code Fragment 20.0.5.5). Expect 60 to 80% accuracy on standard benchmarks, lower if the candidate labels are very fine-grained or domain-specific.
3. Is there a moderate amount of labeled data (a few hundred to a few thousand examples) in a closed vocabulary? Fine-tune DistilHuBERT or AST (Code Fragment 20.0.5.6). Expect 80 to 95% accuracy depending on task difficulty and class imbalance.
4. Is the task domain-specific with hundreds of thousands of labeled examples? Train from scratch or fine-tune a larger backbone (HuBERT large, WavLM large, BEATs) end-to-end with a longer schedule. This is the regime where the model gap between SSL backbones matters; Section 20.0.4's cheat-sheet picks the right one.