Joint Embedding Spaces for Multimodal Retrieval

"Two encoders that agree on the same point in latent space have implicitly written a translation dictionary."

RAG, Cross-Modal-Curious AI Agent

33.1.1 The Contrastive Recipe

The standard contrastive objective trains two encoders (one per modality) to produce similar embeddings for matched pairs and dissimilar embeddings for mismatched pairs. For a batch of $N$ text-image pairs $(t_i, x_i)$:

$$ \mathcal{L} = -\frac{1}{2N}\sum_{i=1}^N \log\frac{\exp(\langle f(t_i), g(x_i)\rangle/\tau)}{\sum_{j=1}^N \exp(\langle f(t_i), g(x_j)\rangle/\tau)} - \frac{1}{2N}\sum_{i=1}^N \log\frac{\exp(\langle g(x_i), f(t_i)\rangle/\tau)}{\sum_{j=1}^N \exp(\langle g(x_i), f(t_j)\rangle/\tau)} $$

where $f, g$ are the text and image encoders, $\langle \cdot, \cdot \rangle$ is dot product on normalized vectors, and $\tau$ is a learnable temperature. The two summands are the InfoNCE losses in each direction (text-to-image and image-to-text); their average is the symmetric CLIP loss.

The training pattern is the same as the contrastive embedding training from Section 31.1, with two distinctions:

• Cross-modal positive pairs: instead of paraphrased text pairs, the positives are (text, image) pairs scraped from the web (alt text, captions, surrounding HTML).
• Hard negatives by construction: every other item in the batch is a negative; large batches (16k+) provide enough hard negatives to push apart subtly different concepts.

33.1.2 CLIP and Its Direct Successors

CLIP (Radford et al., 2021) was the breakthrough. The original paper trained on 400M (image, alt-text) pairs scraped from the web with a ViT-L/14 image encoder and a 63M-parameter text transformer. The result: an embedding space where text-to-image retrieval, zero-shot image classification, and image-to-text similarity all worked without task-specific fine-tuning.

The 2024-2026 successors improve on three axes:

• SigLIP (Zhai et al., 2023): replaces the softmax-over-batch InfoNCE with a per-pair sigmoid loss. Removes the dependency on global batch normalization, allowing smaller-batch training and better scaling.
• EVA-CLIP (Sun et al., 2023): bigger image encoder (1B to 18B parameters) and better masked-image-modeling initialization. Improved zero-shot ImageNet accuracy by 4 to 6 points.
• OpenCLIP / MetaCLIP: open-source CLIP reproductions on larger, better-curated datasets (DataComp, MetaCLIP-1B).
• SigLIP 2 (Zhai et al., 2025): adds NaFlex (native flexible resolution), Locca caption regularization, and dense decoder probes. Now the open-source default in 2026.

# SigLIP 2 inference: encode a text query and a set of images,
# then rank by cosine similarity for retrieval.
from transformers import AutoProcessor, AutoModel
import torch, torch.nn.functional as F

MODEL_ID = "google/siglip2-base-patch16-naflex"
model = AutoModel.from_pretrained(MODEL_ID).cuda().eval()
proc = AutoProcessor.from_pretrained(MODEL_ID)

def encode_text(queries):
    batch = proc(text=queries, return_tensors="pt",
                  padding=True).to("cuda")
    with torch.inference_mode():
        emb = model.get_text_features(**batch)
    return F.normalize(emb, dim=-1)

def encode_images(paths):
    images = [Image.open(path) for path in paths]
    batch = proc(images=images, return_tensors="pt").to("cuda")
    with torch.inference_mode():
        emb = model.get_image_features(**batch)
    return F.normalize(emb, dim=-1)

# Retrieve top-5 images for a query
q = encode_text(["a dog wearing sunglasses"])
img_embs = encode_images(all_image_paths)
scores = (q @ img_embs.T).squeeze()
top5 = scores.topk(5)
for i, idx in enumerate(top5.indices):
    print(f"#{i+1} {all_image_paths[idx]} score {top5.values[i]:.3f}")

33.1.3 ImageBind and Beyond: Six Modalities

ImageBind (Girdhar et al., 2023) extended the joint embedding idea to six modalities: image, text, audio, depth, thermal, IMU (inertial). The key insight: you don't need pairs of every modality combination. As long as each modality is paired with images, the joint space self-organizes through the image hub.

This "image-centered" training is parameter-efficient. ImageBind freezes a pretrained CLIP image encoder and trains each new modality's encoder to align with the CLIP image space. After training:

• Audio-to-text retrieval works (via the shared space).
• Audio-to-depth retrieval works.
• Image-to-audio retrieval works.

None of these pairs ever appeared in the training data directly; the alignment is transitive through the image hub. This is the same trick that powers cross-lingual word embeddings via a pivot language.

33.1.4 Late Fusion by Design

Joint embedding models are late-fusion by design (see Section 37.2). The modalities meet only at the final embedding, with no shared transformer layers. This has two consequences:

• Efficiency at scale: pre-compute image embeddings once; only the text query is encoded at query time. With $N$ images and $M$ queries, you do $N$ image encodes plus $M$ text encodes plus $NM$ dot products (trivially fast).
• Reasoning limits: the model cannot reason jointly about modalities; it can only match them at the embedding level. Asking "what unusual object is in this picture?" cannot be answered by a CLIP-style model; that needs a multimodal LLM.

The right mental model: joint embedding models are for retrieval, not for understanding. They are the first stage of a two-stage pipeline where a downstream multimodal LLM does the reasoning. Section 33.2 covers exactly this pattern.

33.1.5 Practical Considerations

Several engineering details determine whether your joint embedding retrieval actually works in production:

• L2 normalization: always normalize embeddings to unit norm. Without this, cosine similarity and dot product diverge, and ANN indices give wrong results.
• Dimensionality vs accuracy: 512-dim and 768-dim are the sweet spot. Above 1024 you pay storage and ANN cost with marginal accuracy gains. Matryoshka representation learning (MRL) lets you truncate to fewer dimensions at inference for a quality/cost trade-off.
• Domain mismatch: CLIP's training data was internet-scraped, so it knows celebrities and cats well but struggles with medical imagery, technical drawings, or aerial photography. Fine-tune or use a domain-specific variant.
• The hubness problem: in high-dimensional spaces, a few "hub" points appear as nearest neighbors for many queries. Normalize embedding magnitudes carefully and consider hubness-aware reranking.
• Modality gap: text and image embeddings in CLIP do not fully overlap; they occupy slightly different regions of the unit sphere. This is observable as a systematic bias in cross-modal retrieval. Mitigations include centering or adding a learned offset.

33.1.7 Audio-Text Joint Embeddings: CLAP and AudioCLIP

Everything in this section so far has been image-centric. The same recipe transfers almost unchanged to audio, with a few twists that the audio modality forces on the architecture. CLAP (Contrastive Language-Audio Pretraining, Elizalde et al., ICASSP 2023) is the cleanest example.

The training objective is the symmetric InfoNCE loss from Section 33.1.1, retargeted to (audio clip, text caption) pairs. Concretely:

$$ \mathcal{L}_{\text{CLAP}} = -\frac{1}{2N}\sum_{i=1}^N \log\frac{\exp(\langle f(a_i), g(t_i)\rangle/\tau)}{\sum_{j=1}^N \exp(\langle f(a_i), g(t_j)\rangle/\tau)} - \frac{1}{2N}\sum_{i=1}^N \log\frac{\exp(\langle g(t_i), f(a_i)\rangle/\tau)}{\sum_{j=1}^N \exp(\langle g(t_i), f(a_j)\rangle/\tau)} $$

where $f$ is the audio encoder, $g$ is the text encoder, $\tau$ is a learnable temperature, and the two terms are the audio-to-text and text-to-audio InfoNCE losses. The structure is identical to CLIP; only the encoders change. Figure 33.1.3 sketches the resulting dual-tower architecture.

Figure 33.1.3a: CLAP's dual-tower architecture. An audio encoder (HTSAT over log-mel spectrograms) and a text encoder (RoBERTa) project into a shared 512-d embedding space. The symmetric InfoNCE loss pulls each true (audio, caption) pair together and pushes mismatched pairs apart, yielding the same zero-shot retrieval behaviour CLIP brought to images.

Audio breaks two CLIP assumptions and CLAP addresses each one. First, audio clips have very different durations: a doorbell is one second, a podcast episode is one hour, and the audio encoder cannot accept variable input the way ViT accepts fixed-size image patches. CLAP's chunk-and-fuse trick samples three random ten-second chunks from the clip plus one heavily downsampled "global" view, encodes each independently with an HTSAT spectrogram transformer, and fuses the four embeddings with a small attention block. Second, audio datasets like AudioSet are labeled with keyword tags ("dog bark", "rain") rather than full sentences. CLAP fixes the distribution mismatch by passing every tag through a frozen T5 prompted to expand keywords into natural-language sentences ("a dog barking loudly in a backyard"), then training the text encoder on those sentences. The trick gives the model a sentence-shaped training distribution even when the source data is tag-only.

AudioCLIP (Guzhov et al., 2022) is the older cousin: it extends CLIP to text + image + audio by attaching an ESResNeXt audio encoder to a frozen CLIP backbone and aligning all three through a tri-modal contrastive loss. Where CLAP is the audio-only specialist, AudioCLIP gives you audio retrieval that sits in the same space as CLIP image and text embeddings, which is useful when an application needs to mix sound search and image search against a unified index. ImageBind (Section 33.1.3) later generalized this idea to six modalities, but AudioCLIP remains a useful drop-in when you only need the three.

from transformers import pipeline

clap = pipeline(
    task="zero-shot-audio-classification",
    model="laion/clap-htsat-unfused",
)
result = clap(
    "doorbell.wav",
    candidate_labels=["doorbell", "dog bark", "phone ring", "alarm"],
)
print(result)  # list of {label, score} sorted by score

33.1.6 Vector Database Integration

Joint embeddings are the input to a vector database, the same infrastructure covered in Chapter 31. The choices in 2026:

• FAISS: in-process library; best for embedded use cases up to 10s of millions of vectors.
• Qdrant: open-source self-hosted vector database with filtering and metadata support.
• Pinecone, Weaviate: hosted vector databases; production-ready out of the box.
• Vespa, Milvus: large-scale (billions of vectors) systems with advanced filtering.
• pgvector: Postgres extension; convenient when you already have a Postgres database.

For cross-modal RAG, the vector database must support storing metadata alongside vectors (modality type, source URL, timestamp, language) so the application can filter retrievals. Most modern vector databases handle this; FAISS does not natively.