Contrastive Vision-Language: CLIP and SigLIP

"Show me 400 million captioned images and I will teach a vision encoder to speak. Most of the words it learns are 'cat'."

A Contrastively-Aligned AI Agent

22.2.1 The Contrastive Objective

The CLIP training recipe (Radford et al., OpenAI, 2021) is conceptually simple. Take a batch of N (image, caption) pairs. Encode each image through a vision encoder (originally ResNet-50, later ViT-B/16, ViT-L/14, ViT-H/14) and each caption through a text encoder (a 12-layer transformer trained from scratch). Project both into a shared d-dimensional embedding space (typically d=512 or 768). Now you have N image vectors and N text vectors.

The training objective is to make matching (image, caption) pairs have high cosine similarity and non-matching pairs have low similarity. Concretely, CLIP computes the NxN matrix of cosine similarities between every image and every caption in the batch, scales by a learnable temperature, and applies symmetric cross-entropy: the model is trained to assign the highest probability to the correct caption when given an image, and to the correct image when given a caption. This is the InfoNCE objective:

$$L = -\frac{1}{2N} \left[ \sum_i \log \frac{\exp(s_{\text{ii}}/\tau)}{\sum_j \exp(s_{\text{ij}}/\tau)} + \sum_i \log \frac{\exp(s_{\text{ii}}/\tau)}{\sum_j \exp(s_{\text{ji}}/\tau)} \right]$$

where s_ij is the cosine similarity between image i and caption j, and T is the learnable temperature parameter (typically initialized at 0.07).

The genius of the recipe is what it does not require: no per-image labels, no curated dataset, no human annotation. Captions scraped from the web (alt-text on news sites, ecommerce product descriptions, Wikipedia image captions, Reddit posts) are the supervision. OpenAI's training set, named WIT-400M, was 400 million such pairs assembled from public web crawls.

CLIP internals: a five-step walk-through

It helps to follow exactly what happens to an image and a caption as they pass through CLIP at inference time, because every later VLM connector (Section 22.3) borrows pieces of this pipeline.

1. Pool from the ViT. The image is patchified, embedded, and run through the vision transformer (Section 22.1). The output is a sequence of 197 tokens (1 [CLS] + 196 patches for a 224x224 input). CLIP reads off the final hidden state at the [CLS] position as a single 768-dimensional image vector and discards the per-patch tokens. (Subsequent VLMs that need spatial detail keep the patch tokens instead; CLIP throws them away because it only needs one global vector for matching.)
2. Pool from the text encoder. The caption is BPE-tokenized, prepended with a start-of-text token, run through a 12-layer transformer, and pooled at the position of the [EOS] (end-of-text) token. CLIP's text encoder uses causal masking like a small GPT, so the [EOS] position has aggregated information from every preceding token; reading it gives a single 512-dimensional text vector. (BERT-style mean pooling or [CLS] pooling also work but lose 1-2 points of accuracy.)
3. Project to the common latent space. The image vector (768-d) and text vector (512-d) live in different spaces and have different dimensions, so neither can be compared to the other yet. A separate projection layer on each side, a single linear map, sends both into a shared $d$-dimensional space (typically $d=512$ for CLIP-B, $d=768$ for CLIP-L). These two projection matrices are the only modules that look directly at the matching loss; everything else is gradient that backpropagates through them.
4. Normalize. Both projected vectors are L2-normalized to unit length, placing every example on the surface of a unit sphere in $\mathbb{R}^d$. After this step, cosine similarity collapses to a plain dot product (the note above).
5. Compare. For a batch of $N$ images and $N$ captions, the $N \times N$ matrix $S$ of pairwise dot products contains every similarity needed for the InfoNCE loss; for zero-shot inference, the column of $S$ for a single query image plus $K$ candidate text embeddings is enough to rank the $K$ candidates by softmax probability.

Five steps, one matrix multiply per side, and the whole multimodal alignment falls out. Section 22.3 will show that the same five-step structure (encode each modality, project, normalize, compare) reappears inside Q-Former training and Sentence-Image-Alignment classifiers; CLIP is simply the recipe boiled down to its essence.

22.2.2 The Zero-Shot Magic

The transformative property of CLIP is zero-shot classification. To classify an image into one of K classes, you do not need to fine-tune CLIP on labeled examples of those classes. Instead, you write a textual prompt for each class (typically "a photo of a {class_name}"), encode each prompt through the text encoder, encode the image through the vision encoder, and pick the class whose text embedding has the highest cosine similarity with the image embedding. This works for any class describable in natural language.

On ImageNet (1000 classes), zero-shot CLIP-L/14 scores 75.5% top-1 accuracy, matching a ResNet-50 explicitly trained on ImageNet. On more specialized benchmarks (fine-grained bird species, satellite imagery, medical images), zero-shot accuracy is lower but still useful for rapid prototyping without labeled data.

The mechanism that makes zero-shot work is the structure of the embedding space. Because CLIP was trained on diverse web image-text pairs, the text encoder has learned to map descriptions like "a photo of a golden retriever" to a region of embedding space close to images of actual golden retrievers. A new class the model has never seen as a label (say, "a photo of a Boston Terrier in a sweater") inherits structure from the training distribution: "Boston Terrier" maps near other dog breeds, "sweater" near other clothing items, and the composition lands in a plausible region of image-embedding space.

22.2.3 Running CLIP Zero-Shot Classification

The OpenAI CLIP repo and the Hugging Face CLIPModel class both expose CLIP for inference. The Hugging Face version is more convenient for integration with other models. The pattern is: load the model and processor, prepare a list of candidate text prompts, encode both modalities, and compute the similarity matrix.

from transformers import CLIPProcessor, CLIPModel
from PIL import Image
import torch

# Load CLIP-L/14 trained at 336x336 resolution
model = CLIPModel.from_pretrained("openai/clip-vit-large-patch14-336")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-large-patch14-336")

# Image + 5 candidate class prompts
image = Image.open("unknown_animal.jpg").convert("RGB")
prompts = [
    "a photo of a golden retriever",
    "a photo of a Boston Terrier",
    "a photo of a Siamese cat",
    "a photo of a red fox",
    "a photo of a gray wolf",
]

inputs = processor(
    text=prompts, images=image,
    return_tensors="pt", padding=True,
)

with torch.inference_mode():
    outputs = model(**inputs)

# logits_per_image: [batch_images, batch_texts] = [1, 5]
probs = outputs.logits_per_image.softmax(dim=-1)[0]
for prompt, p in zip(prompts, probs):
    print(f"  {p.item():.4f}  {prompt}")

# Direct embedding access for retrieval applications
image_emb = model.get_image_features(pixel_values=inputs.pixel_values)
image_emb /= image_emb.norm(dim=-1, keepdim=True)
print(f"Image embedding shape: {image_emb.shape} (L2-normalized)")

22.2.4 Prompt Engineering for CLIP

Although CLIP is presented as a zero-shot model, prompt engineering substantially affects accuracy. Three patterns dominate. The first is template ensembling: rather than "a photo of a {class}", use 80 templates ("a sketch of a {class}", "a low-resolution photo of a {class}", "an art photo of a {class}", etc.), encode each, average the resulting text embeddings, and use the average for classification. This typically adds 1-3% top-1 accuracy on ImageNet.

The second is class name normalization. CLIP was trained on web text, so colloquial names often outperform Latin or scientific names. "Mountain lion" beats "Puma concolor"; "tiger shark" beats "Galeocerdo cuvier". When class names are ambiguous (multiple senses for the same word, like "crane" for the bird vs. the construction machine), adding a disambiguator helps: "a photo of a crane bird".

The third is multi-modal retrieval. Beyond single-class classification, the same embedding space enables image retrieval ("find images matching this caption") and text retrieval ("find captions matching this image"). FAISS or HNSW indices over normalized image embeddings give millisecond-latency retrieval over tens of millions of images.

22.2.5 SigLIP: The Sigmoid Variant

SigLIP (Zhai et al., Google, 2023) is a refinement of CLIP that replaces the softmax-over-batch contrastive loss with an independent sigmoid loss per (image, text) pair. The change looks small but has two important consequences.

The first is independence from batch size. CLIP's softmax requires the entire batch's similarity matrix to be normalized together, which means the loss for any single pair depends on every other pair in the batch. Doubling the batch size doubles the implicit negative samples and improves training. This forces CLIP to be trained at enormous batch sizes (32k-65k). SigLIP's sigmoid loss treats each pair independently as a binary classification (positive or negative match), so smaller batches work nearly as well as huge batches. SigLIP can be trained effectively at batch sizes of 2k-8k.

The second is calibration. The sigmoid output is interpretable as a probability of match, which makes SigLIP scores comparable across models and tasks. CLIP's softmax probabilities only make sense relative to a specific set of candidates in a batch.

Writing $\mathbf{v}_i$ for the L2-normalized image embedding of the $i$-th sample, $\mathbf{u}_j$ for the L2-normalized text embedding of the $j$-th caption in a batch of size $N$, $t$ for a learnable inverse temperature, and $b$ for a learnable bias initialized so the model is not overwhelmed by the $N^2 - N$ negative pairs at the start of training, the SigLIP loss for a single $(i, j)$ pair is

so the full batch loss is the simple sum $\mathcal{L}_{\mathrm{SigLIP}} = \frac{1}{N} \sum_{i=1}^{N} \sum_{j=1}^{N} \ell_{ij}$. Contrast this with the CLIP softmax loss, which couples every pair through the partition function:

Every $\ell_{ij}$ in the SigLIP sum factorizes, so the gradient for any pair never references any other pair's similarity, which is exactly why the batch size can drop from 32k to 2k without hurting accuracy.

Empirically, SigLIP-So400M (the public flagship released in late 2023) matches or exceeds CLIP-L/14 on zero-shot ImageNet (around 83% vs. 75.5%) and beats CLIP on every multilingual benchmark, primarily because the smaller-batch tolerance let Google train on a large WebLI corpus with reasonable compute budgets. As of 2026, SigLIP-So400M is the most common vision encoder choice for new VLM projects, having largely replaced OpenAI's CLIP-L/14.

import torch
from PIL import Image
from transformers import AutoProcessor, AutoModel

# SigLIP-So400M: 400M-parameter vision encoder, the 2026 default for VLM stacks.
processor = AutoProcessor.from_pretrained("google/siglip-so400m-patch14-384")
model = AutoModel.from_pretrained("google/siglip-so400m-patch14-384").eval()

image = Image.open("photo.jpg").convert("RGB")
labels = [f"a photo of a {c}" for c in ["cat", "dog", "skateboard", "saxophone"]]

inputs = processor(text=labels, images=image, return_tensors="pt",
                   padding="max_length")

with torch.no_grad():
    out = model(**inputs)
# logits_per_image: (1, num_labels); SigLIP returns RAW sigmoid logits,
# so the right reduction is per-label sigmoid, not softmax-over-labels.
probs = torch.sigmoid(out.logits_per_image).squeeze(0)
for label, p in zip(labels, probs.tolist()):
    print(f"{p:.3f}  {label}")
# 0.984  a photo of a cat
# 0.071  a photo of a dog
# 0.004  a photo of a skateboard
# 0.002  a photo of a saxophone

22.2.6 OpenCLIP and the Open-Source Frontier

The original CLIP was released by OpenAI with weights but no training code. OpenCLIP (LAION + community, 2022-present) is the open replica: full training code, hyperparameters, multiple model sizes from B/32 through G/14, training on the LAION-2B and LAION-5B image-text datasets. As of 2026, OpenCLIP supports 30+ model variants and is the dominant choice for academic research and self-hosted production deployments.

OpenCLIP's most impactful variants are the ViT-G/14 (2B parameters total, trained on LAION-2B, ~78.4% zero-shot ImageNet) and the ViT-bigG/14 (2.5B parameters, trained on LAION-2B-39B with batch 160k, ~80.1% zero-shot ImageNet). These models are within a couple of points of OpenAI's never-released private replicas. They are also frequently substituted for OpenAI CLIP in production VLM pipelines when license terms matter; OpenCLIP weights are released under permissive licenses while OpenAI's CLIP has more restrictive terms for commercial use.

from sentence_transformers import SentenceTransformer
from PIL import Image
import torch

model = SentenceTransformer("clip-ViT-B-32")    # also: "clip-ViT-L-14", "sentence-transformers/clip-ViT-L-14-laion2B-s32B-b82K"

image_embs = model.encode(
    [Image.open(p).convert("RGB") for p in ["cat.jpg", "dog.jpg", "car.jpg"]],
    convert_to_tensor=True, normalize_embeddings=True,
)
text_embs = model.encode(
    ["a photo of a kitten", "a sports car at sunset"],
    convert_to_tensor=True, normalize_embeddings=True,
)

scores = text_embs @ image_embs.T              # cosine == dot product on unit vectors
print(scores)
# tensor([[0.30, 0.15, 0.06],     # kitten matches cat
#         [0.05, 0.04, 0.27]])    # car matches car

22.2.7 EVA-CLIP and Data Recipes

EVA-CLIP (BAAI, 2023-2024) demonstrates that CLIP's training recipe can be improved by better data and better initialization, not just larger scale. EVA-CLIP starts from EVA-02 (a masked-image-modeling pretrained ViT) rather than random initialization, applies aggressive learning-rate warmup, and uses a curated subset of LAION + JFT data. The result is that EVA-CLIP-G/14 matches OpenCLIP-G/14 with 40% less training compute and beats it by 3 points on zero-shot ImageNet at the same compute budget.

The 2024 follow-up, EVA-CLIP-18B, scales to 18 billion total parameters by adding 14B of text encoder while keeping the vision encoder at 6B. This is the largest publicly known CLIP-style model, reaching 80.7% zero-shot top-1 averaged across 27 image classification benchmarks (per the EVA-CLIP-18B paper, Feb 2024) and excellent retrieval on dense image-text datasets. Its inference cost (40-60 GB VRAM for fp16) limits production use, but it serves as the gold-standard reference for downstream distillation.

22.2.8 Where CLIP and SigLIP Fit in VLMs

In a generative VLM like LLaVA, the role of the contrastive vision encoder is to produce image embeddings that the connector module can project into the LLM's token-embedding space. The contrastive pretraining ensures these embeddings already encode language-aligned semantic structure: a photo of a dog produces an embedding "near" the text embedding of "a photo of a dog", which means the LLM only needs to learn a relatively simple projection rather than reverse-engineer the visual-language alignment from scratch.

Empirical ablations show this matters dramatically. A VLM with a CLIP-pretrained vision encoder reaches a given accuracy with 4-8x less alignment training than the same architecture with a randomly initialized or ImageNet-pretrained encoder. This is why every production VLM uses a contrastive-pretrained vision encoder, and why understanding CLIP and SigLIP is foundational for understanding the rest of the chapter.

22.2.9 Key Takeaways

22.2.10 Self-Check

22.2.11 Bibliography