Multimodal RAG: Image, Audio, Video Retrieval-Augmented Generation

"A text RAG explains an answer from a document. A multimodal RAG explains an answer from a picture."

RAG, Multimodally-Indexed AI Agent

33.2.1 The Three Cuts of Multimodal RAG

"Multimodal RAG" is an umbrella term for three distinct retrieval patterns:

• Image-as-context: text query retrieves relevant images; the VLM answers the question using those images as visual context. Use case: "Find me product reviews of cameras that look like this picture" or "Which of these screenshots shows an error?"
• Document-with-images RAG: a corpus of mixed text and image documents (PDFs, slide decks, web pages). Retrieval returns pages or sections; the VLM reasons over the multimodal content. Use case: technical document Q&A, financial report analysis. Closely related to Chapter 21's document understanding.
• Audio or video RAG: a corpus of audio or video clips. Retrieval returns time-aligned segments; the generation model summarizes, answers, or transcribes. Use case: lecture search, podcast Q&A, video clip retrieval for editorial work.

Each pattern has different chunking, embedding, and prompt-engineering needs, but all share the basic flow: encode-the-corpus, embed-the-query, retrieve-top-k, prompt-the-VLM.

33.2.2 Image-as-Context Pattern

The simplest multimodal RAG: a text query retrieves images, and a VLM uses the retrieved images as visual context. The flow:

1. Encode all images with SigLIP 2 (or your chosen joint embedding model from Section 33.1). Store in a vector database.
2. At query time, encode the text query, retrieve top-$k$ images.
3. Pass the images plus the original query to a VLM (GPT-4o, Gemini 2.5 Pro, Qwen2-VL).
4. The VLM produces an answer grounded in the retrieved visual context.

# Image-as-context multimodal RAG: SigLIP for retrieval,
# GPT-4o-mini for grounded generation.
from openai import OpenAI
import base64, qdrant_client

qd = qdrant_client.QdrantClient(url=QDRANT_URL)
oai = OpenAI()

def multimodal_rag(query, k=5):
    # 1. Encode the text query into the joint space.
    q_emb = encode_text_siglip([query])[0]
    # 2. Retrieve top-k images from Qdrant.
    hits = qd.search(
        collection_name="images",
        query_vector=q_emb.tolist(),
        limit=k,
    )
    image_payloads = [h.payload["path"] for h in hits]
    # 3. Build a multimodal prompt for the VLM.
    content = [{"type": "text",
                "text": f"Answer using ONLY these {k} images. {query}"}]
    for path in image_payloads:
        with open(path, "rb") as f:
            b64 = base64.b64encode(f.read()).decode()
        content.append({
            "type": "image_url",
            "image_url": {"url": f"data:image/jpeg;base64,{b64}",
                            "detail": "high"},
        })
    # 4. Generate the grounded answer.
    response = oai.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": content}],
        max_tokens=400,
    )
    return response.choices[0].message.content, image_payloads

33.2.3 Document-with-Images RAG

Real-world documents (PDFs, slide decks, technical manuals) interleave text, figures, charts, and tables. A naive approach extracts only text and ignores images, losing critical information. A better approach indexes both text chunks and visual elements, then retrieves jointly.

Three patterns dominate in 2026:

• Page-as-image RAG (ColPali / ColQwen): encode each PDF page as an image with a vision-language model. Skip OCR entirely. ColPali (Faysse et al., 2024) uses PaliGemma to produce multi-vector page representations; ColQwen uses Qwen2-VL. Query-time retrieval matches the text query against page vectors with late interaction (a la ColBERT). State-of-the-art for technical document QA in 2026.
• Hybrid text + image embedding: extract text chunks via OCR or PDF parsing, also encode each page or figure as an image. Both go into the index with metadata pointers. Retrieval returns the top-k items of either type; the VLM consumes both at generation time.
• VLM-summarized chunks: pre-process each page with a VLM to produce a text summary that captures both textual and visual content. Index the summary; retrieve text-to-text. Generation receives the original page image plus the summary. Used by Anthropic's contextual retrieval and similar systems.

33.2.4 Video RAG and Chunking Strategies

A video is a temporal sequence of images plus an audio track. The chunking strategy determines what retrieval can find:

• Keyframe chunking: extract one frame per N seconds (or per scene change). Index each as an image. Lightweight but loses motion information.
• Clip chunking: split into N-second clips, embed each clip with a video encoder (VideoCLIP, InternVideo). Captures motion; higher storage cost.
• Transcript chunking: ASR the audio track, chunk the transcript, retrieve text-to-text. Misses visual content entirely.
• Hybrid (keyframe + transcript): index both keyframes and aligned transcript segments. Retrieve either type; generation receives the time-aligned segment of both.

For long videos (lectures, podcasts), the hybrid pattern with timestamp alignment is the production default. A 1-hour lecture might produce 60 keyframe vectors plus 200 transcript chunk vectors; total index size is small even for huge video libraries.

# Hybrid video chunking: keyframes at scene boundaries + transcript
# segments. Each chunk has a (start_sec, end_sec) range.
import cv2
from faster_whisper import WhisperModel

whisper = WhisperModel("large-v3", device="cuda")

def chunk_video(video_path):
    chunks = []
    # 1. Keyframes via scene-change detection
    cap = cv2.VideoCapture(video_path)
    fps = cap.get(cv2.CAP_PROP_FPS)
    prev_hist = None
    frame_idx = 0
    while True:
        ret, frame = cap.read()
        if not ret: break
        hist = cv2.calcHist([frame], [0,1,2], None,
                            [8,8,8], [0,256]*3)
        if prev_hist is None or \
           cv2.compareHist(prev_hist, hist,
                            cv2.HISTCMP_BHATTACHARYYA) > 0.4:
            t = frame_idx / fps
            chunks.append({"type": "frame", "time": t, "frame": frame})
            prev_hist = hist
        frame_idx += 1
    # 2. Transcript chunks with timestamps
    segments, _ = whisper.transcribe(video_path, word_timestamps=False)
    for seg in segments:
        chunks.append({
            "type": "transcript",
            "start": seg.start,
            "end": seg.end,
            "text": seg.text,
        })
    return chunks

33.2.5 Audio RAG

Pure audio RAG (no video) follows a similar pattern, but with one additional design axis: whether to treat audio as "text after transcription" or as a first-class modality with its own embedding. The right answer almost always depends on what the user is searching for, lexical content vs acoustic phenomena, which sets up the three choices below.

• Transcript-only: Whisper-transcribe the audio, chunk the transcript, retrieve text-to-text. Cheapest and good enough for most podcast/lecture use cases. OpenAI Whisper large-v3 (released November 2023) reaches roughly 6-10% WER on conversational English at $0.006 per minute of audio, which makes transcript-only RAG cost-competitive with text-document RAG once you amortize the one-time transcription pass.
• Acoustic + transcript: embed audio with CLAP or AudioMAE for queries like "find me the part with the audience laughter" that transcripts don't capture. Combine with transcript retrieval via score fusion. Microsoft's 2023 CLAP model (Contrastive Language-Audio Pretraining) embeds 10-second audio clips into a 512-dim space jointly aligned with text; on AudioCaps retrieval it reaches a Recall@10 of about 66%, which is enough to rescue queries the transcript pipeline drops entirely.
• Diarization-aware: include speaker labels in the chunk metadata so the user can filter by speaker. Pyannote's 2024 speaker-diarization pipeline reaches roughly 11% DER (diarization error rate) on the AMI corpus; once paired with Whisper, you can answer queries like "what did the second speaker say about pricing" that pure-transcript retrieval cannot, since Whisper alone collapses all speakers into a single text stream.

For most production audio RAG (call center transcripts, lecture archives), transcript-only retrieval is the right starting point. Add acoustic embeddings only if your use case demands non-linguistic content. Gong.io and Chorus.ai, both 8-figure-ARR conversation-intelligence products as of 2024, run on what is essentially "Whisper plus diarization plus text RAG"; the acoustic-embedding layer is only worth adding when your users start asking the kind of "find the laughter" or "find the angry tone" queries that text cannot serve.

33.2.6 Modality-Aware Reranking

Cross-modal retrieval is noisier than within-modal retrieval. A text-to-image query can return semantically related but spurious matches, the same picture of dogs surfaces for "dog playing" and "dog at the beach". The standard fix is a reranker: a more expensive model (cross-encoder, VLM) scores the top-k retrieved items more carefully.

For image retrieval, a small VLM (Qwen2-VL-2B, Idefics2-8B) reranks by computing a relevance score between the query text and each candidate image. The pattern:

1. Joint-embedding retrieval returns top-50 candidates (cheap).
2. VLM reranker scores each candidate against the query (medium-cost).
3. Top-5 reranked images go to the final generation VLM.

This two-stage retrieve-then-rerank pattern, identical to text RAG (Section 32.1), improves answer quality by 5 to 15 percentage points on most benchmarks at modest extra latency (50 to 200 ms).

33.2.7 Production Trade-offs: Cost, Latency, Quality