Image Generation & Vision-Language Models

"I dream in latent space now. Every pixel is just a probability waiting to be sampled."

Pixel, Pixel-Drunk AI Agent

1. Diffusion Models for Image Generation

Diffusion models generate images by learning to reverse a noise-addition process. During training, the model learns to predict and remove noise from progressively corrupted images. During inference, it starts from pure random noise and iteratively denoises it into a coherent image, guided by a text prompt. This elegant framework has become the dominant paradigm for high-quality image generation, surpassing earlier approaches like GANs and VAEs in both quality and controllability.

The Forward and Reverse Process

The forward process adds Gaussian noise to a clean image over T timesteps until the image becomes pure noise. The reverse process, parameterized by a neural network (typically a U-Net or transformer), learns to denoise at each step. The key insight is that predicting the noise added at each step is equivalent to learning the score function (gradient of the log probability) of the data distribution. Figure 27.1.3 illustrates the forward and reverse diffusion processes.

Figure 27.1.3: The Markov chain of forward (reverse) diffusion process of generating a sample by slowly adding (removing) noise. Source: Ho et al., 2020. Denoising Diffusion Probabilistic Models. Diagram from Lilian Weng, What are Diffusion Models?

Latent Diffusion (Stable Diffusion)

Running diffusion directly in pixel space is computationally expensive. Stable Diffusion solves this by operating in the latent space of a pre-trained variational autoencoder (VAE). The image is first encoded into a compact latent representation (typically 64x64 instead of 512x512), diffusion happens in this smaller space, and the result is decoded back to pixel space. This idea of working in a compressed latent space echoes the embedding concepts from Section 19.1, where high-dimensional data is mapped to dense representations. This reduces computation by roughly 50x while maintaining image quality. Code Fragment 27.1.2 below puts this into practice.

# Stable Diffusion XL: load the SDXL pipeline with a DPM++ scheduler
# for high-quality text-to-image generation using classifier-free guidance
# in the latent space of a pre-trained VAE.
import torch
from diffusers import StableDiffusionXLPipeline, DPMSolverMultistepScheduler

# Load Stable Diffusion XL with optimized scheduler
pipe = StableDiffusionXLPipeline.from_pretrained(
 "stabilityai/stable-diffusion-xl-base-1.0",
 torch_dtype=torch.float16,
 variant="fp16",
)
pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)
pipe = pipe.to("cuda")

# Generate with classifier-free guidance
prompt = "A serene Japanese garden with cherry blossoms, watercolor style"
negative_prompt = "blurry, low quality, distorted"

image = pipe(
 prompt=prompt,
 negative_prompt=negative_prompt,
 num_inference_steps=30,
 guidance_scale=7.5, # Higher = more prompt adherence
 width=1024,
 height=1024,
).images[0]

image.save("japanese_garden.png")

import replicate

output = replicate.run(
 "stability-ai/sdxl:7762fd07cf82c948",
 input={
 "prompt": "A serene Japanese garden with cherry blossoms, watercolor style",
 "negative_prompt": "blurry, low quality, distorted",
 "width": 1024,
 "height": 1024,
 },
)
# output is a URL to the generated image

Flow Matching and Rectified Flows

Flow matching is a newer paradigm that learns a velocity field transporting samples from a noise distribution to the data distribution along straight paths. Unlike diffusion models that follow curved trajectories through noise space, flow matching creates straighter paths that require fewer sampling steps. Stable Diffusion 3 and Flux use this approach, achieving higher quality with fewer inference steps (often 4 to 8 steps instead of 20 to 50 for traditional diffusion). Code Fragment 27.1.3 below puts this into practice.

# Using Flux (flow matching model) via the diffusers library
from diffusers import FluxPipeline

pipe = FluxPipeline.from_pretrained(
 "black-forest-labs/FLUX.1-schnell",
 torch_dtype=torch.bfloat16,
)
pipe = pipe.to("cuda")

# Flux-schnell needs only 4 steps thanks to flow matching
image = pipe(
 prompt="A cyberpunk cityscape at sunset, neon reflections on wet streets",
 num_inference_steps=4,
 guidance_scale=0.0, # Schnell uses guidance distillation
 height=1024,
 width=1024,
).images[0]

DALL-E and Midjourney

OpenAI's DALL-E 3 generates images by first using GPT-4 to rewrite user prompts into detailed descriptions, then feeding those descriptions to a diffusion model. This prompt rewriting step significantly improves output quality, because users often write terse prompts that lack the detail needed for good generation. Midjourney, while proprietary, is notable for its aesthetic quality and has become the benchmark for artistic image generation. Both systems demonstrate that engineering around the core model (prompt rewriting, aesthetic fine-tuning, safety filtering) is as important as the model architecture itself. Code Fragment 27.1.6 below puts this into practice.

# DALL-E 3 image generation via the OpenAI API
# with quality, size, and style parameters for fine-grained control.
from openai import OpenAI

client = OpenAI()

# DALL-E 3 via the OpenAI API
response = client.images.generate(
 model="dall-e-3",
 prompt="An isometric illustration of a cozy bookshop with warm lighting",
 size="1024x1024",
 quality="hd",
 n=1,
)

image_url = response.data[0].url
revised_prompt = response.data[0].revised_prompt
print(f"Revised prompt: {revised_prompt}")
print(f"Image URL: {image_url}")

2. Controlled Image Generation

While text prompts offer creative flexibility, many applications require more precise spatial and stylistic control. ControlNet, IP-Adapter, and related techniques add conditioning signals beyond text, enabling edge-guided generation, pose transfer, style reference, and inpainting.

ControlNet

ControlNet adds spatial conditioning to a pre-trained diffusion model by attaching a trainable copy of the encoder blocks. The original model weights are frozen, and the ControlNet learns to inject spatial information (edges, depth maps, poses, segmentation masks) into the generation process. This allows precise control over the structure of generated images while preserving the model's learned generative capabilities. Figure 27.1.4 shows the ControlNet architecture. Code Fragment 27.1.4 below puts this into practice.

# ControlNet with Canny edge detection: extract edges from a reference
# image and use them to guide Stable Diffusion generation.
from diffusers import StableDiffusionControlNetPipeline, ControlNetModel
from diffusers.utils import load_image
import cv2
import numpy as np
from PIL import Image

# Load a Canny edge ControlNet
controlnet = ControlNetModel.from_pretrained(
 "lllyasviel/control_v11p_sd15_canny",
 torch_dtype=torch.float16,
)

pipe = StableDiffusionControlNetPipeline.from_pretrained(
 "runwayml/stable-diffusion-v1-5",
 controlnet=controlnet,
 torch_dtype=torch.float16,
).to("cuda")

# Extract edges from a reference image
ref_image = load_image("reference_building.jpg")
ref_np = np.array(ref_image)
edges = cv2.Canny(ref_np, 100, 200)
control_image = Image.fromarray(edges)

# Generate with edge guidance
result = pipe(
 prompt="A futuristic glass skyscraper, photorealistic",
 image=control_image,
 num_inference_steps=30,
 controlnet_conditioning_scale=0.8,
).images[0]

IP-Adapter: Style and Subject Transfer

IP-Adapter (Image Prompt Adapter) enables using images as prompts alongside text. It works by encoding a reference image through CLIP's image encoder and injecting those features into the cross-attention layers of the diffusion model. This enables style transfer (generating images in the style of a reference) and subject consistency (maintaining the same character or object across multiple generations).

3. Vision Encoders: Bridging Pixels and Language

While diffusion models generate images from text, vision encoders work in the opposite direction: they convert images into representations that language models can understand. The evolution from ViT to CLIP to SigLIP has produced increasingly powerful visual representations that form the backbone of modern multimodal AI.

Vision Transformer (ViT)

The Vision Transformer applies the transformer architecture directly to images by splitting them into fixed-size patches (typically 16x16 or 14x14 pixels), flattening each patch into a vector, and processing the sequence of patch embeddings with standard transformer layers. A special [CLS] token aggregates information across all patches, producing a single vector representation of the entire image. ViT demonstrated that transformers, originally designed for text, work remarkably well for vision when trained on enough data.

CLIP: Contrastive Language-Image Pre-training

CLIP jointly trains a vision encoder and a text encoder to embed images and their textual descriptions into a shared vector space. Matching image-text pairs are pushed close together while non-matching pairs are pushed apart. Trained on 400 million image-text pairs scraped from the internet, CLIP learns visual concepts from natural language supervision, enabling zero-shot image classification, image search, and serving as the text encoder for diffusion models like Stable Diffusion. Figure 27.1.5 depicts CLIP's contrastive training of image and text encoders.

Image Classification with CLIP

CLIP's shared embedding space enables zero-shot image classification without task-specific training. You encode the image and a set of candidate text labels, then find the label whose embedding is closest to the image embedding. This generalizes across domains without fine-tuning. Code Fragment 27.1.4 below puts this into practice.

# Zero-shot image classification with CLIP: compute cosine similarity
# between image and text embeddings, then softmax to get probabilities.
from transformers import CLIPProcessor, CLIPModel
from PIL import Image

model = CLIPModel.from_pretrained("openai/clip-vit-large-patch14")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-large-patch14")

image = Image.open("photo.jpg")
labels = ["a photo of a cat", "a photo of a dog", "a photo of a car"]

inputs = processor(text=labels, images=image, return_tensors="pt", padding=True)
outputs = model(**inputs)

# Cosine similarity between image and each label
logits_per_image = outputs.logits_per_image
probs = logits_per_image.softmax(dim=1)

for label, prob in zip(labels, probs[0]):
 print(f"{label}: {prob.item():.3f}")

4. Vision-Language Models

Vision-language models (VLMs) go beyond CLIP's contrastive matching to enable full visual reasoning. These models accept images alongside text in their input and generate free-form text responses. They can describe images, answer questions about visual content, extract structured data from screenshots, and reason about spatial relationships. Calling these models through the standard LLM APIs (Chapter 10) follows the same patterns as text-only models. The rapid evolution of VLMs represents one of the most impactful recent developments in AI. Code Fragment 27.1.7 below puts this into practice.

Comparison of Vision-Language Models

Using GPT-4V for Visual Reasoning

This snippet sends an image to GPT-4V and extracts structured visual reasoning from the response.

# GPT-4o Vision API: encode an image as base64 and send it
# alongside a text prompt for visual reasoning and analysis.
from openai import OpenAI
import base64

client = OpenAI()

# Encode image to base64
with open("chart.png", "rb") as f:
 image_b64 = base64.b64encode(f.read()).decode()

response = client.chat.completions.create(
 model="gpt-4o",
 messages=[{
 "role": "user",
 "content": [
 {"type": "text", "text": "Analyze this chart. What trends do you see?"},
 {
 "type": "image_url",
 "image_url": {
 "url": f"data:image/png;base64,{image_b64}",
 "detail": "high" # "high" for detailed analysis
 }
 }
 ]
 }],
 max_tokens=1000,
)

print(response.choices[0].message.content)

Open-Source VLMs with LLaVA

LLaVA (Large Language and Vision Assistant) is the most influential open-source VLM architecture. It connects a CLIP vision encoder to a language model through a simple projection layer (either a linear layer or a small MLP). The model is trained in two stages: first aligning the vision-language features using image-caption pairs, then fine-tuning on visual instruction data. This simple and effective design has spawned many variants and remains the template for most open-source VLMs. Code Fragment 27.1.6 below puts this into practice.

# LLaVA visual question answering: load the LLaVA-v1.6-Mistral model
# and extract structured information from a receipt image.
from transformers import LlavaNextProcessor, LlavaNextForConditionalGeneration
from PIL import Image

model_id = "llava-hf/llava-v1.6-mistral-7b-hf"
processor = LlavaNextProcessor.from_pretrained(model_id)
model = LlavaNextForConditionalGeneration.from_pretrained(
 model_id, torch_dtype=torch.float16, device_map="auto"
)

image = Image.open("receipt.jpg")
prompt = "[INST] <image>\nExtract the total amount and date from this receipt. [/INST]"

inputs = processor(prompt, image, return_tensors="pt").to("cuda")
output = model.generate(**inputs, max_new_tokens=200)
result = processor.decode(output[0], skip_special_tokens=True)
print(result)

Gemini: Natively Multimodal

Google's Gemini models represent a different approach to multimodal AI. Rather than bolting a vision encoder onto a language model, Gemini is trained from the ground up on interleaved text, image, audio, and video data. This native multimodal training enables capabilities that are difficult to achieve with adapter-based approaches, such as understanding spatial relationships across multiple images, processing long videos, and handling interleaved image-text sequences. Gemini 2.0 extends this to native image generation, combining understanding and generation in a single model. Code Fragment 27.1.6 below puts this into practice.

# Gemini multi-image reasoning: pass two images in a single request
# for comparative analysis using Gemini 2.0 Flash.
import google.generativeai as genai
from PIL import Image

genai.configure(api_key="YOUR_API_KEY")
model = genai.GenerativeModel("gemini-2.0-flash")

# Multi-image reasoning
img1 = Image.open("before.jpg")
img2 = Image.open("after.jpg")

response = model.generate_content([
 "Compare these two images and describe what changed:",
 img1,
 img2,
])
print(response.text)

Exercises