Deep Learning Essentials

They told me backpropagation would help me learn from my mistakes. Now I just make them faster, in parallel, on eight GPUs.

Tensor, Backprop-Bruised AI Agent

0.2.1 Neural Network Fundamentals

0.2.1.1 The Perceptron: Your First Artificial Neuron

In 1958, Frank Rosenblatt built a machine called the Mark I Perceptron that could learn to distinguish shapes. The New York Times declared it the embryo of a computer that would "be able to walk, talk, see, write, reproduce itself and be conscious of its existence." The reality was far humbler: a perceptron is the simplest possible neural network, a single unit that takes multiple inputs, multiplies each by a learnable weight, adds a bias term, and passes the result through an activation function to produce an output. Think of it as a tiny decision-maker that draws a single straight line through your data.

What it is: A linear classifier that computes $y = f(w_{1}x_{1} + w_{2}x_{2} + ... + w_{n}x_{n} + b)$, where $f$ is an activation function.

Why it matters: The perceptron is the conceptual atom of every neural network. Understanding it thoroughly makes the rest of deep learning far more intuitive.

Figure 0.2.2: Anatomy of a single perceptron (artificial neuron). Each input is multiplied by a weight, the products are summed with a bias, and an activation function produces the output.

0.2.1.2 Multi-Layer Perceptrons: Stacking LEGO Bricks

A single perceptron can only learn linear boundaries. To capture complex patterns, we stack layers of perceptrons together, forming a Multi-Layer Perceptron (MLP). Think of it exactly like building with LEGO bricks. A single brick is not very interesting. But when you snap bricks together in layers, you can build anything: a house, a castle, a spaceship. Each layer in a neural network transforms its input in a simple way, but the composition of many layers can represent remarkably complex functions.

An MLP has three types of layers:

• Input layer: Receives the raw features (no computation happens here).
• Hidden layers: The intermediate LEGO layers. Each neuron computes a weighted sum followed by an activation. This is where the network learns its internal representations.
• Output layer: Produces the final prediction (a class probability, a regression value, etc.).

0.2.1.3 Activation Functions

What they are: Non-linear functions applied after the weighted sum in each neuron. Without them, stacking layers would be pointless, because a composition of linear functions is just another linear function.

Why they matter: Activation functions are what give neural networks the ability to model non-linear relationships. They are the key ingredient that separates a deep network from a simple linear regression.

Example 1: Building and running an MLP in NumPy

# ReLU and softmax from scratch: ReLU zeros out negatives,
# softmax converts raw logits into a valid probability distribution.
import numpy as np

def relu(z):
 return np.maximum(0, z)

def softmax(z):
 exp_z = np.exp(z - np.max(z)) # subtract max for numerical stability
 return exp_z / exp_z.sum()

# A tiny 2-layer MLP: 3 inputs, 4 hidden, 2 outputs
np.random.seed(42)
W1 = np.random.randn(3, 4) * 0.5 # (3, 4) weight matrix
b1 = np.zeros(4)
W2 = np.random.randn(4, 2) * 0.5 # (4, 2) weight matrix
b2 = np.zeros(2)

# Forward pass with a sample input
x = np.array([1.0, 2.0, 3.0])
hidden = relu(x @ W1 + b1) # hidden layer
output = softmax(hidden @ W2 + b2) # output probabilities

print("Hidden activations:", hidden.round(3))
print("Output probabilities:", output.round(3))
print("Predicted class:", np.argmax(output))

0.2.2 Backpropagation and the Chain Rule

What it is: Backpropagation (backprop) is the algorithm that computes how much each weight in the network contributed to the overall error. It works by applying the chain rule of calculus in reverse, propagating the error signal from the output layer back through every hidden layer.

Why it matters: Without backprop, we would have no efficient way to train deep networks. It is the engine that makes gradient descent possible for networks with millions (or billions) of parameters. PyTorch's autograd system, which we cover hands-on in Section 0.3, automates backpropagation so you rarely need to compute derivatives by hand.

How it works: Consider a simple network where an input $x$ flows through two functions. The forward pass computes $h = f(x)$ and then $y = g(h)$. To find how the loss $L$ changes with respect to the input parameter, the chain rule tells us:

Backprop computes these derivatives from right to left (output to input), reusing intermediate results at each layer.

0.2.2.1 A Concrete Numerical Example

Let us walk through backpropagation with actual numbers. Consider a single neuron with one input, one weight, a bias, and a ReLU activation. The target is $y_{true} = 1.0$, and we use mean squared error loss.

Figure 0.2.4: Backpropagation through a single neuron. The forward pass computes the loss (left to right), then gradients flow backward (right to left) via the chain rule.

Let us trace through this step by step:

1. Forward pass: $z = w \cdot x + b = 0.5 \cdot 2 + 0.5 = 1.5$. After ReLU: $a = \max(0, 1.5) = 1.5$. Loss: $L = (1.5 - 1.0)^{2} = 0.25$.
2. Backward pass (chain rule):
   • $dL/da = 2(a - y_{true}) = 2(1.5 - 1.0) = 1.0$
   • $da/dz = 1$ (ReLU derivative is 1 when z > 0)
   • $dz/dw = x = 2.0$
   • By the chain rule: $dL/dw = 1.0 \times 1 \times 2.0 = \textbf{2.0}$
3. Weight update (with learning rate 0.1): $w_{new} = 0.5 - 0.1 \times 2.0 = \textbf{0.3}$

The weight decreased, which will push the output closer to 1.0 on the next forward pass. This is exactly what gradient descent does: it nudges every parameter in the direction that reduces the loss.

0.2.3 Regularization Techniques

In Section 0.1, you learned that overfitting occurs when a model memorizes training data instead of learning general patterns. Deep networks, with their enormous capacity, are especially prone to this. Here are the three most important tools for fighting overfitting in deep learning.

0.2.3.1 Dropout

What it is: During each training step, dropout randomly "turns off" a fraction of neurons (typically 20% to 50%) by setting their outputs to zero.

Why it matters: It prevents co-adaptation, where neurons become overly dependent on specific other neurons. By randomly removing neurons during training, the network is forced to learn redundant, robust representations.

When to use it: Almost always in fully connected layers. Common dropout rates are 0.1 to 0.5. Use lower rates (0.1) for smaller networks and higher rates (0.3 to 0.5) for larger ones. At test time, dropout is turned off and activations are scaled accordingly.

0.2.3.2 Batch Normalization

What it is: Batch normalization (BatchNorm) normalizes the outputs of a layer across the current mini-batch to have zero mean and unit variance. It then applies two learnable parameters (scale and shift) so the network can undo the normalization if that is optimal.

Why it matters: It dramatically stabilizes and accelerates training. Without it, as weights in early layers change, the distribution of inputs to later layers shifts constantly (a problem called internal covariate shift). BatchNorm keeps these distributions stable.

When to use it: In most deep networks, especially CNNs. Place it after the linear/convolutional layer and before the activation function. For very small batch sizes, consider Softmax instead (which normalizes across features rather than across the batch). Layer Normalization is the standard choice in Transformer architectures (Section 3.3).

0.2.3.3 Weight Initialization

What it is: The strategy used to set the initial values of weights before training begins.

Why it matters: Poor initialization can cause gradients to either vanish (shrink to near zero) or explode (grow uncontrollably) as they propagate through layers. Both scenarios make training extremely slow or impossible.

Example 2: Dropout and BatchNorm in a PyTorch model

# RobustMLP: a production-style feedforward network with BatchNorm,
# Dropout, and He initialization to prevent vanishing gradients.
import torch
import torch.nn as nn

class RobustMLP(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim, dropout_rate=0.3):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(input_dim, hidden_dim),
            nn.BatchNorm1d(hidden_dim), # normalize before activation
            nn.ReLU(),
            nn.Dropout(dropout_rate), # randomly zero 30% of neurons
            nn.Linear(hidden_dim, hidden_dim),
            nn.BatchNorm1d(hidden_dim),
            nn.ReLU(),
            nn.Dropout(dropout_rate),
            nn.Linear(hidden_dim, output_dim),
            )
        # Kaiming initialization for ReLU layers
        for m in self.modules():
            if isinstance(m, nn.Linear):
                nn.init.kaiming_normal_(m.weight, nonlinearity='relu')

                # Forward pass: define computation graph
                def forward(self, x):
                    return self.net(x)

                    model = RobustMLP(input_dim=10, hidden_dim=64, output_dim=3)
                    print(model)
                    print(f"Total parameters: {sum(p.numel() for p in model.parameters()):,}")

0.2.4 Convolutional Neural Networks (CNNs) Overview

What they are: CNNs are specialized neural networks designed for data with spatial structure (images, audio spectrograms, time series). Instead of connecting every neuron to every input, a CNN uses small learnable filters (also called kernels) that slide across the input, detecting local patterns.

Why they matter: Before CNNs, computer vision required hand-crafted feature engineering. CNNs learn the features directly from raw pixels. The same idea of local pattern detection underpins many modern architectures, including those used in speech recognition for conversational AI.

How they work: A CNN typically alternates between two types of layers:

• Convolutional layers: Apply small filters (e.g., 3×3) across the spatial dimensions. Each filter learns to detect a specific pattern (edges, corners, textures). The output is called a feature map.
• Pooling layers: Reduce the spatial dimensions by taking the maximum or average over small regions (e.g., 2×2). This makes the representation more compact and translation-invariant.

After several convolutional and pooling layers, the output is flattened and passed through fully connected layers for the final prediction.

0.2.5 Training Best Practices

Knowing the architecture is only half the battle. How you train a deep network matters just as much as the network's structure. Here are the essential techniques that separate productive training runs from frustrating ones.

0.2.5.1 Learning Rate Scheduling

What it is: A strategy for adjusting the learning rate during training rather than keeping it fixed.

Why it matters: A learning rate that is too high causes the loss to oscillate or diverge. One that is too low wastes compute time. The optimal learning rate often changes as training progresses: you want to take large steps initially (to make fast progress) and smaller steps later (to fine-tune).

Common schedules:

• Step decay: Multiply the learning rate by a factor (e.g., 0.1) every N epochs.
• Cosine annealing: Smoothly decrease the learning rate following a cosine curve. Very popular in practice.
• Warmup + decay: Start with a tiny learning rate, ramp up linearly over the first few hundred steps, then decay. This is standard for Transformer training and critical for the LLM work you will do later.
• ReduceLROnPlateau: Monitor the validation loss and reduce the learning rate when improvement stalls.

0.2.5.2 Early Stopping

What it is: Monitoring the validation loss during training and stopping when it has not improved for a specified number of epochs (the "patience").

Why it matters: It is the simplest and most effective defense against overfitting. Training too long almost always leads to overfitting, so stopping at the right moment saves both time and model quality.

When to use it: Almost always. Set patience to 5 to 10 epochs and save the best model checkpoint based on validation performance.

0.2.5.3 Gradient Clipping

What it is: Capping the magnitude of gradients during backpropagation, either by value or by norm.

Why it matters: In deep or recurrent networks, gradients can sometimes "explode" (grow to enormous values), causing wildly unstable weight updates. Gradient clipping puts a ceiling on how large any single update can be.

When to use it: Always for RNNs and Transformers. A common setting is to clip the global gradient norm to 1.0.

Example 3: Complete training loop with scheduling, early stopping, and gradient clipping

# Complete training loop with cosine LR scheduling, gradient clipping,
# and early stopping that halts when validation loss stops improving.
import torch
import torch.nn as nn
import torch.optim as optim

# Setup: model, optimizer, scheduler
model = nn.Sequential(nn.Linear(10, 64), nn.ReLU(), nn.Linear(64, 1))
optimizer = optim.Adam(model.parameters(), lr=1e-3)
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=50)
criterion = nn.MSELoss()

# Simulated data
torch.manual_seed(42)
X_train = torch.randn(200, 10)
y_train = torch.randn(200, 1)
X_val = torch.randn(50, 10)
y_val = torch.randn(50, 1)

# Early stopping setup
best_val_loss = float('inf')
patience, patience_counter = 5, 0

for epoch in range(50):
    # Training step
    model.train()
    optimizer.zero_grad()
    loss = criterion(model(X_train), y_train)
    loss.backward()

    # Gradient clipping: cap the norm at 1.0
    torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)

    optimizer.step()
    scheduler.step()

    # Validation step
    model.eval()
    with torch.no_grad():
        val_loss = criterion(model(X_val), y_val).item()

        # Early stopping check
        if val_loss < best_val_loss:
            best_val_loss = val_loss
            patience_counter = 0
            torch.save(model.state_dict(), 'best_model.pt') # save best
        else:
            patience_counter += 1

            if patience_counter >= patience:
                print(f"Early stopping at epoch {epoch}")
                break

                if epoch % 10 == 0:
                    lr = optimizer.param_groups[0]['lr']
                    print(f"Epoch {epoch:3d} | Train Loss: {loss.item():.4f} | Val Loss: {val_loss:.4f} | LR: {lr:.6f}")

                    print(f"Best validation loss: {best_val_loss:.4f}")

0.2.5.4 The Optimizer Ladder: From SGD to AdamW

Plain stochastic gradient descent (SGD) is the conceptual atom of training, but the optimizers used to actually train modern LLMs evolved through six distinct generations, each fixing a specific weakness of its predecessor. Reading the ladder in order is the fastest way to develop an intuition for why AdamW is the de-facto default for transformer training in 2026.

Why AdamW won. The original Adam implements L2 regularization by adding $\lambda w$ to the gradient before it enters the adaptive moment estimates. That coupling means the effective weight-decay rate gets divided by the running RMS of past gradients, so parameters with historically large gradients are barely regularized while parameters with small gradients are over-regularized. AdamW decouples weight decay by applying $w \leftarrow w - \eta \lambda w$ as a separate step that does not pass through the adaptive scaling. The fix is one line of code, but it consistently improves generalization for transformers and is the reason essentially every modern LLM (BERT, GPT-2 onward, LLaMA, Mistral, Qwen, Gemma) trains with AdamW rather than Adam. The full LLM-specific recipe (typical hyperparameters, interaction with cosine schedules, gradient accumulation) lives in Section 6.5; what matters here is the genealogy and the one-line intuition.

Example 4: Swapping in AdamW

# AdamW is a drop-in replacement for Adam with decoupled weight decay.
# Typical transformer hyperparameters: lr=1e-4 to 3e-4, weight_decay=0.01 to 0.1.
import torch
import torch.nn as nn
import torch.optim as optim

model = nn.Linear(512, 512)

# Adam (old default): weight_decay is folded into the gradient -> coupled to lr scaling.
opt_adam  = optim.Adam(model.parameters(),  lr=3e-4, weight_decay=0.01)

# AdamW (modern default): weight_decay is applied directly to the parameters.
opt_adamw = optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.01)

# Use AdamW for transformers; the API is identical to Adam.

The genealogy table above explains why AdamW exists; the diagram below makes the coupling fix itself visible side by side. Figure 0.2.6 traces both update rules for the same parameter and the same gradient history, so the only difference is the placement of the weight-decay term.

0.2.6 Putting It All Together: Neural Network Design Checklist

Here is a mental model for how all these pieces connect. When you design and train a neural network:

1. Architecture: Choose your layers (MLPs for tabular data, CNNs for images, Transformers for sequences). Remember the LEGO analogy: each layer is a brick, and depth gives you expressiveness.
2. Activation functions: Use ReLU (or GELU for Transformers) in hidden layers. Use softmax for multi-class outputs, sigmoid for binary.
3. Initialization: Kaiming for ReLU networks, Xavier for tanh/sigmoid.
4. Regularization: Add BatchNorm (or LayerNorm) and dropout between layers.
5. Training loop: Use learning rate warmup plus cosine decay, gradient clipping (especially for Transformers), and early stopping. For hands-on practice building training loops, continue to Section 0.3: PyTorch Tutorial.

This checklist will serve you throughout the book. In the next section, you will implement all of these ideas hands-on with PyTorch.

Exercises

dy = 2 * (y_pred - y_true)        # gradient of MSE w.r.t. y_pred
dW = dy.unsqueeze(-1) @ x.unsqueeze(-2)   # outer product, shape matches W
db = dy
dx = W.T @ dy                     # gradient w.r.t. input, propagated upstream