Convolutional Neural Networks

What a Computer Sees

There are no objects in an image file. There is a grid of numbers, one per pixel, each giving a brightness from 0 (black) to 255 (white). A colour photo has three such grids: red, green and blue.

A small 224 × 224 colour photo:
  224 × 224 × 3 = 150,528 numbers

Your task: write a function that maps
those 150,528 numbers to "cat" or "dog".

Every idea you have will break. Move the cat two pixels left and every number changes. Turn the light down and every number changes. The cat is still a cat.

The Three Ingredients of Machine Learning

Training means adjusting the model's numbers until the loss is small. That is the whole idea. The next eight slides are only the details of how.

Reading the Figure

The solid line is the function: what the neuron outputs. The dashed line is its slope at each point, which is the number training uses to decide how to adjust the weights.

Look at sigmoid on the left. Away from the centre the dashed line is almost flat on the floor. A slope near zero means "adjusting this weight changes nothing", so learning stops. That is the vanishing gradient problem, and it is why deep networks were considered untrainable for years.

Which One to Use

If you are unsure, use ReLU in the hidden layers and softmax at the output. That combination covers the overwhelming majority of real networks.

The Whole Computation, In One Expression

Doing this one neuron at a time is tedious, so a whole layer is written as a matrix multiply. For a layer with weights W and biases b:

Chain three of those together and you have the entire network:

That expression is a complete neural network. Everything that follows in this deck is about choosing a better shape for W, and about how to find good values for the numbers inside it.

Counting the Parameters

Layer sizes: 3 → 4 → 4 → 2

weights: 3×4 + 4×4 + 4×2 = 12 + 16 + 8 = 36
biases :   4  +   4  +   2            = 10
                                  total = 46

Forty-six numbers to learn. A network that recognizes handwritten digits needs roughly 100,000. GPT-scale language models need hundreds of billions. The arithmetic on this slide does not change; only the size of the matrices does.

The Two You Will Actually Use

Mean squared error — for predicting a quantity, such as a house price:

Cross-entropy — for choosing a category, which is what classification means:

Cross-entropy is worth a moment. If the network gives the right answer a probability of 1.0, the loss is −log(1) = 0: no penalty. As that probability approaches zero the loss grows without limit. It punishes confident mistakes far more harshly than uncertain ones, which is exactly the behaviour you want.

Worked Example — Three Predictions, One Truth

The correct answer is "cat". Here is what the loss says about three different networks:

Note the last row. Being confidently wrong costs 460 times more than being confidently right. That steep penalty is what pushes the weights hard in the right direction.

What a Gradient Is

For a single weight, the gradient ∂L/∂w answers one question: if I increase this weight slightly, does the loss go up or down, and by how much?

• Gradient is positive → increasing the weight makes things worse → decrease it.
• Gradient is negative → increasing the weight makes things better → increase it.
• Gradient is near zero → this weight is not the problem → leave it.

Subtracting the gradient handles all three cases at once, which is why the update rule is a single line.

The Update Rule, Term by Term

Every weight in the network is updated by this same rule, independently, at every step. There is no second algorithm hiding behind it.

Three Names for the Same Thing

When somebody says "SGD" they almost always mean mini-batch. The batch size is chosen to keep the hardware busy, and 32 to 256 is the usual range.

The Trick: Work Backwards

Start at the loss, where the error is known exactly. Ask the last layer: given this error, how much did each of your weights contribute, and how much error should I pass to the layer before you? Then ask that layer the same question. Repeat to the start.

Each layer only needs the number handed down from the layer above it, plus values it already saved during the forward pass. Nothing is recomputed from scratch.

The Chain Rule, Which Is All This Is

Read right to left: how the weight affects the sum, times how the sum affects the activation, times how the activation affects the loss. Multiply the three and you have how the weight affects the loss.

This is ordinary calculus from a first course, applied mechanically. The insight in backpropagation is not the mathematics; it is the order of evaluation that lets every weight share the work.

Why This Matters Enormously

For a network with 25 million weights that is the difference between 25 million forward passes and two. Backpropagation is the reason training a large network is possible at all, and it is why the field stalled until it became widely understood in 1986.

Words You Will See in Every Tutorial

The Training Loop — Five Steps, Repeated

for epoch in range(num_epochs):
    for batch_x, batch_y in training_data:

        # 1. Forward: what does the network say?
        prediction = model(batch_x)

        # 2. Loss: how wrong is that?
        loss = loss_function(prediction, batch_y)

        # 3. Clear the gradients from last time
        optimizer.zero_grad()

        # 4. Backward: blame every weight
        loss.backward()

        # 5. Update: w ← w − η · gradient
        optimizer.step()

That is a complete PyTorch training loop. It is the same five steps whether the model has 46 parameters or 46 billion, and it is the same loop used to train every network in the rest of this deck.

The Loss Curve Is Your Instrument Panel

Training loss is measured on the data the network learns from. Validation loss is measured on data it has never seen, held back deliberately. The gap between them is the single most informative thing in machine learning:

• Falling together, staying close — healthy, which is what this run shows.
• Training falls, validation rises — overfitting: it is memorizing rather than learning.
• Neither falls — underfitting: too small a model, or too low a learning rate.

End of Part 1

You now have the complete machinery: neurons, layers, a loss, gradients, backpropagation and a training loop. Everything in Part 2 reuses it without modification.

The next slide asks one question: what happens when we point this exact machinery at a photograph?

Failure 2 — No Translation Structure

A dense layer has an independent weight for every pixel position. A cat in the top-left and the same cat in the bottom-right activate two completely disjoint sets of weights.

The network must learn "cat" separately at every one of the 50,176 spatial positions. Nothing in the architecture says the two are related, so nothing transfers.

Failure 3 — Locality Is Discarded

Flattening the image destroys adjacency. Pixel (10, 10) and pixel (10, 11) are neighbours; after flatten they are indices 2,250 and 2,253 with no relationship the layer can see. Permute all input pixels with a fixed permutation and a dense network trains to the same accuracy. That is the tell: it never used the spatial structure.

What Convolution Changes

The last row is the one that matters most. A 3×3, 3→64 conv layer has 1,792 parameters whether the input is 32×32 or 4096×4096.

Historical Note

Hubel and Wiesel (1962) found cells in the cat visual cortex that respond to edges in a small region of the visual field, and complex cells that pool over simple cells. Fukushima's Neocognitron (1980) turned that into an architecture. LeCun's LeNet-5 (1998) added backpropagation and shipped it on cheque digit recognition. The ideas are old; the compute and the labelled data arrived in 2012.

The Definition

Slide a small weight matrix (the kernel or filter) over the input. At each position take the elementwise product with the patch underneath and sum it to a single number.

Strictly this is cross-correlation. True convolution flips the kernel first. Because the kernel is learned, the flip is irrelevant in practice, and every deep learning framework implements cross-correlation and calls it convolution.

The Input (5×5) and Kernel (3×3)

X =
  1  2  3  0  1
  0  1  2  3  1
  1  0  1  2  0
  2  1  0  1  3
  1  2  1  0  2

W =
  1  0  -1
  1  0  -1
  1  0  -1

# vertical edge
# detector

This kernel computes left column minus right column, summed over three rows. It fires positively on a light-to-dark vertical transition and negatively on dark-to-light.

Sliding the Window

Reading the Output

Nine dot products, nine outputs, and critically the same nine weights every time. Whatever the kernel has learned to detect, it is detected everywhere at once. That is weight sharing, and it is the single most important structural idea in the architecture.

Notice the output shrank from 5×5 to 3×3. A K×K kernel with no padding loses K-1 pixels in each dimension because the window cannot hang off the edge. Stack ten 3×3 layers and a 32×32 image erodes to 12×12 purely from boundary effects. Padding, two slides on, fixes that.

Two Words to Learn Now

How Many Numbers Is That?

A layer with 64 filters of size 3×3
reading a 3-channel colour image:

  weights = 3 · 3 · 3 · 64 = 1,728
  biases  =             64 =    64
                            ───────
                              1,792

The fully connected layer on slide 11
needed 150,528,000 for the same job.

That is roughly 84,000 times fewer numbers to learn, and the convolutional version is the one that actually works. Fewer parameters is not a compromise here; it is the reason it generalizes.

Padding — Three Conventions

• Valid (P = 0). Output shrinks by K−1. No invented data.
• Same (P = (K−1)/2 with S = 1). Output size equals input size. This is why odd kernel sizes dominate: 3, 5, 7 give integer padding, 4 does not.
• Full (P = K−1). Output grows by K−1. Rare in forward passes, but it is exactly what the backward pass computes.

Zero padding is the default but it is a lie about the world: it tells the network there are black pixels outside the frame. reflect and replicate padding avoid that artefact and matter for dense prediction tasks such as segmentation and super-resolution.

Worked Examples

The first two rows are the ResNet stem: 224 → 112 → 56 before a single residual block runs.

Stride — Downsampling Inside the Convolution

Stride 2 halves height and width, which cuts the activation tensor to one quarter the elements and the downstream FLOPs with it. Modern architectures prefer strided convolution over pooling for downsampling because the downsampling itself becomes learnable.

Dilation — Reach Without Cost

Stacking dilations 1, 2, 4 with 3×3 kernels reaches a 15×15 receptive field in three layers at full resolution: r = 1 + 2 = 3, then 3 + 4 = 7, then 7 + 8 = 15. Semantic segmentation networks (DeepLab) use this to get context without ever downsampling and having to upsample back.

Parameter and Compute Formulas

One multiply-accumulate is conventionally counted as 2 FLOPs. Note what is absent from the parameter formula: the spatial size. Note what is present in the MAC formula: the spatial size. Parameters and compute scale completely differently, which is why parameter count is a poor proxy for latency.

Worked: AlexNet conv1

Input   : 3 × 224 × 224
Filters : 96 of shape (3, 11, 11), stride 4
Output  : 96 × 55 × 55

params = (11·11·3 + 1) · 96
       = (363 + 1) · 96
       = 34,944

MACs   = 11·11·3 · 96 · 55 · 55
       = 34,848 · 3,025
       = 105.4M  (≈ 0.21 GFLOPs)

The Shape Flow

The 1×1 Convolution Is Not a No-Op

With K = 1 there is no spatial mixing at all, but there is still full mixing across channels. A 1×1, 256→64 conv is a learned linear projection from a 256-dimensional vector to a 64-dimensional vector, applied independently at every pixel.

Parameters: 1·1·256·64 + 64 = 16,448. It is the cheapest way to change channel depth, and it is the workhorse of Inception bottlenecks, ResNet bottleneck blocks, and depthwise separable convolutions. Slide 27 quantifies the saving.

What Convolution Does Not Give You for Free

The honest summary: a CNN is a translation-equivariant model, and every other invariance is bolted on by the training pipeline rather than the architecture.

The Statistical Payoff, Quantified

Compare a dense layer and a conv layer, both mapping a 32×32×3 image to 32×32×64 features:

dense: 3072 · 65536 = 201,326,592
conv : 3·3·3·64 + 64 = 1,792

Fewer parameters is not the point in itself. The point is that each parameter sees far more training signal: a conv weight receives a gradient contribution from all 1,024 spatial positions of every image, so one image gives it 1,024 effective updates rather than one.

The Feature Hierarchy That Emerges

Worked Trace

Eight layers reach 24×24. After the first pool, each 3×3 conv adds (3−1)·2 = 4 pixels rather than 2. After the second pool it adds 8.

Two 3×3 Beat One 5×5 — The VGG Argument

Stacked small kernels are strictly better for a fixed receptive field: fewer parameters, fewer FLOPs, and more nonlinearity, which means more expressive power. This is the observation that made 3×3 the near-universal default after VGG in 2014, and it is why 11×11 and 7×7 kernels vanished from everything except the network stem.

The Effective Receptive Field Is Much Smaller

The theoretical receptive field is the set of pixels that can influence a unit. Luo et al. (2016) measured the gradient magnitude and found the influence is approximately Gaussian and concentrated in the centre: the effective receptive field grows as O(√L) in depth while the theoretical one grows as O(L).

Practical consequence: a network whose theoretical receptive field just barely covers the object will underperform. Design for a theoretical receptive field two to three times the size of the objects you care about, or add dilation, or add explicit global context such as a pooling pyramid.

Dead ReLUs — A Real Failure Mode

If a unit's pre-activation is negative for every input in the dataset, its gradient is zero forever and the unit never recovers. This typically follows a large learning rate driving the bias strongly negative. Symptoms: a large fraction of channels output identically zero, and accuracy plateaus below what the capacity should allow.

Fixes, in order of preference: lower the learning rate, add BatchNorm before the activation, or switch to Leaky ReLU or GELU.

Pooling — Downsample and Buy Local Invariance

Global Average Pooling Replaced the Classifier Head

VGG-16 flattens a 512×7×7 map into 25,088 values and feeds three dense layers: 123.6M of its 138M parameters live in that head, roughly 90 percent of the model, doing almost none of the feature work.

fc1: 25088 × 4096 = 102,760,448
fc2:  4096 × 4096 =  16,777,216
fc3:  4096 × 1000 =   4,096,000
                    ─────────────
                    123,633,664

Global average pooling collapses C×H×W to C by averaging each channel, then a single linear layer maps C→classes. For ResNet-50 that is 2048×1000 = 2.05M parameters instead of 123.6M, and it accepts any input resolution. Introduced in Network-in-Network (2014), adopted by GoogLeNet and everything after.

The Two Trends as You Move Right

The Full Thing in Twelve Lines of PyTorch

model = nn.Sequential(
    nn.Conv2d(3,  32, 3, padding=1),  # find edges
    nn.ReLU(),
    nn.MaxPool2d(2),                  # 32×32 → 16×16

    nn.Conv2d(32, 64, 3, padding=1),  # find shapes
    nn.ReLU(),
    nn.MaxPool2d(2),                  # 16×16 → 8×8

    nn.Flatten(),
    nn.Linear(64*8*8, 10),        # decide
)

That is a working image classifier. Train it with the identical five-step loop from slide 9 — forward, loss, zero_grad, backward, step. Nothing about training changes.

Batch Size Sensitivity

BatchNorm estimates statistics from the batch, so the estimate degrades as the batch shrinks. Below roughly 8 to 16 images per device the noise becomes harmful; at batch size 1 the variance is zero and the layer is meaningless. Detection and segmentation, which often run 2 images per GPU because of resolution, hit this constantly. The workarounds are SyncBatchNorm across devices, frozen BatchNorm from a pretrained model, or switching to GroupNorm.

What Gets Averaged Over

GroupNorm with 32 groups matches BatchNorm accuracy on ImageNet at large batch and clearly beats it below batch size 8, which is why detection frameworks default to it.

Why It Helps — The Story Changed

The original 2015 paper attributed the benefit to reducing "internal covariate shift". Santurkar et al. (2018) tested that directly by injecting noise after BatchNorm to deliberately increase covariate shift, and training still improved. The explanation that survived scrutiny is that BatchNorm smooths the loss landscape: it bounds the gradient magnitude and makes it more predictable, which permits much larger learning rates.

The practical effects are not in dispute and are large:

• Learning rates 10 to 30 times higher become stable.
• Networks past roughly 20 layers become trainable at all.
• Weight initialization stops being delicate.
• Batch statistics add noise, which regularizes; models with BatchNorm often need less dropout.

Ordering and the Bias Term

Conv → BatchNorm → ReLU     # the standard

nn.Conv2d(..., bias=False)   # bias is redundant:
                            # BN subtracts the mean,
                            # so β absorbs any bias

At inference the BatchNorm affine transform is a per-channel scale and shift, so it can be folded into the preceding convolution weights: the layer disappears entirely and costs nothing. Every deployment toolchain does this automatically, which is why BatchNorm is free in production and only expensive during training.

Explicit Regularizers

Do not apply weight decay to BatchNorm's γ and β or to bias terms. Shrinking γ toward zero attenuates the signal for no regularization benefit, and excluding those parameters is typically worth a few tenths of a percent of accuracy.

Dropout Largely Left Convolutional Layers

Standard dropout zeroes individual activations independently. In a convolutional feature map neighbouring activations are strongly correlated, so the surviving neighbours carry the dropped information and little is actually removed. Two things replaced it:

• Spatial dropout drops entire channels, which does remove a feature.
• BatchNorm already injects batch-dependent noise, and combining the two often hurts because the variance BatchNorm calibrates during training differs from the variance at inference.

ResNet uses no dropout at all in its convolutional trunk; VGG uses 0.5 but only in its dense head.

Reading the Curves

Shape and Parameter Trace

MACs are per image. Multiply by 2 for FLOPs: roughly 58 MFLOPs per forward pass.

Activation Memory — What Actually Fills the GPU

Stored activations per image (fp32):
  conv1 out   32·32·32 =  32,768
  conv2 out   32·32·32 =  32,768
  pool1        32·16·16 =   8,192
  conv3 out    64·16·16 =  16,384
  conv4 out    64·16·16 =  16,384
  pool2         64·8·8  =   4,096
  conv5 out    128·8·8  =   8,192
  pool3        128·4·4  =   2,048
                        ─────────
                          120,832 floats
                        ≈ 0.47 MB per image

batch 128  →  ≈ 60 MB of activations
weights    →  141K × 4 B = 0.56 MB

Activations exceed weights by more than 100×. Backpropagation must retain every one of them for the backward pass, which is why memory scales with batch size and why gradient checkpointing (recompute instead of store) is the standard remedy when a model will not fit.

The Same Model in PyTorch

def block(cin, cout):
    return nn.Sequential(
        nn.Conv2d(cin, cout, 3, padding=1, bias=False),
        nn.BatchNorm2d(cout),
        nn.ReLU(inplace=True),
    )

model = nn.Sequential(
    block(3, 32),  block(32, 32),  nn.MaxPool2d(2),
    block(32, 64), block(64, 64),  nn.MaxPool2d(2),
    block(64, 128),
    nn.AdaptiveAvgPool2d(1), nn.Flatten(),
    nn.Linear(128, 10),
)

AdaptiveAvgPool2d(1) is global average pooling and works for any input resolution. This model reaches roughly 92 to 93 percent on CIFAR-10 with standard augmentation and a cosine schedule.

1. Gradient With Respect to the Weights

W[m,n] was used at every output position, so by the multivariable chain rule its gradient sums over all of them:

That expression is itself a cross-correlation: the input correlated with the output gradient. This summation is the mathematical face of weight sharing. A single 3×3 kernel applied to a 56×56 map accumulates 3,136 gradient contributions per image per channel pair, which is exactly why conv weights are so well determined by comparatively little data.

2. Gradient With Respect to the Bias

One scalar per output channel: sum the output gradient over batch and both spatial dimensions.

3. Gradient With Respect to the Input

Element X[p,q] contributed to every output whose window covered it. Collecting those terms:

The index signs are flipped relative to the forward pass. Concretely, this is a full convolution of dL/dY with W rotated by 180°, which is precisely the transposed convolution operation. Interior elements receive K² contributions; border elements receive fewer, because fewer windows covered them.

Backward Through the Other Layers

Max pooling requires storing the argmax indices from the forward pass, which is why it carries a memory cost despite having no parameters. The BatchNorm backward pass is the subtle one: treating μ and σ² as constants gives a wrong gradient, and it is a classic error in hand-rolled implementations.

The Fix — Learn the Residual Instead

Rather than asking the block to produce the desired mapping H(x), ask it to produce F(x) = H(x) − x and add x back. Now the identity is the default: drive the weights to zero and the block passes its input through unchanged. A block that is not useful can cheaply become a no-op, so adding depth can no longer hurt.

The gradient view is equally direct. Differentiating the skip path:

The +1 is an unattenuated gradient highway straight to earlier layers. Vanishing gradients require the product of many small factors; the additive identity term prevents that product from ever collapsing.

Basic Block and Bottleneck Block

The Bottleneck Arithmetic

Plain 3×3 at 256 → 256:
  3·3·256·256           = 589,824

Bottleneck 256→64→64→256:
  1·1·256·64            =  16,384
  3·3·64·64             =  36,864
  1·1·64·256            =  16,384
                          ─────────
                          69,632   (8.5× fewer)

The 1×1 layers compress to a cheap subspace, do the expensive spatial work there, then expand back. This is what lets ResNet-50 run 50 layers on 4.1 GFLOPs while VGG-16 needs 15.5 GFLOPs for 16.

The Full Family

Depthwise convolution is grouped convolution with g = C_in. Standard convolution is grouped convolution with g = 1. It is one operator with a single dial.

Grouped Convolution

Split the input channels into g groups and convolve each group only with its own filters. Parameters and FLOPs divide by g exactly. AlexNet used 2 groups purely because the model did not fit on one GTX 580 with 3 GB; the accidental discovery was that it cost almost no accuracy.

ResNeXt made it deliberate, replacing the bottleneck 3×3 with a grouped 3×3 at cardinality 32 and reporting better ImageNet accuracy than ResNet at matched FLOPs and parameters. The cost: groups never exchange information, so ShuffleNet interleaves a channel shuffle between grouped layers to restore cross-group flow.

nn.Conv2d(256, 256, 3, padding=1, groups=32)
# 589,824 → 18,432 weights

nn.Conv2d(64, 64, 3, padding=1, groups=64)
# depthwise: 36,864 → 576 weights

The Shared Structure

Almost every vision task reuses a classification backbone and changes only the head and the loss. This is why ImageNet pretraining transferred so broadly and why backbone improvements propagate to every downstream task at once.

Object Detection

The recurring problem is class imbalance: a dense detector evaluates roughly 100,000 candidate locations of which a handful contain objects. Focal loss down-weights easy negatives so the rare positives are not drowned out.

U-Net — Why the Skips Are Load-Bearing

An encoder-decoder alone loses the fine detail during downsampling and cannot invent it back. U-Net concatenates each encoder feature map into the matching decoder stage, so the decoder has both the semantics from the deep path and the boundary precision from the shallow path.

encoder: 572 → 284 → 140 → 68 → 32
                │      │      │     │
                └──────┴──────┴─────┘  concat
decoder:  32 →  68 → 140 → 284 → 388

It was designed for biomedical images with only 30 annotated training examples, and the skip connections plus aggressive elastic augmentation are what made that possible. The pattern generalized far beyond its original domain, and diffusion models use the same backbone today.

Model and Data

import torch, torch.nn as nn
from torchvision import datasets, transforms

MEAN, STD = (0.4914, 0.4822, 0.4465), (0.247, 0.243, 0.261)

train_tf = transforms.Compose([
    transforms.RandomCrop(32, padding=4),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize(MEAN, STD),
])
# Validation is deterministic. No random transforms.
val_tf = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize(MEAN, STD),
])

train_ds = datasets.CIFAR10("./data", train=True,
                            download=True, transform=train_tf)
val_ds   = datasets.CIFAR10("./data", train=False,
                            transform=val_tf)

train_dl = torch.utils.data.DataLoader(
    train_ds, batch_size=128, shuffle=True,
    num_workers=8, pin_memory=True, drop_last=True)
val_dl = torch.utils.data.DataLoader(
    val_ds, batch_size=256, shuffle=False, num_workers=4)

Normalization constants are the per-channel mean and standard deviation of the training set. Using ImageNet constants on CIFAR is a small but free loss of accuracy, and it is a very common copy-paste error.

The Loop

device = "cuda"
model = build_model().to(device, memory_format=torch.channels_last)

opt = torch.optim.SGD(
    model.parameters(), lr=0.1, momentum=0.9,
    weight_decay=5e-4, nesterov=True)
sched = torch.optim.lr_scheduler.OneCycleLR(
    opt, max_lr=0.1, epochs=EPOCHS, steps_per_epoch=len(train_dl))
scaler = torch.amp.GradScaler()
criterion = nn.CrossEntropyLoss(label_smoothing=0.1)

for epoch in range(EPOCHS):
    model.train()                       # BN uses batch stats
    for x, y in train_dl:
        x = x.to(device, non_blocking=True,
                 memory_format=torch.channels_last)
        y = y.to(device, non_blocking=True)

        opt.zero_grad(set_to_none=True)
        with torch.autocast("cuda", dtype=torch.bfloat16):
            loss = criterion(model(x), y)
        scaler.scale(loss).backward()
        scaler.step(opt)
        scaler.update()
        sched.step()                    # per step, not per epoch

    model.eval()                        # BN uses running stats
    correct = total = 0
    with torch.inference_mode():
        for x, y in val_dl:
            x, y = x.to(device), y.to(device)
            correct += (model(x).argmax(1) == y).sum().item()
            total   += y.numel()
    print(f"epoch {epoch}  val acc {correct/total:.4f}")

Choosing a Strategy

The governing risk with a small dataset is that full fine-tuning at a normal learning rate destroys the pretrained features in the first few hundred steps, before the randomly initialized head has produced a useful gradient signal. Warm up the head first.

Feature Extraction — Freeze the Trunk

from torchvision.models import resnet50, ResNet50_Weights

model = resnet50(weights=ResNet50_Weights.IMAGENET1K_V2)

for p in model.parameters():
    p.requires_grad = False

# New head. requires_grad=True by default.
model.fc = nn.Linear(model.fc.in_features, NUM_CLASSES)

opt = torch.optim.AdamW(model.fc.parameters(), lr=1e-3)

Fast, needs little memory, and hard to get wrong. Also call model.eval() on the frozen trunk during training if you do not want its BatchNorm running statistics to drift toward your small dataset.

Fine-Tuning With Discriminative Learning Rates

# Stage 1: head only, a few epochs, so the head
# stops producing garbage gradients.

# Stage 2: unfreeze, and give earlier layers a
# much smaller learning rate than later ones.
for p in model.parameters():
    p.requires_grad = True

opt = torch.optim.AdamW([
    {"params": model.layer1.parameters(), "lr": 1e-5},
    {"params": model.layer2.parameters(), "lr": 3e-5},
    {"params": model.layer3.parameters(), "lr": 1e-4},
    {"params": model.layer4.parameters(), "lr": 3e-4},
    {"params": model.fc.parameters(),     "lr": 1e-3},
], weight_decay=1e-4)

Rationale: early layers hold the most general features and need the least change; the head is random and needs the most. A geometric ladder spanning roughly two orders of magnitude is the standard recipe.

When Transfer Learning Does Not Help

• Very different input statistics. Medical scans, satellite multispectral bands, audio spectrograms. The gain shrinks, though it is rarely negative.
• Non-3-channel input. Adapt the stem: average the RGB filters across the channel dimension for grayscale, or replicate them for more bands.
• Abundant in-domain data. Past roughly a million labelled examples, pretraining mainly buys faster convergence rather than final accuracy.
• Licensing. Pretrained weights carry licences. Check them before shipping.

Winograd and FFT

Winograd trades multiplications for additions and for transform overhead, which is a good trade because multipliers are the scarce resource. It is numerically less stable and is normally restricted to stride 1 with small kernels — that is, exactly the 3×3 layers that dominate modern networks. FFT convolution loses to Winograd at K = 3 because the transform overhead is not amortized.

Where the Time and Memory Actually Go

Before optimizing the model, profile. A surprising share of "slow training" turns out to be a starved input pipeline: too few dataloader workers, JPEG decoding on the main process, or images being read one at a time from network storage. Watch GPU utilization; if it is not near 100 percent, the model is not the problem.

Memory Budget for Training

Total ≈ weights
      + gradients        (= weights)
      + optimizer state  (2× weights for Adam)
      + activations      (batch × per-image)
      + workspace        (cuDNN scratch)

ResNet-50, batch 256, fp32:
  weights          25.6M × 4 B  =  0.10 GB
  gradients                     =  0.10 GB
  Adam state                    =  0.20 GB
  activations      ≈ 25 MB/img  =  6.4 GB   ← dominates

Activations dominate by an order of magnitude, and they scale linearly with batch size. The levers, in order: mixed precision (halves activations), gradient checkpointing (recompute rather than store, roughly 30 percent slower for a large memory reduction), then gradient accumulation to simulate a large batch with a small one.

The Convergence

The two families borrowed from each other and largely met in the middle. Swin Transformer reintroduced locality through windowed attention and a hierarchical pyramid. ConvNeXt (2022) took a plain ResNet and applied the transformer-era training recipe and design choices — AdamW, heavy augmentation, LayerNorm, GELU, depthwise 7×7 kernels, fewer activations — and matched Swin at equal FLOPs.

The honest conclusion from ConvNeXt is that a meaningful share of the reported transformer advantage was the training recipe rather than the attention mechanism.

Practical Selection

Known CNN Failure Modes

• Texture bias. Geirhos et al. (2019) showed ImageNet CNNs classify predominantly by texture rather than shape, contrary to human perception. Training on stylized images shifts the bias and improves robustness.
• Shift instability. A one-pixel translation can change the prediction, because strided downsampling aliases.
• Adversarial fragility. Perturbations far below the threshold of human perception flip predictions with high confidence.
• Spurious correlation. The model will happily learn that grass predicts "cow" if the dataset says so, and then fail on a cow at the beach.

None of these are transformer-free problems. They are consequences of training discriminatively on finite, biased data, and they persist across architectures.

Primary Sources

• LeCun et al., Gradient-Based Learning Applied to Document Recognition, 1998 — LeNet-5.
• Krizhevsky et al., ImageNet Classification with Deep CNNs, 2012 — AlexNet.
• Simonyan and Zisserman, Very Deep Convolutional Networks, 2014 — VGG.
• Szegedy et al., Going Deeper with Convolutions, 2014 — Inception.
• Ioffe and Szegedy, Batch Normalization, 2015.
• He et al., Deep Residual Learning for Image Recognition, 2015 — ResNet.
• Luo et al., Understanding the Effective Receptive Field, 2016.
• Liu et al., A ConvNet for the 2020s, 2022 — ConvNeXt.