CNN architecture for image classification in PyTorch - Deep Dive

Overview - CNN architecture for image classification

What is it?

A CNN, or Convolutional Neural Network, is a special type of computer program designed to look at pictures and learn what is in them. It uses layers that scan small parts of the image to find patterns like edges or colors. These patterns help the network understand the whole picture and decide what it shows, like a cat or a dog. CNNs are very good at recognizing images because they focus on local details and combine them step by step.

Why it matters

Without CNNs, computers would struggle to understand images clearly and quickly. Before CNNs, image recognition was slow and inaccurate, making tasks like photo tagging, medical image analysis, or self-driving cars much harder. CNNs let machines see and understand pictures almost like humans do, enabling many technologies we use daily, such as face recognition on phones or automatic photo sorting.

Where it fits

Before learning CNNs, you should know basic neural networks and how computers handle numbers and simple math. After CNNs, you can explore more advanced topics like transfer learning, object detection, or segmentation, which build on CNNs to solve complex image tasks.

Mental Model

Core Idea

A CNN learns to recognize images by scanning small parts repeatedly to find simple patterns, then combining these patterns to understand the whole picture.

Think of it like...

Imagine reading a book by looking at one word at a time, then one sentence, then one paragraph, gradually understanding the story. CNNs do the same with images, looking at small patches first and then the bigger picture.

Input Image
   │
   ▼
[Convolution Layer] -- scans small patches for features
   │
   ▼
[Activation Layer] -- adds non-linearity
   │
   ▼
[Pooling Layer] -- shrinks image to focus on important parts
   │
   ▼
(repeat convolution + activation + pooling layers)
   │
   ▼
[Flatten Layer] -- turns 2D features into 1D list
   │
   ▼
[Fully Connected Layer] -- decides what the image is
   │
   ▼
[Output] -- class probabilities (e.g., cat, dog, car)

Build-Up - 7 Steps

1

Foundation

Understanding Image Data as Numbers

Concept: Images are made of pixels, which are numbers representing colors or brightness.

Every image is a grid of tiny dots called pixels. Each pixel has a number showing how bright or what color it is. For example, a black-and-white image has pixels from 0 (black) to 255 (white). Color images have three numbers per pixel for red, green, and blue. Computers only understand numbers, so images become big tables of numbers.

Result

You can represent any picture as a set of numbers arranged in rows and columns.

Understanding that images are just numbers helps you see why math and neural networks can work with pictures.

2

Foundation

Basics of Neural Networks for Classification

Concept: Neural networks learn to classify data by adjusting connections between simple units called neurons.

A neural network has layers of neurons. Each neuron takes numbers, multiplies them by weights, adds them up, and passes the result through a function. By changing weights during training, the network learns to recognize patterns and classify inputs into categories.

Result

A simple network can learn to tell apart different types of data by adjusting its weights.

Knowing how neurons combine inputs to make decisions is key to understanding more complex CNN layers.

3

Intermediate

Convolution Layer: Scanning for Patterns

🤔Before reading on: do you think the convolution layer looks at the whole image at once or small parts? Commit to your answer.

Concept: The convolution layer scans small parts of the image to find simple features like edges or colors.

Instead of looking at the whole image at once, convolution layers use small filters (like tiny windows) that slide over the image. Each filter looks for a specific pattern, such as a vertical edge or a color patch. The output is a new image showing where these patterns appear.

Result

The network creates feature maps highlighting important local patterns in the image.

Understanding that convolution focuses on local patterns explains why CNNs are good at recognizing images regardless of where objects appear.

4

Intermediate

Pooling Layer: Simplifying Information

🤔Before reading on: does pooling increase or decrease the size of the image representation? Commit to your answer.

Concept: Pooling layers reduce the size of feature maps to keep important information and make computation easier.

Pooling looks at small areas of the feature map and picks one number to represent that area, usually the biggest (max pooling). This shrinks the image size but keeps the strongest signals, helping the network focus on important features and be faster.

Result

The image representation becomes smaller but still keeps key information.

Knowing pooling reduces data size while preserving important features helps understand how CNNs stay efficient and avoid overfitting.

5

Intermediate

Activation Functions: Adding Non-Linearity

🤔Before reading on: do you think activation functions make the network linear or non-linear? Commit to your answer.

Concept: Activation functions allow the network to learn complex patterns by adding non-linear transformations.

After convolution, the network applies an activation function like ReLU, which changes all negative numbers to zero but keeps positive numbers. This step helps the network learn more complex shapes and patterns beyond simple lines.

Result

The network can model complicated relationships in the image data.

Understanding activation functions is crucial because without them, the network would only learn simple, limited patterns.

6

Advanced

Fully Connected Layers for Decision Making

🤔Before reading on: do you think fully connected layers keep spatial information or flatten it? Commit to your answer.

Concept: Fully connected layers take all features and combine them to decide the image's class.

After several convolution and pooling layers, the feature maps are flattened into a long list of numbers. Fully connected layers treat this list like input features and learn to weigh them to predict the correct class, such as 'cat' or 'dog'.

Result

The network outputs probabilities for each class, enabling classification.

Knowing how fully connected layers summarize learned features into decisions explains the final step of image classification.

7

Expert

Building a CNN Model in PyTorch

🤔Before reading on: do you think the CNN model code should include convolution, activation, pooling, and fully connected layers? Commit to your answer.

Concept: Implementing a CNN in PyTorch involves stacking layers correctly and defining forward data flow.

Here is a simple CNN for image classification in PyTorch: import torch import torch.nn as nn import torch.nn.functional as F class SimpleCNN(nn.Module): def __init__(self): super().__init__() self.conv1 = nn.Conv2d(3, 16, kernel_size=3, padding=1) # input 3 channels (RGB), output 16 self.pool = nn.MaxPool2d(2, 2) # reduce size by half self.conv2 = nn.Conv2d(16, 32, kernel_size=3, padding=1) self.fc1 = nn.Linear(32 * 8 * 8, 128) # assuming input images 32x32 self.fc2 = nn.Linear(128, 10) # 10 classes def forward(self, x): x = self.pool(F.relu(self.conv1(x))) # conv1 + relu + pool x = self.pool(F.relu(self.conv2(x))) # conv2 + relu + pool x = x.view(-1, 32 * 8 * 8) # flatten x = F.relu(self.fc1(x)) x = self.fc2(x) # output logits return x This model takes a color image of size 32x32, applies two convolution layers with ReLU and pooling, then uses fully connected layers to classify into 10 categories.

Result

The model outputs raw scores (logits) for each class, which can be converted to probabilities for classification.

Seeing the full PyTorch code connects theory to practice and shows how CNN components work together in real code.

Under the Hood

CNNs work by sliding small filters over the input image to detect local features. Each filter multiplies its weights with the image patch and sums the result, creating a feature map. Activation functions add non-linearity so the network can learn complex patterns. Pooling layers reduce spatial size to focus on important features and reduce computation. Fully connected layers at the end combine all features to make a final decision. During training, the network adjusts filter weights using backpropagation to minimize classification errors.

Why designed this way?

CNNs were designed to mimic how the human visual cortex processes images, focusing on local receptive fields and hierarchical feature extraction. This design reduces the number of parameters compared to fully connected networks, making training feasible on large images. Alternatives like fully connected networks were too large and inefficient for images. The layered approach also allows CNNs to learn from simple edges to complex objects step by step.

Input Image
   │
   ▼
╔══════════════╗
║ Convolution  ║  -- filters scan image patches
╚══════╤═══════╝
       │
       ▼
╔══════════════╗
║ Activation   ║  -- adds non-linearity
╚══════╤═══════╝
       │
       ▼
╔══════════════╗
║ Pooling      ║  -- reduces size, keeps key info
╚══════╤═══════╝
       │
       ▼
(repeat layers)
       │
       ▼
╔══════════════╗
║ Flatten      ║  -- converts 2D to 1D
╚══════╤═══════╝
       │
       ▼
╔══════════════╗
║ Fully Connected ║ -- combines features to classify
╚══════╤═══════╝
       │
       ▼
    Output Classes

Myth Busters - 4 Common Misconceptions

Quick: Does a convolution layer look at the entire image at once or small parts? Commit to your answer.

Common Belief: Convolution layers analyze the whole image at once like a fully connected layer.

Reality: Convolution layers scan small local patches of the image using filters, not the entire image at once.

Why it matters: Thinking convolution sees the whole image leads to misunderstanding how CNNs detect local patterns and why they are efficient.

Quick: Does pooling add new information or just reduce size? Commit to your answer.

Common Belief: Pooling layers create new features by combining information in complex ways.

Reality: Pooling layers only reduce the size of feature maps by selecting representative values, they do not add new information.

Why it matters: Believing pooling adds information can cause confusion about how CNNs learn and why pooling is used mainly for efficiency.

Quick: Does the CNN output directly give class probabilities? Commit to your answer.

Common Belief: The CNN model outputs probabilities for each class directly after the last layer.

Reality: CNNs usually output raw scores (logits); probabilities are obtained by applying a softmax function afterward.

Why it matters: Misunderstanding output format can cause errors in interpreting model predictions and loss calculations.

Quick: Do CNNs require fixed-size input images? Commit to your answer.

Common Belief: CNNs can handle any image size without changes.

Reality: Most CNN architectures require fixed-size inputs because fully connected layers expect fixed-length vectors.

Why it matters: Ignoring input size requirements leads to errors or poor performance when feeding images of different sizes.

Expert Zone

1

CNN filters learn to detect features that are translation invariant, meaning they recognize patterns regardless of where they appear in the image.

2

Batch normalization layers, often added between convolution and activation, stabilize training and allow higher learning rates.

3

Deeper CNNs can suffer from vanishing gradients; skip connections (like in ResNet) help by allowing gradients to flow directly.

When NOT to use

CNNs are less effective for data without spatial structure, like tabular data or sequences where recurrent networks or transformers may be better. For very small datasets, CNNs can overfit; simpler models or transfer learning should be preferred.

Production Patterns

In real systems, CNNs are often pretrained on large datasets and fine-tuned for specific tasks. Architectures like ResNet, EfficientNet, or MobileNet are used depending on accuracy and speed needs. CNNs are combined with data augmentation and regularization to improve robustness.

Connections

Human Visual Cortex

CNNs are inspired by the hierarchical processing of visual information in the brain.

Understanding biological vision helps explain why CNNs use local receptive fields and layered feature extraction.

Signal Processing

Convolution in CNNs is mathematically the same as convolution in signal processing used for filtering signals.

Knowing signal processing concepts clarifies how filters detect edges and patterns in images.

Natural Language Processing (NLP)

CNNs are also used in NLP to detect local patterns in text sequences, similar to image patches.

Recognizing CNNs' role beyond images shows their power in finding local features in different data types.

Common Pitfalls

#1 Feeding images of varying sizes directly into a CNN with fixed fully connected layers.

Wrong approach:

model = SimpleCNN()
input = torch.randn(1, 3, 64, 64) # 64x64 instead of 32x32
output = model(input)

Correct approach:

Resize all input images to 32x32 before feeding into the model:
from torchvision import transforms
transform = transforms.Resize((32, 32))
input_resized = transform(input)
output = model(input_resized)

Root cause:

Fully connected layers expect fixed-size inputs; varying image sizes cause shape mismatches.

#2 Using convolution without activation functions between layers.

Wrong approach:

x = self.pool(self.conv1(x)) # missing activation like ReLU

Correct approach:

x = self.pool(F.relu(self.conv1(x))) # apply activation after convolution

Root cause:

Without activation, the network behaves like a linear model and cannot learn complex patterns.

#3 Interpreting raw model outputs as probabilities without applying softmax.

Wrong approach:

predicted_class = torch.argmax(model(input)) # directly on logits without softmax

Correct approach:

probabilities = F.softmax(model(input), dim=1)
predicted_class = torch.argmax(probabilities)

Root cause:

Logits are unnormalized scores; softmax converts them to probabilities.

Key Takeaways

CNNs process images by scanning small patches to find simple patterns, then combine these to understand the whole image.

Convolution layers detect local features, pooling layers reduce data size while keeping important information, and activation functions enable learning complex patterns.

Fully connected layers at the end use all learned features to classify the image into categories.

Implementing CNNs in PyTorch involves stacking convolution, activation, pooling, and fully connected layers with correct input and output shapes.

Understanding CNN internals and common pitfalls helps build efficient and accurate image classification models.