FCN (Fully Convolutional Network) in Computer Vision - Deep Dive

Overview - FCN (Fully Convolutional Network)

What is it?

A Fully Convolutional Network (FCN) is a type of neural network designed to process images and produce outputs that keep spatial information, like segmenting parts of an image. Unlike traditional networks that use fixed-size layers, FCNs use only convolutional layers, which can handle images of any size and output a map showing what each pixel belongs to. This makes FCNs very useful for tasks where understanding the location of objects in an image is important.

Why it matters

Before FCNs, image tasks like segmentation were hard because networks lost spatial details when using fixed-size layers. FCNs solve this by keeping spatial information, allowing computers to understand images more like humans do—knowing not just what is in the image but exactly where. Without FCNs, many applications like self-driving cars, medical image analysis, and photo editing would be less accurate and slower.

Where it fits

Learners should first understand basic convolutional neural networks (CNNs) and image processing concepts. After FCNs, they can explore advanced segmentation models like U-Net, Mask R-CNN, and learn about applications in object detection and scene understanding.

Mental Model

Core Idea

An FCN replaces fixed-size layers with only convolutional layers to produce spatially meaningful outputs for every pixel in an image.

Think of it like...

Imagine painting a wall with a stencil that moves over every part of the wall, coloring each spot based on what it sees, instead of painting the whole wall at once and losing details.

Input Image
   │
[Convolution Layers]
   │
[Feature Maps with spatial info]
   │
[Upsampling Layers]
   │
Output: Pixel-wise prediction map

Each step keeps the image shape or restores it, so output matches input size.

Build-Up - 7 Steps

1

Foundation

Basics of Convolutional Neural Networks

Concept: Understanding how CNNs extract features from images using filters.

CNNs use small filters that slide over an image to detect edges, colors, and shapes. Each filter creates a feature map showing where certain patterns appear. Pooling layers reduce size but lose some detail. CNNs usually end with fully connected layers that output a single label for the whole image.

Result

CNNs can classify images but lose exact location details of objects inside.

Knowing how CNNs work helps see why they struggle with tasks needing pixel-level understanding.

2

Foundation

Limitations of Fully Connected Layers

Concept: Why fixed-size fully connected layers limit spatial output.

Fully connected layers flatten the image features into a single vector, losing the 2D layout. This means the network can only say what is in the image, not where. Also, input images must be a fixed size to match the layer dimensions.

Result

Networks with fully connected layers cannot produce outputs that map back to the original image size.

Understanding this limitation motivates the need for networks that keep spatial info.

3

Intermediate

Replacing Fully Connected Layers with Convolutions

🤔Before reading on: Do you think a convolutional layer can replace a fully connected layer without losing spatial info? Commit to yes or no.

Concept: Fully connected layers can be seen as convolutions with filters covering the entire input, so replacing them with smaller convolutions keeps spatial info.

By turning fully connected layers into convolutional layers with 1x1 filters, the network can process inputs of any size and keep spatial dimensions. This means the output is a feature map instead of a single vector, preserving location information.

Result

The network outputs a spatial map showing predictions for different parts of the image.

Knowing fully connected layers are special convolutions unlocks the design of FCNs.

4

Intermediate

Upsampling to Restore Image Size

🤔Before reading on: Does upsampling add new information or just increase size? Commit to your answer.

Concept: After downsampling through pooling, FCNs use upsampling layers to increase feature map size back to input dimensions.

Upsampling methods like transposed convolution or interpolation enlarge the smaller feature maps to the original image size. This allows the network to output a prediction for every pixel, matching the input resolution.

Result

The output is a pixel-wise prediction map aligned with the input image.

Understanding upsampling is key to producing detailed spatial outputs.

5

Intermediate

End-to-End Training for Pixel-wise Tasks

Concept: FCNs can be trained to predict labels for every pixel directly from input images.

By using loss functions like cross-entropy on each pixel, FCNs learn to classify every pixel into categories, such as road, car, or sky. This end-to-end training means the network learns both feature extraction and pixel labeling simultaneously.

Result

The model produces accurate segmentation maps after training.

Training FCNs end-to-end simplifies the pipeline and improves accuracy.

6

Advanced

Skip Connections for Detail Preservation

🤔Before reading on: Do you think deeper layers alone can recover fine image details? Commit to yes or no.

Concept: Skip connections combine deep, coarse features with shallow, fine features to improve output detail.

FCNs add connections from early layers directly to later upsampling layers. This helps the network keep fine details lost during downsampling, improving segmentation edges and small object detection.

Result

Outputs have sharper boundaries and better spatial accuracy.

Knowing skip connections balance detail and context is crucial for high-quality segmentation.

7

Expert

Challenges and Surprises in FCN Training

🤔Before reading on: Does training FCNs require special tricks compared to regular CNNs? Commit to yes or no.

Concept: Training FCNs can be tricky due to class imbalance, spatial resolution, and upsampling artifacts.

FCNs often face issues like many background pixels dominating loss, causing poor learning for small classes. Also, naive upsampling can create checkerboard artifacts. Experts use weighted losses, multi-scale inputs, and careful upsampling design to overcome these.

Result

Proper training techniques lead to robust, artifact-free segmentation models.

Understanding these challenges prevents common pitfalls and improves real-world FCN performance.

Under the Hood

FCNs work by applying convolutional filters across the entire image, producing feature maps that keep spatial layout. Instead of flattening features, they use convolutional layers to maintain 2D structure. Downsampling reduces size but captures context, while upsampling restores size for pixel-wise output. Skip connections merge features from different depths to combine detail and context. During training, pixel-wise loss functions guide the network to assign correct labels to each pixel.

Why designed this way?

Traditional CNNs were designed for classification, losing spatial info in fully connected layers. FCNs were created to solve segmentation by removing these layers and using only convolutions, allowing flexible input sizes and spatial outputs. This design balances capturing global context and preserving local details, which was not possible with older architectures.

Input Image
   │
┌───────────────┐
│ Convolutional │
│   Layers      │
└──────┬────────┘
       │
┌──────▼───────┐
│ Downsampling │
│ (Pooling)    │
└──────┬───────┘
       │
┌──────▼─────────────┐
│ Fully Convolutional │
│     Layers          │
└──────┬──────────────┘
       │
┌──────▼─────────────┐
│ Upsampling Layers   │
│ (Transposed Conv)   │
└──────┬──────────────┘
       │
┌──────▼─────────────┐
│ Pixel-wise Output   │
└────────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does an FCN always require fixed-size input images? Commit to yes or no.

Common Belief: FCNs need fixed-size images because neural networks usually do.

Reality: FCNs can handle variable-sized images because they use only convolutional layers without fixed-size fully connected layers.

Why it matters: Believing fixed size is required limits the use of FCNs in real applications where image sizes vary.

Quick: Does upsampling add new image details? Commit to yes or no.

Common Belief: Upsampling creates new image details lost during downsampling.

Reality: Upsampling only increases resolution; it cannot create new true details but can combine features to approximate them.

Why it matters: Expecting upsampling to restore lost details leads to overconfidence in output quality and poor model design.

Quick: Are skip connections optional and only for speed? Commit to yes or no.

Common Belief: Skip connections are just shortcuts to speed up training.

Reality: Skip connections are crucial for preserving fine spatial details and improving segmentation accuracy.

Why it matters: Ignoring skip connections results in blurry outputs and poor boundary detection.

Quick: Does training an FCN use the same loss as image classification? Commit to yes or no.

Common Belief: FCNs use the same loss functions as classification tasks.

Reality: FCNs use pixel-wise loss functions that evaluate each pixel's prediction separately, often with class balancing.

Why it matters: Using classification loss causes poor segmentation performance and ignores spatial structure.

Expert Zone

1

FCNs often require careful balancing of receptive field size to capture context without losing local detail.

2

The choice of upsampling method (transposed convolution vs interpolation) affects artifact presence and model smoothness.

3

Class imbalance in segmentation datasets demands weighted or focal loss to prevent bias toward dominant classes.

When NOT to use

FCNs are less effective for tasks needing instance-level separation or very fine object boundaries; in such cases, models like Mask R-CNN or attention-based networks are better choices.

Production Patterns

In production, FCNs are often combined with post-processing steps like Conditional Random Fields (CRFs) to refine edges, and deployed with model quantization for faster inference on edge devices.

Connections

U-Net

Builds on FCN by adding symmetric encoder-decoder structure with skip connections.

Understanding FCNs helps grasp how U-Net improves segmentation by better combining features at multiple scales.

Autoencoders

Shares the encoder-decoder architecture pattern with FCNs for reconstructing inputs.

Knowing FCNs clarifies how autoencoders compress and restore spatial information in images.

Human Visual Cortex

Biological inspiration: hierarchical processing and spatial feature extraction.

Recognizing FCNs mimic how the brain processes visual scenes deepens appreciation of their design and limitations.

Common Pitfalls

#1 Using fully connected layers at the end of the network, losing spatial output.

Wrong approach:

model.add(Dense(1000)) # Fully connected layer after convolutions

Correct approach:

model.add(Conv2D(filters=1000, kernel_size=1)) # 1x1 convolution replacing dense layer

Root cause:

Misunderstanding that fully connected layers fix output size and discard spatial info.

#2 Naively upsampling with large strides causing checkerboard artifacts.

Wrong approach:

model.add(Conv2DTranspose(filters=64, kernel_size=4, strides=2)) # without careful design

Correct approach:

Use carefully designed kernel sizes and strides or interpolation followed by convolution to reduce artifacts.

Root cause:

Ignoring how transposed convolution parameters affect output smoothness.

#3 Training with unbalanced pixel classes leading to poor minority class detection.

Wrong approach:

loss = tf.keras.losses.SparseCategoricalCrossentropy() # no class weighting

Correct approach:

loss = weighted_cross_entropy(minority_class_weight) # apply class weights in loss

Root cause:

Not accounting for class imbalance in pixel-wise segmentation tasks.

Key Takeaways

Fully Convolutional Networks replace fixed-size layers with convolutional layers to keep spatial information for pixel-wise predictions.

Upsampling restores the reduced spatial size after pooling but does not create new image details by itself.

Skip connections are essential to combine deep semantic features with shallow spatial details for accurate segmentation.

Training FCNs requires pixel-wise loss functions and handling class imbalance to achieve good performance.

FCNs form the foundation for many advanced image segmentation models and are inspired by how biological vision processes scenes.