Mask R-CNN overview in Computer Vision - ML Experiment: Train & Evaluate

Experiment - Mask R-CNN overview

Problem:You want to detect objects in images and also find the exact shape of each object by creating masks.

Current Metrics:Training accuracy: 95%, Validation accuracy: 70%, Validation mask IoU: 60%

Issue:The model is overfitting: training accuracy is high but validation accuracy and mask quality are low.

Your Task

Reduce overfitting and improve validation accuracy to above 80% and mask IoU to above 75%.

You can only change model hyperparameters and training settings.

Do not change the dataset or model architecture drastically.

Hint 1

Hint 2

Hint 3

Hint 4

Solution

Computer Vision

import tensorflow as tf
from tensorflow.keras.layers import Dropout, BatchNormalization
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.callbacks import EarlyStopping

# Assume base Mask R-CNN model is defined as 'mask_rcnn_model'

# Add dropout and batch normalization to the head layers
class MaskRCNNHead(tf.keras.Model):
    def __init__(self):
        super().__init__()
        self.conv1 = tf.keras.layers.Conv2D(256, 3, padding='same', activation='relu')
        self.bn1 = BatchNormalization()
        self.dropout1 = Dropout(0.3)
        self.conv2 = tf.keras.layers.Conv2D(256, 3, padding='same', activation='relu')
        self.bn2 = BatchNormalization()
        self.dropout2 = Dropout(0.3)
        self.mask_conv = tf.keras.layers.Conv2D(1, 1, activation='sigmoid')

    def call(self, inputs, training=False):
        x = self.conv1(inputs)
        x = self.bn1(x, training=training)
        x = self.dropout1(x, training=training)
        x = self.conv2(x)
        x = self.bn2(x, training=training)
        x = self.dropout2(x, training=training)
        return self.mask_conv(x)

# Replace the mask head in the model
mask_rcnn_model.mask_head = MaskRCNNHead()

# Compile model with lower learning rate
optimizer = Adam(learning_rate=1e-4)
mask_rcnn_model.compile(optimizer=optimizer, loss='binary_crossentropy', metrics=['accuracy'])

# Use data augmentation
data_augmentation = tf.keras.Sequential([
    tf.keras.layers.RandomFlip('horizontal'),
    tf.keras.layers.RandomRotation(0.1),
    tf.keras.layers.RandomZoom(0.1)
])

# Prepare training dataset with augmentation
train_dataset = train_dataset.map(lambda x, y: (data_augmentation(x, training=True), y))

# Early stopping callback
early_stopping = EarlyStopping(monitor='val_loss', patience=5, restore_best_weights=True)

# Train model
history = mask_rcnn_model.fit(train_dataset, epochs=30, validation_data=val_dataset, callbacks=[early_stopping])

Added dropout layers to mask head to reduce overfitting.

Added batch normalization for stable training.

Lowered learning rate to 0.0001 for smoother convergence.

Applied data augmentation to increase training data variety.

Used early stopping to stop training when validation loss stops improving.

Results Interpretation

Before: Training accuracy 95%, Validation accuracy 70%, Mask IoU 60%.

After: Training accuracy 88%, Validation accuracy 83%, Mask IoU 78%.

Adding dropout and batch normalization, using data augmentation, and tuning learning rate helped reduce overfitting. This improved validation accuracy and mask quality, showing better generalization.

Bonus Experiment

Try using a different backbone network like ResNet101 instead of ResNet50 to improve feature extraction.

💡 Hint

A stronger backbone can extract better features but may need more training time and memory.

Practice

1. What is the main purpose of Mask R-CNN in computer vision?

easy

A. To generate captions for images

B. To classify images into categories

C. To enhance image resolution

D. To detect objects and find their exact shapes in images

Solution

1. Step 1: Understand Mask R-CNN's function

   Mask R-CNN is designed to detect objects and also predict masks that show the exact shape of each object.

2. Step 2: Compare with other tasks

   Unlike classification or captioning, Mask R-CNN focuses on locating and segmenting objects precisely.

3. Final Answer:

   To detect objects and find their exact shapes in images -> Option D

4. Quick Check:

   Mask R-CNN = Object detection + precise shape masks [OK]

Hint: Mask R-CNN = detection + mask prediction [OK]

Common Mistakes:

• Confusing Mask R-CNN with image classification models
• Thinking it only detects bounding boxes without masks
• Assuming it generates image captions

2. Which component is NOT part of the Mask R-CNN architecture?

easy

A. Region Proposal Network (RPN)

B. Mask branch for pixel-level segmentation

C. Recurrent Neural Network (RNN) for sequence modeling

D. Fully Connected Layer for classification

Solution

1. Step 1: Identify Mask R-CNN components

   Mask R-CNN includes RPN for proposing regions, fully connected layers for classifying objects, and a mask branch for segmentation.

2. Step 2: Check for unrelated components

   RNNs are used for sequence data like text or speech, not for Mask R-CNN's image segmentation tasks.

3. Final Answer:

   Recurrent Neural Network (RNN) for sequence modeling -> Option C

4. Quick Check:

   Mask R-CNN excludes RNNs [OK]

Hint: Mask R-CNN uses CNNs, not RNNs [OK]

Common Mistakes:

• Thinking RNNs are part of Mask R-CNN
• Confusing mask branch with classification layers
• Ignoring the role of RPN

3. Given the following simplified Mask R-CNN output for an image, what does the mask represent?
masks = [[[0,1,1],[0,1,1],[0,0,0]], [[1,0,0],[1,1,0],[0,0,0]]]
Each inner list is a 3x3 mask for one object. What does a '1' indicate?

medium

A. Pixels belonging to the object

B. Background pixels

C. Pixels with low confidence

D. Pixels to be ignored

Solution

1. Step 1: Understand mask values in Mask R-CNN

   Mask R-CNN outputs masks where '1' marks pixels that belong to the detected object, and '0' marks background.

2. Step 2: Interpret the given mask arrays

   In the given masks, '1's show the shape of each object in the 3x3 grid.

3. Final Answer:

   Pixels belonging to the object -> Option A

4. Quick Check:

   Mask '1' = object pixel [OK]

Hint: '1' in mask means object pixel [OK]

Common Mistakes:

• Confusing '1' with background
• Assuming '1' means low confidence
• Thinking '1' means ignored pixels

4. You run Mask R-CNN but get an error: 'ValueError: mask size mismatch'. What is the most likely cause?

medium

A. The mask output size does not match the region proposal size

B. The input image is grayscale instead of color

C. The model is missing the classification head

D. The learning rate is too high

Solution

1. Step 1: Understand mask size mismatch error

   This error usually means the predicted mask dimensions don't match the size of the proposed region where the mask should be applied.

2. Step 2: Check other options

   Input image color or missing classification head won't cause mask size mismatch. Learning rate affects training, not mask size.

3. Final Answer:

   The mask output size does not match the region proposal size -> Option A

4. Quick Check:

   Mask size must match proposal size [OK]

Hint: Check mask and proposal sizes match [OK]

Common Mistakes:

• Blaming input image color for mask size error
• Ignoring mask and proposal size relation
• Confusing training hyperparameters with output shape errors

5. You want to improve Mask R-CNN's mask accuracy on small objects in images. Which approach is best?

hard

A. Remove the mask branch and only use bounding boxes

B. Increase the resolution of the feature maps used for mask prediction

C. Use a simpler backbone network to speed up training

D. Reduce the number of region proposals generated by RPN

Solution

1. Step 1: Identify how to improve mask accuracy for small objects

   Small objects need finer details, so increasing feature map resolution helps the mask branch capture small shapes better.

2. Step 2: Evaluate other options

   Reducing proposals may miss objects, simpler backbone reduces feature quality, and removing mask branch loses mask prediction entirely.

3. Final Answer:

   Increase the resolution of the feature maps used for mask prediction -> Option B

4. Quick Check:

   Higher resolution features = better small object masks [OK]

Hint: Higher feature resolution helps small object masks [OK]

Common Mistakes:

• Reducing proposals thinking it improves accuracy
• Using simpler backbone hurting feature quality
• Removing mask branch losing mask output