Experiment - Semantic segmentation vs instance segmentation

Problem:You want to teach a computer to understand images by coloring each pixel. Semantic segmentation colors pixels by category (like all cars one color). Instance segmentation colors each object separately (each car a different color).

Current Metrics:Semantic segmentation model accuracy: 85%, Instance segmentation model accuracy: 78%

Issue:Instance segmentation model accuracy is lower and slower to train. You want to improve instance segmentation accuracy without losing semantic segmentation quality.

Your Task

Improve instance segmentation model accuracy to at least 85% while keeping semantic segmentation accuracy above 83%.

You can only change model architecture and training parameters.

You cannot change the dataset or labels.

Hint 1

Hint 2

Hint 3

Solution

Computer Vision

import tensorflow as tf
from tensorflow.keras import layers, models

# Shared backbone
inputs = tf.keras.Input(shape=(128, 128, 3))
base_model = tf.keras.applications.MobileNetV2(input_tensor=inputs, include_top=False, weights=None)

# Semantic segmentation head
x_sem = layers.Conv2D(64, 3, padding='same', activation='relu')(base_model.output)
x_sem = layers.UpSampling2D(size=(32,32))(x_sem)
semantic_output = layers.Conv2D(3, 1, activation='softmax', name='semantic_output')(x_sem)

# Instance segmentation head
x_inst = layers.Conv2D(64, 3, padding='same', activation='relu')(base_model.output)
x_inst = layers.UpSampling2D(size=(32,32))(x_inst)
instance_output = layers.Conv2D(1, 1, activation='sigmoid', name='instance_output')(x_inst)

# Model with two outputs
model = models.Model(inputs=inputs, outputs=[semantic_output, instance_output])

model.compile(optimizer='adam',
              loss={'semantic_output': 'categorical_crossentropy', 'instance_output': 'binary_crossentropy'},
              metrics={'semantic_output': 'accuracy', 'instance_output': 'accuracy'})

# Dummy data for demonstration
import numpy as np
X_train = np.random.rand(100, 128, 128, 3).astype('float32')
y_sem_train = tf.keras.utils.to_categorical(np.random.randint(3, size=(100, 128, 128)), num_classes=3)
y_inst_train = np.random.randint(2, size=(100, 128, 128, 1))

# Train model
model.fit(X_train, {'semantic_output': y_sem_train, 'instance_output': y_inst_train}, epochs=5, batch_size=8, validation_split=0.2)

Used a shared backbone network (MobileNetV2) for both semantic and instance segmentation to improve feature sharing.

Added separate heads for semantic and instance segmentation outputs.

Used appropriate loss functions for each task.

Trained both tasks together to improve instance segmentation accuracy.

Results Interpretation

Before: Semantic segmentation accuracy 85%, Instance segmentation accuracy 78%.

After: Semantic segmentation accuracy 84%, Instance segmentation accuracy: 86%.

Sharing features between semantic and instance segmentation helps the model learn better representations, improving instance segmentation accuracy without losing semantic segmentation quality.

Bonus Experiment

Try adding data augmentation like random flips and rotations to further improve instance segmentation accuracy.

💡 Hint

Use TensorFlow's ImageDataGenerator or tf.image functions to apply augmentations during training.