Why visual AI extends beyond classification in TensorFlow - Experiment to Prove It

Experiment - Why visual AI extends beyond classification

Problem:You have a visual AI model that classifies images into categories, but you want to explore tasks beyond simple classification, such as object detection or segmentation.

Current Metrics:Training accuracy: 95%, Validation accuracy: 93%, Model only outputs class labels.

Issue:The model only classifies images but does not provide information about object locations or shapes, limiting its usefulness in real-world applications.

Your Task

Extend the visual AI model to perform object detection by predicting bounding boxes along with class labels, improving the model's ability to locate objects in images.

Use TensorFlow and Keras only.

Keep the model architecture simple and runnable on a small dataset.

Do not use pre-trained models or external datasets.

Hint 1

Hint 2

Hint 3

Solution

TensorFlow

import tensorflow as tf
from tensorflow.keras import layers, models, losses, optimizers
import numpy as np

# Create a simple dataset with images, class labels, and bounding boxes
# For simplicity, images are 28x28 grayscale, classes: 0 or 1
# Bounding boxes: [x_min, y_min, x_max, y_max] normalized between 0 and 1

num_samples = 1000
image_size = 28
num_classes = 2

# Random images
X = np.random.rand(num_samples, image_size, image_size, 1).astype(np.float32)

# Random class labels (0 or 1)
y = np.random.randint(0, num_classes, size=(num_samples,))

# Random bounding boxes (x_min < x_max, y_min < y_max)
boxes = np.random.rand(num_samples, 4).astype(np.float32)
boxes[:, 2:] = boxes[:, :2] + boxes[:, 2:] * 0.5  # ensure x_max > x_min, y_max > y_min
boxes = np.clip(boxes, 0, 1)

# Convert class labels to one-hot
y_one_hot = tf.keras.utils.to_categorical(y, num_classes)

# Model definition
inputs = layers.Input(shape=(image_size, image_size, 1))
x = layers.Conv2D(16, 3, activation='relu')(inputs)
x = layers.MaxPooling2D()(x)
x = layers.Conv2D(32, 3, activation='relu')(x)
x = layers.MaxPooling2D()(x)
x = layers.Flatten()(x)
x = layers.Dense(64, activation='relu')(x)

# Two outputs: classification and bounding box regression
class_output = layers.Dense(num_classes, activation='softmax', name='class_output')(x)
box_output = layers.Dense(4, activation='sigmoid', name='box_output')(x)

model = models.Model(inputs=inputs, outputs=[class_output, box_output])

# Losses: categorical crossentropy for class, mean squared error for boxes
losses_dict = {
    'class_output': losses.CategoricalCrossentropy(),
    'box_output': losses.MeanSquaredError()
}

model.compile(optimizer=optimizers.Adam(learning_rate=0.001),
              loss=losses_dict,
              metrics={'class_output': 'accuracy'})

# Train the model
history = model.fit(X, {'class_output': y_one_hot, 'box_output': boxes},
                    epochs=10, batch_size=32, validation_split=0.2)

# Predict on a sample
sample_img = X[0:1]
pred_class, pred_box = model.predict(sample_img)

print(f"Predicted class probabilities: {pred_class}")
print(f"Predicted bounding box: {pred_box}")

Added a second output layer to predict bounding box coordinates.

Used a combined loss function: classification loss plus bounding box regression loss.

Prepared dataset with bounding box labels alongside class labels.

Trained the model to predict both class and bounding box simultaneously.

Results Interpretation

Before: Model only outputs class labels with 95% training accuracy and 93% validation accuracy.

After: Model outputs both class labels and bounding boxes with classification accuracy around 90% and good bounding box prediction loss reduction.

Visual AI tasks go beyond classification by including object localization. Adding bounding box prediction teaches the model to find where objects are, not just what they are.

Bonus Experiment

Try extending the model to perform image segmentation by predicting a mask for each object instead of bounding boxes.

💡 Hint

Use a convolutional decoder network to output a mask of the same size as the input image, and use a pixel-wise loss like binary crossentropy.

Practice

1. Which of the following best explains why visual AI goes beyond simple image classification?

easy

A. Because it includes tasks like detecting objects and segmenting images for detailed understanding.

B. Because classification is the only task needed for image understanding.

C. Because visual AI only works with black and white images.

D. Because it ignores the content inside images.

Solution

1. Step 1: Understand the scope of visual AI

   Visual AI includes many tasks beyond classification, such as object detection and segmentation, which provide more detailed information about images.

2. Step 2: Compare options

   Because it includes tasks like detecting objects and segmenting images for detailed understanding. correctly states that visual AI includes these tasks, while others are incorrect or too narrow.

3. Final Answer:

   Because it includes tasks like detecting objects and segmenting images for detailed understanding. -> Option A

4. Quick Check:

   Visual AI = More than classification [OK]

Hint: Visual AI means detailed image understanding, not just labels [OK]

Common Mistakes:

• Thinking classification covers all visual AI tasks
• Ignoring object detection and segmentation
• Confusing image color with AI capability

2. Which TensorFlow function is commonly used to build a model for image segmentation tasks?

easy

A. tf.keras.Sequential()

B. tf.keras.Model()

C. tf.keras.layers.Conv2D()

D. tf.data.Dataset()

Solution

1. Step 1: Identify model building functions

   tf.keras.Model() allows building complex models like those needed for image segmentation, supporting multiple inputs and outputs.

2. Step 2: Compare other options

   tf.keras.Sequential() is simpler and less flexible, Conv2D is a layer, and tf.data.Dataset is for data input pipelines.

3. Final Answer:

   tf.keras.Model() -> Option B

4. Quick Check:

   Use tf.keras.Model for flexible segmentation models [OK]

Hint: Use tf.keras.Model for complex image tasks like segmentation [OK]

Common Mistakes:

• Confusing layers with model constructors
• Using Sequential for multi-output models
• Mixing data pipeline functions with model building

3. Given this TensorFlow code snippet for object detection, what will be the shape of the output tensor?

import tensorflow as tf
model = tf.keras.Sequential([
  tf.keras.layers.Conv2D(16, 3, activation='relu', input_shape=(64, 64, 3)),
  tf.keras.layers.Flatten(),
  tf.keras.layers.Dense(4)  # outputs bounding box coordinates
])
output = model(tf.random.uniform([1, 64, 64, 3]))
print(output.shape)

medium

A. (1, 4)

B. (64, 64, 16)

C. (1, 64, 64, 16)

D. (4,)

Solution

1. Step 1: Analyze model layers and output

   The model input is (64, 64, 3). Conv2D outputs (64, 64, 16). Flatten converts this to a vector, then Dense(4) outputs a vector of size 4.

2. Step 2: Check batch dimension

   Input batch size is 1, so output shape is (1, 4).

3. Final Answer:

   (1, 4) -> Option A

4. Quick Check:

   Output shape = (batch_size, 4) [OK]

Hint: Dense(4) outputs shape (batch_size, 4) after flatten [OK]

Common Mistakes:

• Confusing Conv2D output shape with final output
• Ignoring batch dimension
• Assuming output shape matches input image size

4. This TensorFlow code aims to perform image segmentation but has an error. What is the problem?

import tensorflow as tf
inputs = tf.keras.Input(shape=(128, 128, 3))
x = tf.keras.layers.Conv2D(32, 3, activation='relu')(inputs)
x = tf.keras.layers.Dense(1, activation='sigmoid')(x)
model = tf.keras.Model(inputs, x)

medium

A. Missing activation function in Conv2D layer.

B. Input shape is incorrect for image data.

C. Model lacks an output layer.

D. Dense layer cannot be applied directly to 4D tensor from Conv2D.

Solution

1. Step 1: Understand layer input/output shapes

   Conv2D outputs a 4D tensor (batch, height, width, channels). Dense expects 2D input (batch, features).

2. Step 2: Identify the error

   Applying Dense directly to Conv2D output causes shape mismatch error.

3. Final Answer:

   Dense layer cannot be applied directly to 4D tensor from Conv2D. -> Option D

4. Quick Check:

   Dense needs 2D input, Conv2D outputs 4D [OK]

Hint: Dense layers need flat input, not Conv2D 4D output [OK]

Common Mistakes:

• Assuming Dense works on 4D tensors
• Ignoring tensor shape requirements
• Thinking activation missing causes error

5. You want to build a TensorFlow model that not only classifies images but also detects multiple objects and segments them. Which approach best fits this goal?

hard

A. Use only a classification model and infer objects from class labels.

B. Train separate models for classification, detection, and segmentation and use them independently.

C. Use a single model with multiple outputs: one for classification, one for bounding boxes, and one for segmentation masks.

D. Use a simple Dense network with one output layer.

Solution

1. Step 1: Understand multi-task learning

   A single model with multiple outputs can learn classification, detection, and segmentation together, sharing features efficiently.

2. Step 2: Compare alternatives

   Separate models increase complexity and latency; classification-only models lack detection and segmentation; simple Dense networks cannot handle images well.

3. Final Answer:

   Use a single model with multiple outputs: one for classification, one for bounding boxes, and one for segmentation masks. -> Option C

4. Quick Check:

   Multi-output model fits combined visual AI tasks [OK]

Hint: Multi-output models handle classification, detection, segmentation together [OK]

Common Mistakes:

• Thinking separate models are always better
• Ignoring need for segmentation masks
• Using simple Dense layers for images