Face landmark detection in Computer Vision - ML Experiment: Train & Evaluate

Experiment - Face landmark detection

Problem:Detect key points on faces such as eyes, nose, and mouth using a neural network.

Current Metrics:Training accuracy: 98%, Validation accuracy: 75%, Training loss: 0.05, Validation loss: 0.25

Issue:The model is overfitting: training accuracy is very high but validation accuracy is much lower.

Your Task

Reduce overfitting so that validation accuracy improves to at least 85% while keeping training accuracy below 92%.

Do not change the dataset or add more data.

Keep the same neural network architecture base.

Only adjust hyperparameters and add regularization techniques.

Hint 1

Hint 2

Hint 3

Hint 4

Solution

Computer Vision

import tensorflow as tf
from tensorflow.keras import layers, models
from tensorflow.keras.callbacks import EarlyStopping

# Assume X_train, y_train, X_val, y_val are preloaded face images and landmarks

model = models.Sequential([
    layers.Conv2D(32, (3,3), activation='relu', input_shape=(96, 96, 1)),
    layers.BatchNormalization(),
    layers.MaxPooling2D(2,2),
    layers.Dropout(0.25),

    layers.Conv2D(64, (3,3), activation='relu'),
    layers.BatchNormalization(),
    layers.MaxPooling2D(2,2),
    layers.Dropout(0.25),

    layers.Flatten(),
    layers.Dense(128, activation='relu'),
    layers.Dropout(0.5),
    layers.Dense(30)  # 15 landmarks x 2 coordinates
])

model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.0005),
              loss='mean_squared_error')

early_stop = EarlyStopping(monitor='val_loss', patience=10, restore_best_weights=True)

history = model.fit(X_train, y_train,
                    epochs=100,
                    batch_size=32,
                    validation_data=(X_val, y_val),
                    callbacks=[early_stop])

Added dropout layers after convolution and dense layers to reduce overfitting.

Added batch normalization layers to stabilize training.

Reduced learning rate from default to 0.0005 for smoother convergence.

Added early stopping to stop training when validation loss stops improving.

Results Interpretation

Before: Training accuracy 98%, Validation accuracy 75%, Training loss 0.05, Validation loss 0.25

After: Training accuracy 90%, Validation accuracy 87%, Training loss 0.08, Validation loss 0.15

Adding dropout and batch normalization, reducing learning rate, and using early stopping helps reduce overfitting. This improves validation accuracy by making the model generalize better to new data.

Bonus Experiment

Try using data augmentation techniques like random rotations and shifts to further improve validation accuracy.

💡 Hint

Use TensorFlow's ImageDataGenerator to apply simple transformations on training images to increase data variety.

Practice

1. What is the main purpose of face landmark detection in computer vision?

easy

A. To detect the presence of multiple faces in an image

B. To classify the face into different age groups

C. To find key points on a face that represent facial features

D. To convert a face image into grayscale

Solution

1. Step 1: Understand face landmark detection

   Face landmark detection identifies specific points on a face like eyes, nose, and mouth corners.

2. Step 2: Differentiate from other tasks

   Other tasks like age classification or face detection do not focus on key facial points.

3. Final Answer:

   To find key points on a face that represent facial features -> Option C

4. Quick Check:

   Face landmark detection = key facial points [OK]

Hint: Landmark detection means finding facial points, not classifying faces [OK]

Common Mistakes:

• Confusing face detection with landmark detection
• Thinking it classifies face attributes like age or emotion
• Assuming it changes image colors

2. Which of the following is the correct way to load a pre-trained face landmark model using MediaPipe in Python?

easy

A. mp_face_mesh = mp.solutions.face_mesh.FaceMesh()

B. mp_face_mesh = mp.solutions.load_face_mesh()

C. mp_face_mesh = mp.FaceMesh()

D. mp_face_mesh = mp.solutions.face_landmark()

Solution

1. Step 1: Recall MediaPipe face mesh syntax

   The correct class to load face landmark detection is FaceMesh under mp.solutions.face_mesh.

2. Step 2: Check each option

   Only mp_face_mesh = mp.solutions.face_mesh.FaceMesh() uses the correct full path and class name.

3. Final Answer:

   mp_face_mesh = mp.solutions.face_mesh.FaceMesh() -> Option A

4. Quick Check:

   Correct MediaPipe class = FaceMesh [OK]

Hint: MediaPipe face landmarks use mp.solutions.face_mesh.FaceMesh() [OK]

Common Mistakes:

• Using incorrect method names like load_face_mesh()
• Omitting the full module path
• Using non-existent classes like face_landmark()

3. Given the following code snippet using MediaPipe to detect face landmarks, what will be the output type of results.multi_face_landmarks if a face is detected?

medium

A. A list of face landmark objects

B. A single integer representing number of faces

C. A dictionary with landmark coordinates

D. None

Solution

1. Step 1: Understand MediaPipe output for face landmarks

   When a face is detected, results.multi_face_landmarks returns a list of landmark objects, one per face.

2. Step 2: Analyze the options

   Only A list of face landmark objects correctly describes this as a list of face landmark objects.

3. Final Answer:

   A list of face landmark objects -> Option A

4. Quick Check:

   Detected faces landmarks = list of objects [OK]

Hint: multi_face_landmarks returns a list, not a number or dict [OK]

Common Mistakes:

• Thinking it returns a count (integer)
• Assuming it returns a dictionary
• Confusing None with detected results

4. You wrote this code to draw face landmarks but get an error: mp_drawing.draw_landmarks(image, results.face_landmarks, mp_face_mesh.FACEMESH_TESSELATION). What is the likely cause?

medium

A. FACEMESH_TESSELATION is misspelled

B. mp_drawing is not imported

C. image must be grayscale

D. results.face_landmarks should be results.multi_face_landmarks[0]

Solution

1. Step 1: Check results object structure

   MediaPipe stores landmarks in results.multi_face_landmarks as a list, not results.face_landmarks.

2. Step 2: Correct usage for drawing

   To draw landmarks for the first face, use results.multi_face_landmarks[0].

3. Final Answer:

   results.face_landmarks should be results.multi_face_landmarks[0] -> Option D

4. Quick Check:

   Use multi_face_landmarks list for drawing [OK]

Hint: Use multi_face_landmarks[0] to access first face landmarks [OK]

Common Mistakes:

• Using a non-existent attribute results.face_landmarks
• Forgetting to import mp_drawing
• Assuming image must be grayscale

5. You want to build an app that applies virtual glasses on a user's face using face landmark detection. Which landmarks should you primarily use to position the glasses correctly?

hard

A. Landmarks on the lips and chin

B. Landmarks around the eyes and nose bridge

C. Landmarks on the forehead and cheeks

D. Landmarks on the ears and jawline

Solution

1. Step 1: Identify glasses placement on face

   Glasses rest on the nose bridge and cover the eyes area.

2. Step 2: Choose landmarks matching these areas

   Landmarks around eyes and nose bridge provide accurate positioning points.

3. Final Answer:

   Landmarks around the eyes and nose bridge -> Option B

4. Quick Check:

   Glasses position = eyes + nose bridge landmarks [OK]

Hint: Glasses align with eyes and nose bridge landmarks [OK]

Common Mistakes:

• Using lip or chin landmarks which are too low
• Choosing forehead or cheeks which don't fit glasses
• Selecting ears which are not detected by standard landmarks