Annotation quality in Computer Vision - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Why is annotation quality important in supervised learning?

Imagine you are training a model to recognize cats in photos. Why does the quality of the annotations (labels) matter?

A Annotations only affect the speed of training, not the model accuracy.

B Poor annotations can confuse the model, leading to wrong predictions.

C High-quality annotations make the model run faster on new data.

D Annotation quality is not important if the dataset is very large.

💡 Hint

Think about what happens if the model learns from wrong labels.

Good annotations help the model learn correct patterns. Bad labels cause confusion and reduce accuracy.

❓ Metrics

intermediate

Measuring annotation consistency

You have two annotators labeling images for object detection. Which metric best measures how much they agree on the labels?

A Precision-Recall Curve

B Intersection over Union (IoU)

C Mean Squared Error (MSE)

D Cohen's Kappa

💡 Hint

Look for a metric that measures agreement between annotators.

Cohen's Kappa measures agreement beyond chance between two annotators.

❓ Predict Output

advanced

Output of annotation quality check code

What is the output of this Python code that calculates annotation agreement?

Computer Vision

from sklearn.metrics import cohen_kappa_score
labels_annotator1 = [1, 0, 1, 1, 0]
labels_annotator2 = [1, 0, 0, 1, 0]
kappa = cohen_kappa_score(labels_annotator1, labels_annotator2)
print(round(kappa, 2))

A 0.62

B 0.75

C 1.00

D 0.40

💡 Hint

Calculate agreement considering chance agreement.

The Cohen's Kappa score for these labels is 0.62, indicating moderate agreement.

❓ Model Choice

advanced

Choosing a model to handle noisy annotations

You have a dataset with some incorrect labels. Which model approach is best to reduce the impact of noisy annotations?

A A deep neural network with dropout and early stopping

B A simple linear model without regularization

C A decision tree with no pruning

D A nearest neighbor model using all training points

💡 Hint

Think about models that can avoid overfitting noisy data.

Dropout and early stopping help neural networks avoid fitting noise, improving robustness to bad labels.

🔧 Debug

expert

Debugging annotation error impact on model accuracy

After training a classifier, you notice low accuracy. You suspect annotation errors. Which step below will best help identify if annotation quality caused the problem?

A Increase the learning rate to speed up training

B Train the model on a smaller subset of data without checking labels

C Manually review a random sample of annotations and compare with model predictions

D Add more layers to the model to improve capacity

💡 Hint

Think about how to verify if labels are correct.

Manually reviewing annotations helps find label mistakes that may cause low accuracy.