K-Nearest Neighbors (KNN) in ML Python - Model Metrics & Evaluation

Metrics & Evaluation - K-Nearest Neighbors (KNN)

Which metric matters for K-Nearest Neighbors (KNN) and WHY

KNN is often used for classification tasks. The key metrics to evaluate KNN are Accuracy, Precision, Recall, and F1-score. These metrics tell us how well KNN predicts the correct class.

Accuracy shows overall correctness, but if classes are imbalanced, Precision and Recall become more important. Precision tells us how many predicted positives are actually correct, while Recall tells us how many actual positives were found. F1-score balances both.

For KNN, since it relies on neighbors, it can be sensitive to noisy data, so these metrics help us understand if it is making good predictions or just guessing.

Confusion Matrix Example for KNN

    Actual \ Predicted | Positive | Negative
    -------------------|----------|---------
    Positive           |    40    |   10    
    Negative           |    5     |   45    

    Total samples = 40 + 10 + 5 + 45 = 100

    Precision = TP / (TP + FP) = 40 / (40 + 5) = 0.89
    Recall = TP / (TP + FN) = 40 / (40 + 10) = 0.80
    F1-score = 2 * (0.89 * 0.80) / (0.89 + 0.80) ≈ 0.84
    Accuracy = (TP + TN) / Total = (40 + 45) / 100 = 0.85
    

Precision vs Recall Tradeoff in KNN

Imagine KNN is used to detect spam emails.

• High Precision: Most emails marked as spam really are spam. This means fewer good emails get wrongly blocked.
• High Recall: Most spam emails are caught. This means fewer spam emails get through.

If KNN has high precision but low recall, it misses many spam emails (bad). If it has high recall but low precision, many good emails are wrongly marked as spam (also bad).

Choosing K (number of neighbors) affects this tradeoff. A smaller K may catch more spam (higher recall) but also misclassify good emails (lower precision). A larger K smooths predictions, improving precision but possibly missing spam.

What Good vs Bad Metrics Look Like for KNN

Good KNN model:

• Accuracy above 80% on balanced data
• Precision and Recall both above 75%
• F1-score close to Precision and Recall, showing balance

Bad KNN model:

• Accuracy near random guess (e.g., 50% for two classes)
• Precision very low (e.g., below 50%), meaning many false alarms
• Recall very low, meaning many missed positives
• F1-score much lower than either Precision or Recall, showing imbalance

Common Pitfalls in KNN Metrics

• Accuracy Paradox: High accuracy can be misleading if classes are imbalanced. For example, if 95% of data is negative, predicting all negative yields 95% accuracy but zero recall for positives.
• Data Leakage: If test data leaks into training, KNN will appear perfect because neighbors are identical.
• Overfitting: Very small K (like 1) can memorize training data, causing poor generalization and unstable metrics on new data.
• Ignoring Distance Scaling: Features with different scales can distort neighbor distances, hurting metrics.

Self-Check Question

Your KNN model has 98% accuracy but only 12% recall on the positive class (e.g., fraud detection). Is this model good for production? Why or why not?

Answer: No, it is not good. Although accuracy is high, the model misses 88% of actual positive cases (fraud). This low recall means many fraud cases go undetected, which is risky. For fraud detection, high recall is critical to catch as many frauds as possible.

Key Result

KNN evaluation focuses on balanced Precision and Recall to ensure reliable classification beyond simple Accuracy.

Practice

1. What does the K in K-Nearest Neighbors (KNN) represent?

easy

A. The number of closest data points used to predict the label

B. The number of features in the dataset

C. The number of classes in the target variable

D. The number of training samples

Solution

1. Step 1: Understand the role of K in KNN

   K represents how many nearest neighbors the algorithm looks at to decide the label for a new point.

2. Step 2: Differentiate K from other terms

   K is not about features, classes, or total samples but specifically the count of neighbors considered.

3. Final Answer:

   The number of closest data points used to predict the label -> Option A

4. Quick Check:

   K = number of neighbors [OK]

Hint: K means neighbors counted to decide label [OK]

Common Mistakes:

• Confusing K with number of features
• Thinking K is number of classes
• Assuming K is total data points

2. Which of the following is the correct way to import KNeighborsClassifier from scikit-learn in Python?

easy

A. import sklearn.neighbors.KNeighborsClassifier

B. import KNeighborsClassifier from sklearn.neighbors

C. from sklearn import KNeighborsClassifier

D. from sklearn.neighbors import KNeighborsClassifier

Solution

1. Step 1: Recall correct Python import syntax

   In Python, the correct way to import a class from a module is 'from module import Class'.

2. Step 2: Match with scikit-learn KNN import

   The KNeighborsClassifier is inside sklearn.neighbors, so the correct import is 'from sklearn.neighbors import KNeighborsClassifier'.

3. Final Answer:

   from sklearn.neighbors import KNeighborsClassifier -> Option D

4. Quick Check:

   Correct import uses 'from ... import ...' [OK]

Hint: Use 'from module import Class' syntax [OK]

Common Mistakes:

• Using 'import Class from module' which is invalid
• Trying to import directly from sklearn without neighbors
• Using dot notation in import statement

3. What will be the predicted label for the point (3, 3) using KNN with k=3 and the following training data?

X = [[1, 2], [2, 3], [3, 4], [6, 7]]
y = ['A', 'A', 'B', 'B']

medium

A. 'B'

B. 'A'

C. 'C'

D. Error due to missing label

Solution

1. Step 1: Calculate distances from (3,3) to each training point

   Distances: to (1,2) ≈ 2.24, to (2,3) = 1, to (3,4) = 1, to (6,7) ≈ 5

2. Step 2: Find 3 nearest neighbors and their labels

   Nearest neighbors: (2,3) 'A', (3,4) 'B', (1,2) 'A'. Labels are ['A', 'B', 'A'].

3. Step 3: Determine majority label

   Label 'A' appears twice, 'B' once, so predicted label is 'A'.

4. Final Answer:

   'A' -> Option B

5. Quick Check:

   Majority vote among 3 neighbors = 'A' [OK]

Hint: Pick majority label among closest 3 points [OK]

Common Mistakes:

• Ignoring distance and picking random label
• Choosing label of closest single neighbor only
• Miscounting neighbors or labels

4. Identify the error in this KNN code snippet:

from sklearn.neighbors import KNeighborsClassifier
knn = KNeighborsClassifier(n_neighbors=3)
knn.fit(X_train, y_train)
preds = knn.predict(X_test)
print(preds)

Assuming X_train and y_train are numpy arrays but y_train contains string labels.

medium

A. Missing data scaling causes error

B. y_train must be numeric, strings cause error

C. No error, code runs fine with string labels

D. n_neighbors must be less than number of samples

Solution

1. Step 1: Check if KNN supports string labels

   KNN in scikit-learn supports string labels for classification without error.

2. Step 2: Verify other possible errors

   Data scaling is recommended but not mandatory; n_neighbors=3 is valid if training samples ≥3.

3. Final Answer:

   No error, code runs fine with string labels -> Option C

4. Quick Check:

   KNN accepts string labels directly [OK]

Hint: KNN accepts string labels; no numeric conversion needed [OK]

Common Mistakes:

• Assuming string labels cause errors
• Confusing scaling as mandatory
• Ignoring sample size for n_neighbors

5. You have a dataset with features on very different scales (e.g., age 0-100, income 0-100000). Why is it important to scale features before using KNN, and what could happen if you don't?

hard

A. Scaling ensures all features contribute equally; without it, large-scale features dominate distance calculations

B. Scaling is not needed; KNN automatically adjusts for feature scales

C. Scaling reduces training time but does not affect predictions

D. Scaling changes labels to numeric values for KNN

Solution

1. Step 1: Understand distance calculation in KNN

   KNN uses distance (like Euclidean) between points; features with larger scales affect distance more.

2. Step 2: Explain effect of unscaled features

   If features are not scaled, large-scale features (like income) dominate, making smaller-scale features (like age) ignored.

3. Step 3: Importance of scaling

   Scaling puts all features on similar scale so each contributes fairly to distance and prediction.

4. Final Answer:

   Scaling ensures all features contribute equally; without it, large-scale features dominate distance calculations -> Option A

5. Quick Check:

   Scale features to balance distances [OK]

Hint: Scale features so no one dominates distance [OK]

Common Mistakes:

• Thinking KNN auto-scales features
• Believing scaling changes labels
• Assuming scaling only affects speed