Choosing K (elbow method, silhouette score) in ML Python - Model Metrics & Evaluation

Metrics & Evaluation - Choosing K (elbow method, silhouette score)

Which metric matters for choosing K and WHY

When picking the number of clusters (K) in clustering, we want to find groups that are tight and well separated. Two common ways to check this are:

• Elbow method: Looks at how the total distance inside clusters drops as K grows. We want to find the point where adding more clusters doesn't help much. This point is called the "elbow".
• Silhouette score: Measures how similar each point is to its own cluster compared to other clusters. Scores near 1 mean good clusters, near 0 means overlapping, and negative means wrong grouping.

These metrics help us pick a K that balances simplicity and clear groups.

Confusion matrix or equivalent visualization

Clustering does not have a confusion matrix because it is unsupervised. Instead, we use plots:

  Total Within-Cluster Sum of Squares (WCSS) vs K (Elbow plot):

  K:       1    2    3    4    5    6
  WCSS:  500  300  200  150  140  135

  The "elbow" is at K=3 where WCSS stops dropping fast.

  Silhouette scores for K=2 to 6:

  K:           2     3     4     5     6
  Silhouette: 0.55  0.65  0.50  0.45  0.40

  Highest score at K=3 means best cluster separation.
  

Precision vs Recall tradeoff (or equivalent) with concrete examples

In clustering, the tradeoff is between having too few clusters (underfitting) and too many clusters (overfitting):

• Too few clusters: Groups are too broad, mixing different types together. This is like low recall in classification because you miss details.
• Too many clusters: Groups are too small or split unnecessarily. This is like low precision because clusters are not meaningful.

The elbow method helps find a balance by showing when adding clusters stops improving fit much. The silhouette score shows how well separated clusters are.

Example: If you cluster customers into 2 groups, you might miss important differences (low recall). If you use 10 groups, some might be too small or noisy (low precision). Choosing 3 or 4 clusters might be best.

What "good" vs "bad" metric values look like for this use case

• Elbow method: A clear bend or "elbow" in the WCSS plot means a good choice of K. If the plot is smooth with no bend, it is hard to pick K.
• Silhouette score: Values close to 1 (e.g., >0.5) mean good, well-separated clusters. Values near 0 mean overlapping clusters. Negative values mean wrong clustering.

Good example: K=3 with silhouette 0.65 and clear elbow at 3.

Bad example: No clear elbow and silhouette scores all below 0.3, meaning clusters are not distinct.

Metrics pitfalls

• No clear elbow: Sometimes the WCSS plot is smooth, making it hard to pick K. This can confuse beginners.
• Silhouette score bias: It prefers spherical clusters and may not work well if clusters have weird shapes.
• Ignoring domain knowledge: Metrics alone may not capture what makes sense in real life. Always check if clusters are meaningful.
• Overfitting with too many clusters: Picking a very high K can give perfect WCSS but meaningless groups.

Self-check question

Your clustering model shows a smooth WCSS plot with no clear elbow and silhouette scores around 0.2 for K=2 to 6. Is this a good choice? Why or why not?

Answer: No, this is not good. The low silhouette scores mean clusters overlap or are not well separated. The lack of an elbow means no clear best K. You should try other methods or features.

Key Result

Use the elbow method to find where adding clusters stops helping and silhouette score to measure cluster quality; good clusters have a clear elbow and silhouette above 0.5.

Practice

1. What is the main goal when choosing the number of clusters K in clustering algorithms?

easy

A. To find the number of groups that best represent the data

B. To maximize the number of clusters regardless of data

C. To minimize the number of clusters to one

D. To randomly select a number without analysis

Solution

1. Step 1: Understand the purpose of clustering

   Clustering groups similar data points together to find patterns.

2. Step 2: Identify the role of choosing K

   Choosing K means deciding how many groups best represent the data structure.

3. Final Answer:

   To find the number of groups that best represent the data -> Option A

4. Quick Check:

   Choosing K = best data groups [OK]

Hint: Choose K to best represent data groups, not randomly [OK]

Common Mistakes:

• Picking K without checking data patterns
• Choosing too many clusters blindly
• Ignoring the meaning of clusters

2. Which of the following Python code snippets correctly computes the silhouette score for clustering results stored in labels with data X?

easy

A. from sklearn.metrics import silhouette_score
score = silhouette_score(labels, X)

B. from sklearn.cluster import silhouette_score
score = silhouette_score(labels, X)

C. from sklearn.metrics import silhouette_score
score = silhouette_score(X, labels)

D. from sklearn.cluster import silhouette_score
score = silhouette_score(X, labels)

Solution

1. Step 1: Check the correct import

   The silhouette_score function is in sklearn.metrics, not sklearn.cluster.

2. Step 2: Verify function parameters order

   The function takes data first (X), then cluster labels (labels).

3. Final Answer:

   from sklearn.metrics import silhouette_score\nscore = silhouette_score(X, labels) -> Option C

4. Quick Check:

   Correct import and parameter order = from sklearn.metrics import silhouette_score score = silhouette_score(X, labels) [OK]

Hint: Import from metrics and pass data first, labels second [OK]

Common Mistakes:

• Importing silhouette_score from wrong module
• Swapping data and labels in function call
• Using cluster module instead of metrics

3. Given the following code, what will be the printed silhouette score?

from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score
import numpy as np

X = np.array([[1, 2], [1, 4], [1, 0], [10, 2], [10, 4], [10, 0]])
kmeans = KMeans(n_clusters=2, random_state=42).fit(X)
labels = kmeans.labels_
score = silhouette_score(X, labels)
print(round(score, 2))

medium

A. 0.50

B. 0.74

C. 0.30

D. 1.00

Solution

1. Step 1: Understand the data and clustering

   The data has two clear groups around [1, x] and [10, x], so KMeans with 2 clusters fits well.

2. Step 2: Calculate silhouette score

   Silhouette score measures cluster quality; for well-separated clusters, it is close to 1. Running the code gives approximately 0.74.

3. Final Answer:

   0.74 -> Option B

4. Quick Check:

   Well-separated clusters give silhouette ~0.74 [OK]

Hint: Well-separated clusters usually have silhouette > 0.7 [OK]

Common Mistakes:

• Expecting silhouette score to be 1.0 always
• Confusing silhouette with inertia
• Ignoring cluster separation in score

4. You run the elbow method but the inertia keeps decreasing smoothly without a clear elbow point. What is the best way to fix this issue?

medium

A. Try silhouette score to find the best K instead

B. Increase the number of clusters to the maximum possible

C. Ignore the elbow method and pick K=1

D. Use random K values without checking metrics

Solution

1. Step 1: Identify the problem with elbow method

   If inertia decreases smoothly, no clear elbow means it's hard to pick K by this method.

2. Step 2: Use alternative metric

   Silhouette score measures cluster quality differently and can help find the best K when elbow is unclear.

3. Final Answer:

   Try silhouette score to find the best K instead -> Option A

4. Quick Check:

   Use silhouette when elbow is unclear [OK]

Hint: Switch to silhouette score if elbow is unclear [OK]

Common Mistakes:

• Forcing a large K without reason
• Ignoring metrics and guessing K
• Picking K=1 without analysis

5. You have a dataset with 5 natural clusters but the elbow method suggests K=3 and silhouette score suggests K=5. Which K should you choose and why?

hard

A. Choose K=3 because elbow method always gives the best K

B. Choose K=10 to overfit the data

C. Choose K=1 to simplify the model

D. Choose K=5 because silhouette score better reflects cluster separation

Solution

1. Step 1: Compare elbow and silhouette methods

   Elbow method focuses on inertia reduction, which can miss subtle clusters; silhouette measures cluster quality and separation.

2. Step 2: Match method to data nature

   Since the dataset has 5 natural clusters, silhouette score's suggestion aligns better with true structure.

3. Final Answer:

   Choose K=5 because silhouette score better reflects cluster separation -> Option D

4. Quick Check:

   Silhouette score > elbow for true cluster count [OK]

Hint: Trust silhouette score for natural cluster counts [OK]

Common Mistakes:

• Blindly trusting elbow method over silhouette
• Choosing too few clusters ignoring data
• Overfitting by choosing too many clusters