Flat clustering (fcluster) in SciPy - Deep Dive

Overview - Flat clustering (fcluster)

What is it?

Flat clustering is a way to group data points into distinct clusters without any hierarchy. The function fcluster in scipy helps to cut a hierarchical clustering tree into flat clusters by setting a threshold. This means you decide how similar points must be to belong to the same group. It simplifies complex nested groups into clear, separate clusters.

Why it matters

Without flat clustering, it would be hard to decide where to stop in a hierarchical clustering tree, making it difficult to use the results practically. Flat clustering lets you choose a clear number or distance for groups, which is essential for tasks like customer segmentation or image grouping. It turns complex relationships into actionable groups that businesses and researchers can use.

Where it fits

Before learning flat clustering, you should understand basic clustering concepts and hierarchical clustering methods. After mastering flat clustering, you can explore advanced clustering evaluation techniques and other clustering algorithms like k-means or DBSCAN.

Mental Model

Core Idea

Flat clustering cuts a hierarchical tree at a chosen level to form distinct, non-overlapping groups of data points.

Think of it like...

Imagine a family tree that shows all relatives connected by branches. Flat clustering is like cutting the tree at a certain height so you only see groups of cousins, ignoring deeper or higher connections.

Hierarchical Tree
  Root
   │
   ├── Cluster A
   │     ├── Point 1
   │     └── Point 2
   ├── Cluster B
   │     ├── Point 3
   │     └── Point 4
Cut at threshold ──────────────▶ Flat clusters: {A, B}

Build-Up - 6 Steps

1

Foundation

Understanding hierarchical clustering basics

Concept: Hierarchical clustering builds a tree of clusters by merging or splitting data points step-by-step.

Hierarchical clustering starts with each point as its own cluster. It then merges the closest clusters step-by-step until all points form one big cluster. This process creates a tree called a dendrogram, showing how clusters join at different distances.

Result

You get a dendrogram that visually shows cluster relationships and distances between points.

Understanding the dendrogram is key because flat clustering depends on cutting this tree at the right place.

2

Foundation

What is flat clustering?

Concept: Flat clustering groups data points into separate clusters without any nested structure.

Unlike hierarchical clustering, flat clustering gives you a simple list of clusters. Each point belongs to exactly one cluster, and there is no hierarchy or subgroups inside clusters.

Result

You get clear, separate groups that are easy to use for analysis or decision-making.

Flat clustering simplifies complex hierarchical data into practical groups.

3

Intermediate

Using fcluster to cut dendrograms

🤔Before reading on: do you think fcluster cuts clusters by number of clusters or by distance threshold? Commit to your answer.

Concept: The fcluster function cuts the hierarchical tree at a chosen distance or criterion to form flat clusters.

In scipy, fcluster takes a linkage matrix from hierarchical clustering and a threshold value. It assigns cluster labels to points by cutting the dendrogram where the distance between clusters exceeds the threshold.

Result

You get an array of cluster labels, one for each data point, defining flat clusters.

Knowing that fcluster uses a distance threshold helps you control cluster size and similarity.

4

Intermediate

Different criteria for fcluster cutting

🤔Before reading on: do you think fcluster supports cutting by maximum distance only, or are there other ways? Commit to your answer.

Concept: fcluster supports multiple criteria to decide how to cut the dendrogram, like maximum distance or maximum number of clusters.

Besides cutting by distance, fcluster can cut by specifying the maximum number of clusters or by inconsistent cluster depth. This flexibility lets you choose clusters based on your problem's needs.

Result

You can create flat clusters by different rules, not just distance, adapting to different data shapes.

Understanding multiple cutting criteria lets you tailor clustering to your specific goals.

5

Advanced

Interpreting fcluster output labels

🤔Before reading on: do you think cluster labels from fcluster are always sorted or can they be arbitrary? Commit to your answer.

Concept: fcluster returns cluster labels as integers, but their order or numbering does not imply size or order of clusters.

The cluster labels are arbitrary integers assigned to each cluster. For example, cluster 1 is not necessarily the biggest or first cluster. You should treat labels as identifiers only.

Result

You get a label array like [1,1,2,2,3], but label numbers have no inherent meaning beyond grouping.

Knowing labels are arbitrary prevents wrong assumptions about cluster importance or order.

6

Expert

Limitations and surprises of fcluster usage

🤔Before reading on: do you think fcluster always produces stable clusters if you slightly change the threshold? Commit to your answer.

Concept: Small changes in the threshold can cause large changes in cluster assignments due to dendrogram structure.

Because hierarchical clustering merges clusters stepwise, cutting at slightly different distances can merge or split clusters unexpectedly. This sensitivity means you must carefully choose thresholds and validate clusters.

Result

Cluster assignments can jump suddenly with small threshold changes, affecting analysis stability.

Understanding threshold sensitivity helps avoid misleading conclusions and encourages robustness checks.

Under the Hood

fcluster works by traversing the hierarchical clustering tree (linkage matrix) and assigning cluster labels to points based on where the tree is cut. It compares the distances at which clusters merge to the threshold and groups points accordingly. Internally, it uses efficient tree traversal and union-find structures to assign labels quickly.

Why designed this way?

The design reflects the need to convert a nested hierarchy into flat groups for practical use. Using a threshold on merge distances is intuitive and flexible. Alternatives like fixed cluster counts exist but are less adaptable to data shape. The linkage matrix format is standard in hierarchical clustering, making fcluster a natural extension.

Linkage Matrix (distance sorted merges)
┌─────────────┐
│   Merge 1   │
│ Points 1 & 2│
│ Distance d1 │
└─────┬───────┘
      │
┌─────▼───────┐
│   Merge 2   │
│ Merge1 & 3  │
│ Distance d2 │
└─────┬───────┘
      │
  Cut threshold
      │
  ┌───▼───┐
  │Clusters│
  └───────┘

Myth Busters - 4 Common Misconceptions

Quick: Does fcluster always produce the same clusters if you change the threshold slightly? Commit yes or no.

Common Belief: fcluster clusters change smoothly and predictably with threshold changes.

Reality: Small threshold changes can cause sudden jumps in cluster assignments due to the hierarchical merge steps.

Why it matters: Assuming smooth changes can lead to unstable cluster interpretations and poor decisions.

Quick: Do cluster labels from fcluster indicate cluster size or importance? Commit yes or no.

Common Belief: Lower cluster labels mean bigger or more important clusters.

Reality: Cluster labels are arbitrary identifiers without size or importance meaning.

Why it matters: Misinterpreting labels can cause wrong assumptions about data structure.

Quick: Can fcluster be used without hierarchical clustering? Commit yes or no.

Common Belief: You can use fcluster on any data without hierarchical clustering first.

Reality: fcluster requires a linkage matrix from hierarchical clustering as input.

Why it matters: Trying to use fcluster alone leads to errors and confusion.

Quick: Does fcluster always produce the same number of clusters if you specify a distance threshold? Commit yes or no.

Common Belief: A fixed distance threshold always results in the same number of clusters.

Reality: The number of clusters depends on data structure; the same threshold can yield different cluster counts for different datasets.

Why it matters: Expecting fixed cluster counts can mislead analysis and parameter tuning.

Expert Zone

1

The choice of linkage method (single, complete, average) affects how fcluster cuts the dendrogram and the resulting clusters.

2

Inconsistent or dynamic tree cutting criteria can produce more meaningful clusters than fixed distance thresholds in complex data.

3

fcluster labels are not stable across different hierarchical clustering runs due to randomness or tie-breaking in merges.

When NOT to use

Avoid fcluster when data has noise or irregular shapes better handled by density-based clustering like DBSCAN. Also, if you need overlapping clusters or fuzzy memberships, use soft clustering methods instead.

Production Patterns

In real systems, fcluster is used after hierarchical clustering to segment customers or group similar items. Often combined with silhouette analysis or gap statistics to choose thresholds. It is also used to preprocess data for supervised learning by creating cluster-based features.

Connections

Hierarchical clustering

fcluster builds directly on hierarchical clustering results by cutting its dendrogram.

Understanding hierarchical clustering is essential to use fcluster effectively and interpret its output.

Thresholding in signal processing

Both use thresholds to separate meaningful groups or signals from noise.

Knowing thresholding concepts in other fields helps grasp how cutting dendrograms isolates clusters.

Taxonomy in biology

Taxonomy organizes species hierarchically, and flat clustering is like choosing a taxonomic rank to group species.

Seeing clustering as taxonomy clarifies why cutting at different levels changes group granularity.

Common Pitfalls

#1 Using fcluster without a proper linkage matrix.

Wrong approach:

from scipy.cluster.hierarchy import fcluster
labels = fcluster(data, t=1.5, criterion='distance')

Correct approach:

from scipy.cluster.hierarchy import linkage, fcluster
Z = linkage(data, method='ward')
labels = fcluster(Z, t=1.5, criterion='distance')

Root cause:

Misunderstanding that fcluster needs hierarchical clustering output, not raw data.

#2 Assuming cluster labels indicate cluster size or order.

Wrong approach:

print('Largest cluster is label 1')
# Using label 1 as biggest cluster without checking sizes

Correct approach:

import numpy as np
unique, counts = np.unique(labels, return_counts=True)
largest = unique[np.argmax(counts)]
print(f'Largest cluster label is {largest}')

Root cause:

Confusing arbitrary cluster labels with meaningful cluster properties.

#3 Setting threshold too high or too low without validation.

Wrong approach:

labels = fcluster(Z, t=10, criterion='distance') # Arbitrary large threshold

Correct approach:

from scipy.cluster.hierarchy import dendrogram
import matplotlib.pyplot as plt
plt.figure()
dendrogram(Z)
plt.show()
# Choose threshold based on dendrogram visualization
labels = fcluster(Z, t=3, criterion='distance')

Root cause:

Ignoring dendrogram structure leads to poor cluster choices.

Key Takeaways

Flat clustering simplifies hierarchical clustering by cutting the dendrogram at a chosen threshold to form distinct groups.

The fcluster function in scipy requires a linkage matrix and uses distance or other criteria to assign cluster labels.

Cluster labels from fcluster are arbitrary identifiers and do not indicate size or importance.

Small changes in the threshold can cause large changes in cluster assignments, so threshold choice must be done carefully.

Flat clustering is useful for turning complex nested data into actionable groups but has limits when data is noisy or clusters overlap.

Practice

1. What is the main purpose of the fcluster function in scipy's hierarchical clustering?

easy

A. To cut a hierarchical cluster tree into flat clusters based on a threshold

B. To compute the distance matrix between data points

C. To perform dimensionality reduction before clustering

D. To normalize data before clustering

Solution

1. Step 1: Understand hierarchical clustering output

   Hierarchical clustering produces a tree (dendrogram) showing nested clusters.

2. Step 2: Role of fcluster

   fcluster cuts this tree at a chosen threshold to form flat, non-overlapping clusters.

3. Final Answer:

   To cut a hierarchical cluster tree into flat clusters based on a threshold -> Option A

4. Quick Check:

   Flat clustering = cutting tree with threshold [OK]

Hint: Remember: fcluster cuts dendrogram into flat groups [OK]

Common Mistakes:

• Confusing fcluster with distance calculation
• Thinking fcluster normalizes data
• Assuming fcluster reduces dimensions

2. Which of the following is the correct syntax to assign flat clusters using fcluster with a distance threshold of 1.5 from a linkage matrix Z?

easy

A. clusters = fcluster(Z, threshold=1.5, criterion='distance')

B. clusters = fcluster(Z, 1.5, method='distance')

C. clusters = fcluster(Z, 1.5, criterion='distance')

D. clusters = fcluster(Z, 1.5, criterion='maxclust')

Solution

1. Step 1: Check fcluster parameters

   The function signature is fcluster(Z, t, criterion='distance') where t is the threshold.

2. Step 2: Identify correct usage

   clusters = fcluster(Z, 1.5, criterion='distance') uses t=1.5 and criterion='distance', which is correct syntax.

3. Final Answer:

   clusters = fcluster(Z, 1.5, criterion='distance') -> Option C

4. Quick Check:

   Threshold = 1.5, criterion = 'distance' [OK]

Hint: Use t for threshold and criterion='distance' in fcluster [OK]

Common Mistakes:

• Using 'method' instead of 'criterion'
• Passing threshold as keyword 'threshold'
• Using wrong criterion like 'maxclust' for distance cut

3. Given the linkage matrix Z = [[0, 1, 0.5, 2], [2, 3, 1.5, 2], [4, 5, 2.5, 4]], what is the output of fcluster(Z, 1.0, criterion='distance')?

medium

A. [1 1 1 1]

B. [1 2 3 4]

C. [1 2 2 3]

D. [1 1 2 3]

Solution

1. Step 1: Understand linkage matrix and threshold

   The linkage matrix Z shows merges with distances: 0.5, 1.5, 2.5. Threshold is 1.0.

2. Step 2: Assign clusters by cutting at distance 1.0

   Clusters merge if distance ≤ 1.0. The first merge (0.5) joins points 0 and 1 into cluster 1. The second merge (1.5 > 1.0) does not merge points 2 and 3, so point 2 gets cluster 2 and point 3 gets cluster 3.

3. Final Answer:

   [1 1 2 3] -> Option D

4. Quick Check:

   Distance ≤ 1.0 merges points 0 and 1 only [OK]

Hint: Cut dendrogram at threshold; merges below threshold cluster together [OK]

Common Mistakes:

• Merging clusters above threshold
• Assigning all points to one cluster
• Misreading linkage matrix format

4. You run the code clusters = fcluster(Z, 2, criterion='maxclust') but get an error. What is the likely cause?

medium

A. The linkage matrix Z is not defined or invalid

B. The criterion 'maxclust' requires an integer number of clusters, but 2 is passed as a float

C. The threshold parameter must be a float when using 'maxclust'

D. The criterion 'maxclust' expects the threshold to be the maximum cluster distance

Solution

1. Step 1: Check parameter types for 'maxclust'

   When using criterion='maxclust', the threshold t must be an integer specifying the number of clusters.

2. Step 2: Identify common error

   If Z is not defined or invalid, fcluster raises an error unrelated to parameter types.

3. Step 3: Analyze options

   The criterion 'maxclust' requires an integer number of clusters, but 2 is passed as a float is incorrect because 2 as an integer or float is accepted; The threshold parameter must be a float when using 'maxclust' is wrong because threshold can be int; The criterion 'maxclust' expects the threshold to be the maximum cluster distance is false because 'maxclust' uses number of clusters, not distance.

4. Final Answer:

   The linkage matrix Z is not defined or invalid -> Option A

5. Quick Check:

   Undefined Z causes error, not threshold type [OK]

Hint: Ensure linkage matrix Z is valid before calling fcluster [OK]

Common Mistakes:

• Passing float instead of int for maxclust threshold
• Misunderstanding criterion parameter
• Ignoring linkage matrix validity

5. You have a dataset with 10 points clustered hierarchically. You want exactly 3 clusters. Which fcluster call correctly achieves this?

hard

A. fcluster(Z, 0.5, criterion='inconsistent')

B. fcluster(Z, 3, criterion='maxclust')

C. fcluster(Z, 0.5, criterion='maxclust')

D. fcluster(Z, 3, criterion='distance')

Solution

1. Step 1: Understand criteria for exact cluster count

   To get exactly 3 clusters, use criterion='maxclust' with t=3 specifying number of clusters.

2. Step 2: Analyze options

   fcluster(Z, 3, criterion='maxclust') correctly uses maxclust with 3 clusters. fcluster(Z, 3, criterion='distance') uses distance criterion which does not guarantee exact cluster count. Options A and C use incorrect thresholds or criteria.

3. Final Answer:

   fcluster(Z, 3, criterion='maxclust') -> Option B

4. Quick Check:

   maxclust + t=number of clusters = exact clusters [OK]

Hint: Use criterion='maxclust' with t = desired cluster count [OK]

Common Mistakes:

• Using distance criterion to get exact cluster count
• Passing float threshold for maxclust
• Confusing inconsistent criterion with maxclust