Controller broker in Kafka - Step-by-Step Execution

Process Flow - Controller broker

Start Kafka Cluster

↓

Elect Controller Broker

↓

Controller Broker Manages Cluster

↓

Monitor Broker States

↓

Handle Partition Leadership Changes

↓

Update Metadata for Brokers

↓

Detect Broker Failures

↓

Trigger Reassignments

↓

Continue Managing Until Controller Changes

The controller broker is elected when the Kafka cluster starts and manages cluster metadata, broker states, and partition leadership until a new controller is elected.

Execution Sample

Kafka

1. Start Kafka cluster
2. Elect controller broker
3. Controller monitors brokers
4. Handle leader changes
5. Update metadata
6. Detect failures and reassign

This sequence shows how the controller broker manages cluster state and broker leadership dynamically.

Process Table

Step	Action	Controller State	Cluster Metadata	Output
1	Start Kafka cluster	No controller	Empty	Cluster starts
2	Elect controller broker	Controller elected (Broker 1)	Empty	Controller ready
3	Controller monitors brokers	Active	Metadata updated with brokers	Monitoring brokers
4	Handle leader changes	Active	Partition leaders updated	Leaders assigned
5	Update metadata	Active	Metadata refreshed	Metadata broadcasted
6	Detect broker failure	Active	Metadata marks broker down	Trigger reassignment
7	Trigger reassignment	Active	Partition leadership reassigned	Cluster stable
8	Controller continues managing	Active	Metadata current	Waiting for events
9	Controller failure	Controller lost	Metadata frozen	New controller election triggered
10	New controller elected	Controller elected (Broker 2)	Metadata refreshed	Cluster managed again

💡 Controller failure triggers new election, restarting management cycle

Status Tracker

Variable	Start	After Step 2	After Step 5	After Step 7	After Step 9	After Step 10
Controller State	No controller	Controller elected (Broker 1)	Active	Active	Controller lost	Controller elected (Broker 2)
Cluster Metadata	Empty	Empty	Metadata refreshed	Partition leadership reassigned	Metadata frozen	Metadata refreshed
Broker Status	Unknown	Known	Known	Broker 3 down	Broker 3 down	Known

Key Moments - 3 Insights

Why does the controller broker change during cluster operation?

When the current controller fails or loses connection (see step 9 in execution_table), a new controller is elected to maintain cluster management.

How does the controller broker handle broker failures?

It detects failures (step 6), updates metadata to mark brokers down, and triggers partition leadership reassignment (step 7) to keep the cluster stable.

What happens to cluster metadata when the controller changes?

Metadata is frozen during controller loss (step 9) and refreshed by the new controller after election (step 10) to ensure accurate cluster state.

Visual Quiz - 3 Questions

Look at the execution_table, what is the controller state after step 5?

A No controller

B Controller elected (Broker 1)

C Active

D Controller lost

💡 Hint

Check the 'Controller State' column at step 5 in execution_table

At which step does the controller detect a broker failure?

A Step 4

B Step 6

C Step 8

D Step 10

💡 Hint

Look for 'Detect broker failure' action in execution_table

If the controller never fails, which step would NOT occur?

A Step 9: Controller failure

B Step 7: Trigger reassignment

C Step 4: Handle leader changes

D Step 5: Update metadata

💡 Hint

Refer to execution_table steps related to controller failure and election

Concept Snapshot

Controller broker manages Kafka cluster metadata and broker states.
It is elected at cluster start and handles leader changes and failures.
Detects broker failures and triggers partition reassignment.
If controller fails, a new one is elected to continue management.
Ensures cluster stability and metadata consistency.

Full Transcript

The controller broker in Kafka is elected when the cluster starts. It manages cluster metadata, monitors broker states, handles partition leadership changes, updates metadata, detects broker failures, and triggers reassignment of partitions. If the controller fails, a new controller is elected to maintain cluster stability. This process ensures the Kafka cluster runs smoothly by coordinating brokers and partition leadership dynamically.