Controller broker in Kafka - Mini Project: Build & Apply

Controller Broker Status Checker with Kafka

📖 Scenario: You are working with Apache Kafka, a system that helps send messages between different parts of a program. One important part is the controller broker, which manages how messages flow and keeps everything running smoothly.In this project, you will write a simple Python program to check which broker is the controller in a Kafka cluster. This helps you understand how Kafka manages its controllers.

🎯 Goal: Build a Python program that connects to a Kafka cluster, finds the controller broker ID, and prints it out.

📋 What You'll Learn

Create a dictionary called brokers with broker IDs as keys and their hostnames as values: 1: 'broker1.example.com', 2: 'broker2.example.com', 3: 'broker3.example.com'

Create a variable called controller_id and set it to 2

Use a for loop with variables broker_id and host to iterate over brokers.items() and find the controller broker

Print the message "Controller broker is ID {controller_id} at {host}" using an f-string

💡 Why This Matters

🌍 Real World

In real Kafka clusters, knowing which broker is the controller helps in monitoring and troubleshooting message flow and cluster health.

💼 Career

Understanding controller brokers is useful for roles like DevOps engineers, system administrators, and backend developers working with distributed systems and message queues.

1

Create the brokers dictionary

Create a dictionary called brokers with these exact entries: 1: 'broker1.example.com', 2: 'broker2.example.com', 3: 'broker3.example.com'

Kafka

# Create the brokers dictionary
# Your code here

Hint

Use curly braces {} to create a dictionary with keys and values separated by colons.

2

Set the controller broker ID

Create a variable called controller_id and set it to 2

Kafka

brokers = {1: 'broker1.example.com', 2: 'broker2.example.com', 3: 'broker3.example.com'}
# Set the controller broker ID
# Your code here

Hint

Just assign the number 2 to the variable controller_id.

3

Find the controller broker host

Use a for loop with variables broker_id and host to iterate over brokers.items(). Inside the loop, check if broker_id equals controller_id. If yes, assign host to a variable called controller_host

Kafka

brokers = {1: 'broker1.example.com', 2: 'broker2.example.com', 3: 'broker3.example.com'}
controller_id = 2
# Find the controller broker host
# Your code here

Hint

Use for broker_id, host in brokers.items(): to loop. Then use if broker_id == controller_id: to check. Assign host to controller_host.

4

Print the controller broker information

Write a print statement using an f-string to display: "Controller broker is ID {controller_id} at {controller_host}"

Kafka

brokers = {1: 'broker1.example.com', 2: 'broker2.example.com', 3: 'broker3.example.com'}
controller_id = 2
for broker_id, host in brokers.items():
    if broker_id == controller_id:
        controller_host = host
# Print the controller broker info
# Your code here

Hint

Use print(f"Controller broker is ID {controller_id} at {controller_host}") to show the message.

Practice

1.

What is the main role of the controller broker in a Kafka cluster?

easy

A. Manage leader election and cluster metadata updates

B. Store all the messages permanently

C. Serve as a client to produce messages

D. Handle consumer group offsets only

Solution

1. Step 1: Understand controller broker responsibilities

   The controller broker is responsible for managing cluster-wide tasks such as leader election and metadata updates.

2. Step 2: Compare other options

   Storing messages, producing messages, and handling consumer offsets are roles of other brokers or clients, not the controller.

3. Final Answer:

   Manage leader election and cluster metadata updates -> Option A

4. Quick Check:

   Controller broker = leader election manager [OK]

Hint: Controller broker manages cluster tasks like leader election [OK]

Common Mistakes:

• Confusing controller with brokers that store data
• Thinking controller handles only consumer offsets
• Assuming controller is a client

2.

Which of the following is the correct way to identify the controller broker using Kafka's command line tool?

kafka-metadata-quorum.sh --bootstrap-server localhost:9092 status

easy

A. Find the broker with the lowest port number

B. Check the broker with the highest broker ID

C. Look for the LeaderId in the output

D. Look for the broker with the 'leader' role in the output

Solution

1. Step 1: Understand the command output

   The kafka-metadata-quorum.sh status tool shows metadata quorum details including LeaderId, which is the ID of the current controller broker.

2. Step 2: Identify controller in output

   The controller broker is identified by the LeaderId field, not by broker ID, port number, or generic leader role.

3. Final Answer:

   Look for the LeaderId in the output -> Option C

4. Quick Check:

   LeaderId = controller broker ID [OK]

Hint: LeaderId in metadata quorum status identifies controller [OK]

Common Mistakes:

• Assuming highest broker ID means controller
• Using port number to find controller
• Confusing generic leader role with LeaderId

3.

Given the following code snippet using Kafka AdminClient in Java, what will controllerId hold after execution?

AdminClient admin = AdminClient.create(props);
Node controller = admin.describeCluster().controller().get();
int controllerId = controller.id();
admin.close();

medium

A. The total number of brokers in the cluster

B. The topic partition count

C. The port number of the controller broker

D. The broker ID of the current controller broker

Solution

1. Step 1: Analyze AdminClient describeCluster method

   The describeCluster().controller() returns the current controller broker as a Node object.

2. Step 2: Understand Node.id()

   The id() method returns the broker ID of that Node, which is the controller broker's ID.

3. Final Answer:

   The broker ID of the current controller broker -> Option D

4. Quick Check:

   controller().id() = controller broker ID [OK]

Hint: controller().id() returns controller broker's ID [OK]

Common Mistakes:

• Confusing controller ID with number of brokers
• Thinking it returns port or partition count
• Ignoring the blocking get() call

4.

Consider this snippet to get the controller broker ID using Kafka AdminClient in Java:

AdminClient admin = AdminClient.create(props);
Node controller = admin.describeCluster().controller().get();
int controllerId = controller.id();
admin.close();

What is the likely cause if this code throws a java.util.concurrent.ExecutionException?

medium

A. The close() method was called too early

B. The AdminClient failed to connect to any broker

C. The id() method returns a string, not int

D. The controller() method does not exist

Solution

1. Step 1: Understand ExecutionException cause

   An ExecutionException usually wraps an error during the async call, often connection failure.

2. Step 2: Check AdminClient connectivity

   If AdminClient cannot connect to any broker, the get() call on controller() future will fail with this exception.

3. Final Answer:

   The AdminClient failed to connect to any broker -> Option B

4. Quick Check:

   ExecutionException = connection failure [OK]

Hint: ExecutionException often means connection problem [OK]

Common Mistakes:

• Thinking method does not exist causes ExecutionException
• Confusing return types of id() method
• Assuming close() causes this error

5.

You want to write a Kafka monitoring tool that automatically detects when the controller broker changes and logs the new controller's broker ID. Which approach is best?

1. Use AdminClient to poll describeCluster().controller() periodically.
2. Subscribe to Kafka internal topics to listen for controller changes.
3. Manually check broker logs for controller election messages.
4. Restart all brokers to force controller re-election and detect changes.

hard

A. Use AdminClient to poll describeCluster().controller() periodically

B. Subscribe to Kafka internal topics to listen for controller changes

C. Manually check broker logs for controller election messages

D. Restart all brokers to force controller re-election and detect changes

Solution

1. Step 1: Evaluate monitoring approaches

   Polling describeCluster().controller() via AdminClient is a reliable, programmatic way to detect controller changes.

2. Step 2: Assess other options

   Subscribing to internal topics is complex and not guaranteed; checking logs is manual and slow; restarting brokers is disruptive and impractical.

3. Final Answer:

   Use AdminClient to poll describeCluster().controller() periodically -> Option A

4. Quick Check:

   Polling controller() is best for monitoring [OK]

Hint: Poll controller() with AdminClient for changes [OK]

Common Mistakes:

• Trying to read internal topics for controller changes
• Relying on manual log inspection
• Restarting brokers unnecessarily