Key broker metrics in Kafka - Commands & Configuration

Introduction

Kafka brokers handle messages between producers and consumers. Monitoring key broker metrics helps keep the system healthy and fast by showing how well brokers are working and if there are any problems.

When you want to check if your Kafka broker is handling messages without delays

When you need to find out if brokers are overloaded or under heavy load

When you want to monitor network traffic between brokers and clients

When you want to track message processing rates to ensure smooth data flow

When you want to detect errors or failures in broker operations early

Commands

This command connects to the Kafka broker's JMX interface to get the current rate of incoming messages per second. It helps you see how many messages the broker is receiving.

Terminal

kafka-run-class.sh kafka.tools.JmxTool --jmx-url service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi --object-name kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec

Expected Output

kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec Count=12345.0 OneMinuteRate=150.0 FiveMinuteRate=140.0 FifteenMinuteRate=130.0 MeanRate=145.0

→

--jmx-url - Specifies the JMX connection URL to the Kafka broker

→

--object-name - Specifies the exact metric to query from the broker

This command fetches the rate of Produce requests per second handled by the broker. It shows how many produce operations the broker processes.

Terminal

kafka-run-class.sh kafka.tools.JmxTool --jmx-url service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi --object-name kafka.network:type=RequestMetrics,name=RequestsPerSec,request=Produce

Expected Output

kafka.network:type=RequestMetrics,name=RequestsPerSec,request=Produce Count=6789.0 OneMinuteRate=80.0 FiveMinuteRate=75.0 FifteenMinuteRate=70.0 MeanRate=77.0

→

--jmx-url - Connects to the broker's JMX interface

→

--object-name - Specifies the Produce request metric

This command checks how many partitions are under-replicated, meaning they don't have enough copies for safety. It helps detect risks to data durability.

Terminal

kafka-run-class.sh kafka.tools.JmxTool --jmx-url service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi --object-name kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions

Expected Output

kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions Value=0

→

--jmx-url - Connects to the broker's JMX interface

→

--object-name - Checks under-replicated partitions metric

This command shows the rate of failed requests per second on the broker. It helps identify if the broker is having trouble processing requests.

Terminal

kafka-run-class.sh kafka.tools.JmxTool --jmx-url service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi --object-name kafka.network:type=RequestMetrics,name=FailedRequestsPerSec

Expected Output

kafka.network:type=RequestMetrics,name=FailedRequestsPerSec Count=2.0 OneMinuteRate=0.01 FiveMinuteRate=0.02 FifteenMinuteRate=0.01 MeanRate=0.015

→

--jmx-url - Connects to the broker's JMX interface

→

--object-name - Fetches failed requests metric

Key Concept

If you remember nothing else from this pattern, remember: monitoring key Kafka broker metrics through JMX helps you keep your messaging system healthy and responsive.

Common Mistakes

Trying to run JMX commands without enabling JMX on the Kafka broker

The commands fail because the broker is not exposing metrics for JMX to connect

Start the Kafka broker with JMX enabled by setting environment variable KAFKA_JMX_OPTS before running the broker

Using wrong JMX URL or port when running kafka-run-class.sh JmxTool

The tool cannot connect to the broker's JMX interface and returns connection errors

Verify the JMX port and URL configured on the broker and use the exact same in the command

Querying incorrect or misspelled metric names in --object-name

The tool returns no data or errors because the metric does not exist

Use exact metric names as documented in Kafka metrics reference

Summary

Use kafka-run-class.sh with JmxTool to query Kafka broker metrics via JMX.

Check metrics like MessagesInPerSec, RequestsPerSec, UnderReplicatedPartitions, and FailedRequestsPerSec to monitor broker health.

Ensure JMX is enabled and the correct JMX URL and metric names are used for successful queries.

Practice

1.

What does the MessagesInPerSec metric in Kafka broker monitoring indicate?

easy

A. The number of messages the broker receives per second

B. The number of active connections to the broker

C. The total disk space used by the broker

D. The number of partitions in the broker

Solution

1. Step 1: Understand the metric name

   The metric MessagesInPerSec literally means messages coming in per second to the broker.

2. Step 2: Match the metric to its meaning

   This metric counts how many messages the broker receives every second, showing its input load.

3. Final Answer:

   The number of messages the broker receives per second -> Option A

4. Quick Check:

   MessagesInPerSec = messages received per second [OK]

Hint: Look for 'In' to find incoming message rate [OK]

Common Mistakes:

• Confusing message count with connection count
• Thinking it measures disk usage
• Mixing it up with partition count

2.

Which of the following is the correct JMX metric name to monitor the broker's outgoing bytes per second?

easy

A. MessagesOutPerSec

B. BytesInPerSec

C. BytesOutPerSec

D. BytesTotalPerSec

Solution

1. Step 1: Identify the direction of data flow

   Outgoing bytes means data sent out from the broker, so the metric should say 'Out'.

2. Step 2: Match the exact metric name

   The correct metric for outgoing bytes per second is BytesOutPerSec.

3. Final Answer:

   BytesOutPerSec -> Option C

4. Quick Check:

   Outgoing bytes = BytesOutPerSec [OK]

Hint: Look for 'Out' and 'Bytes' in metric name [OK]

Common Mistakes:

• Choosing BytesInPerSec which is incoming data
• Confusing messages with bytes
• Using a non-existent metric name

3.

Given the following JMX metric values for a Kafka broker:
MessagesInPerSec = 500
BytesInPerSec = 1024000
BytesOutPerSec = 2048000
What does this tell you about the broker's activity?

medium

A. The broker's incoming and outgoing bytes are equal

B. The broker is receiving more messages than it sends out

C. The broker is idle with no message traffic

D. The broker is sending out twice as much data as it receives in messages

Solution

1. Step 1: Analyze the metric values

   MessagesInPerSec is 500, BytesInPerSec is 1,024,000 bytes, BytesOutPerSec is 2,048,000 bytes.

2. Step 2: Compare incoming and outgoing bytes

   Outgoing bytes (2,048,000) is exactly twice the incoming bytes (1,024,000), meaning the broker sends out twice as much data as it receives.

3. Final Answer:

   The broker is sending out twice as much data as it receives in messages -> Option D

4. Quick Check:

   BytesOutPerSec = 2 x BytesInPerSec [OK]

Hint: Compare BytesOutPerSec and BytesInPerSec values [OK]

Common Mistakes:

• Ignoring the difference between bytes and messages
• Assuming equal incoming and outgoing bytes
• Misreading the numeric values

4.

Consider this snippet monitoring Kafka broker metrics in Java:

double bytesIn = jmxClient.getAttribute("kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec");
double bytesOut = jmxClient.getAttribute("kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec");
if (bytesIn > bytesOut) {
    System.out.println("More incoming data");
} else {
    System.out.println("More outgoing data");
}

What is the likely bug in this code?

medium

A. The code does not handle the case when bytesIn equals bytesOut

B. The comparison should use >= to handle equal values

C. The metric names are incorrect and will cause an error

D. The JMX attribute values should be cast to int, not double

Solution

1. Step 1: Review the if-else logic

   The code checks if bytesIn is greater than bytesOut, else prints "More outgoing data".

2. Step 2: Identify missing case handling

   If bytesIn equals bytesOut, the else block runs, printing "More outgoing data" which is misleading.

3. Final Answer:

   The code does not handle the case when bytesIn equals bytesOut -> Option A

4. Quick Check:

   Equal values need explicit handling [OK]

Hint: Check if equal values are handled in if-else [OK]

Common Mistakes:

• Assuming metric names are wrong
• Ignoring equal value scenario
• Casting to wrong data type without error

5.

You want to alert if the Kafka broker's RequestHandlerAvgIdlePercent metric drops below 20%, indicating high load. Which approach correctly implements this check in pseudocode?

1. Fetch RequestHandlerAvgIdlePercent metric value
2. If value < 20 then alert 'High broker load'

hard

A. if (RequestHandlerAvgIdlePercent > 20) { alert('High broker load'); }

B. if (RequestHandlerAvgIdlePercent < 20) { alert('High broker load'); }

C. if (RequestHandlerAvgIdlePercent == 20) { alert('High broker load'); }

D. if (RequestHandlerAvgIdlePercent != 20) { alert('High broker load'); }

Solution

1. Step 1: Understand the metric meaning

   RequestHandlerAvgIdlePercent shows how idle the request handler threads are; lower means busier.

2. Step 2: Apply the alert condition

   If this value is less than 20%, it means the broker is busy, so alerting on < 20 is correct.

3. Final Answer:

   if (RequestHandlerAvgIdlePercent < 20) { alert('High broker load'); } -> Option B

4. Quick Check:

   Idle percent < 20 means high load [OK]

Hint: Alert when idle percent is less than threshold [OK]

Common Mistakes:

• Using greater than instead of less than
• Checking equality only
• Using not equal which triggers too often