Client metrics monitoring in Kafka - Commands & Configuration

Introduction

Monitoring client metrics helps you understand how your Kafka clients perform and behave. It solves the problem of not knowing if your producers or consumers are working efficiently or facing issues.

When you want to track how many messages your producer sends per second to Kafka.

When you need to check if your consumer is lagging behind the latest messages.

When you want to monitor network errors or retries happening in your Kafka clients.

When you want to see the average time your client takes to send or receive messages.

When you want to collect metrics to alert you if your Kafka client stops working properly.

Config File - client.properties

client.properties

bootstrap.servers=localhost:9092
client.id=my-kafka-client
metric.reporters=org.apache.kafka.common.metrics.JmxReporter
metrics.sample.window.ms=30000
metrics.num.samples=2

bootstrap.servers: Kafka broker addresses your client connects to.

client.id: A name to identify your client in metrics.

metric.reporters: Enables JMX reporting for metrics collection.

metrics.sample.window.ms: Time window for metrics sampling.

metrics.num.samples: Number of samples to keep for averaging metrics.

Commands

Starts a Kafka producer using the client.properties file to send messages to 'test-topic'. This enables metrics collection as configured.

Terminal

kafka-console-producer --broker-list localhost:9092 --topic test-topic --producer.config client.properties

Expected Output

No output (command runs silently)

→

--broker-list - Specifies Kafka broker addresses to connect.

→

--topic - Specifies the topic to send messages to.

→

--producer.config - Loads client configuration including metrics settings.

Opens Java JMX console to connect to the Kafka client process and view metrics exposed via JMX.

Terminal

jconsole

Expected Output

No output (command runs silently)

Shows consumer group details including lag, which is a key metric to monitor consumer health.

Terminal

kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group my-consumer-group

Expected Output

GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG  CONSUMER-ID                                    HOST            CLIENT-ID

my-consumer-group  test-topic     0          15              20              5    consumer-1-12345-67890-abcde                  /127.0.0.1      consumer-1

→

--bootstrap-server - Specifies Kafka broker addresses to connect.

→

--describe - Shows detailed information about the consumer group.

→

--group - Specifies the consumer group to describe.

Key Concept

If you remember nothing else from client metrics monitoring, remember: enabling JMX reporting in your client config lets you collect and view detailed performance metrics easily.

Common Mistakes

Not enabling metric.reporters in the client configuration.

Without metric.reporters, no metrics are exposed for monitoring.

Add 'metric.reporters=org.apache.kafka.common.metrics.JmxReporter' to your client.properties file.

Trying to view metrics without connecting to the correct Java process in JConsole.

JConsole won't show metrics if not connected to the Kafka client JVM process.

Start your Kafka client with JMX enabled and connect JConsole to that process.

Checking consumer lag without specifying the correct consumer group.

You will not see relevant lag metrics if the wrong group is queried.

Use the exact consumer group name with '--group' flag in kafka-consumer-groups command.

Summary

Configure your Kafka client with JMX metric reporting enabled in client.properties.

Run your Kafka producer or consumer with this configuration to start collecting metrics.

Use JConsole to connect to the client JVM and view detailed metrics.

Check consumer lag using kafka-consumer-groups command to monitor consumer health.