Error handling in clients in Kafka - Commands & Configuration

Introduction

When clients communicate with Kafka, errors can happen like network issues or wrong data. Handling these errors properly helps keep your app running smoothly and avoids losing messages.

When your app sends messages to Kafka and you want to retry if sending fails

When your app reads messages and needs to skip or log bad data without crashing

When network problems cause temporary disconnects and you want to reconnect automatically

When you want to log errors for later debugging without stopping your app

When you want to handle specific Kafka errors differently, like authorization failures

Commands

Starts a simple Kafka producer to send messages to the topic 'example-topic'. This helps test sending messages and observe error handling.

Terminal

kafka-console-producer --broker-list localhost:9092 --topic example-topic

Expected Output

No output (command runs silently)

→

--broker-list - Specifies the Kafka broker addresses to connect to

→

--topic - Specifies the topic to send messages to

Starts a simple Kafka consumer to read messages from 'example-topic' from the start. Useful to see messages and test error handling when consuming.

Terminal

kafka-console-consumer --bootstrap-server localhost:9092 --topic example-topic --from-beginning

Expected Output

No output (command runs silently)

→

--bootstrap-server - Specifies the Kafka broker addresses to connect to

→

--topic - Specifies the topic to read messages from

→

--from-beginning - Reads all messages from the start of the topic

Runs a Java Kafka producer client that includes error handling code to retry sending messages on failure.

Terminal

java -cp kafka-clients.jar:. ExampleKafkaProducer

Expected Output

Message sent successfully

Retrying to send message due to error: NetworkException

Message sent successfully

Runs a Java Kafka consumer client that handles errors like deserialization failures by logging and skipping bad messages.

Terminal

java -cp kafka-clients.jar:. ExampleKafkaConsumer

Expected Output

Consumed message: Hello

Error deserializing message, skipping

Consumed message: World

Key Concept

If you remember nothing else from this pattern, remember: always catch and handle errors in Kafka clients to keep your app stable and avoid losing messages.

Common Mistakes

Not catching exceptions in the Kafka client code

Uncaught exceptions crash the app and stop message processing

Wrap send and receive calls in try-catch blocks and handle errors gracefully

Ignoring retry logic on transient errors

Temporary network glitches cause message loss without retries

Implement retries with backoff for sending messages

Failing to log errors for later debugging

Without logs, it's hard to find and fix issues

Log all errors with clear messages and context

Summary

Use kafka-console-producer and kafka-console-consumer to test sending and receiving messages.

Write client code that catches exceptions when sending or receiving messages.

Implement retry logic and error logging to handle temporary failures and bad data.