Error handling in clients in Kafka - Mini Project: Build & Apply

Error handling in clients

📖 Scenario: You are building a Kafka client application that consumes messages from a topic. Sometimes, messages might cause errors during processing. You want to handle these errors gracefully to avoid crashing the client and to log the errors for later review.

🎯 Goal: Build a Kafka consumer client that reads messages from a topic, handles errors during message processing by catching exceptions, and logs error messages without stopping the client.

📋 What You'll Learn

Create a Kafka consumer subscribed to a topic named test-topic.

Add a variable called max_retries set to 3 to limit retry attempts.

Use a try-except block inside the message processing loop to catch errors.

Print the error message when an exception occurs and continue processing the next message.

💡 Why This Matters

🌍 Real World

Kafka clients often consume streams of data where some messages might be malformed or cause errors. Handling these errors without crashing is important for reliable data processing.

💼 Career

Understanding error handling in Kafka clients is useful for roles in data engineering, backend development, and real-time data processing.

1

Set up Kafka consumer

Create a Kafka consumer called consumer subscribed to the topic 'test-topic' using the KafkaConsumer class from kafka package.

Kafka

# Your code here

Hint

Use KafkaConsumer('test-topic', bootstrap_servers='localhost:9092') to create the consumer.

2

Add retry configuration

Create a variable called max_retries and set it to 3 to limit the number of retry attempts for processing a message.

Kafka

from kafka import KafkaConsumer

consumer = KafkaConsumer('test-topic', bootstrap_servers='localhost:9092')
# Your code here

Hint

Just create a variable max_retries and assign 3.

3

Add error handling in message processing

Use a for loop with variable message to iterate over consumer. Inside the loop, add a try-except block. In the try block, simulate processing by printing message.value. In the except block, catch Exception as e and print "Error processing message:" followed by e.

Kafka

from kafka import KafkaConsumer

consumer = KafkaConsumer('test-topic', bootstrap_servers='localhost:9092')
max_retries = 3
# Your code here

Hint

Use for message in consumer: and inside it a try-except block to catch errors.

4

Print final output

Add a print statement outside the loop that prints "Consumer stopped." to indicate the client has stopped.

Kafka

from kafka import KafkaConsumer

consumer = KafkaConsumer('test-topic', bootstrap_servers='localhost:9092')
max_retries = 3

for message in consumer:
    try:
        print(message.value)
    except Exception as e:
        print("Error processing message:", e)
# Your code here

Hint

Just add print("Consumer stopped.") after the loop.

Practice

1. What is the main purpose of error handling in Kafka clients?

easy

A. To increase the speed of message processing

B. To keep the client stable and responsive during failures

C. To reduce the size of messages sent

D. To automatically create Kafka topics

Solution

1. Step 1: Understand error handling role

   Error handling helps the client manage unexpected problems without crashing.

2. Step 2: Identify the main goal in Kafka clients

   Keeping the client stable and responsive ensures smooth operation even if errors occur.

3. Final Answer:

   To keep the client stable and responsive during failures -> Option B

4. Quick Check:

   Error handling = stability and responsiveness [OK]

Hint: Error handling keeps your client stable and responsive [OK]

Common Mistakes:

• Thinking error handling speeds up processing
• Confusing error handling with message size control
• Assuming error handling creates topics automatically

2. Which syntax correctly wraps a Kafka producer send call to catch exceptions in Java?

easy

A. producer.send(record).catch(e => console.log(e));

B. if (producer.send(record)) { } else { handleError(); }

C. try { producer.send(record); } catch (Exception e) { e.printStackTrace(); }

D. try catch producer.send(record);

Solution

1. Step 1: Identify correct Java try-catch syntax

   Java requires try { ... } catch (Exception e) { ... } to handle exceptions.

2. Step 2: Match syntax with Kafka producer send call

   try { producer.send(record); } catch (Exception e) { e.printStackTrace(); } correctly wraps the send call in try-catch block.

3. Final Answer:

   try { producer.send(record); } catch (Exception e) { e.printStackTrace(); } -> Option C

4. Quick Check:

   Java try-catch syntax = try { producer.send(record); } catch (Exception e) { e.printStackTrace(); } [OK]

Hint: Use try-catch blocks in Java to catch Kafka errors [OK]

Common Mistakes:

• Using if-else instead of try-catch for exceptions
• Using JavaScript syntax in Java code
• Missing braces in try-catch block

3. Given this Java Kafka consumer code snippet, what will be printed if a network error occurs during poll?

try {
  ConsumerRecords records = consumer.poll(Duration.ofMillis(100));
  for (var record : records) {
    System.out.println(record.value());
  }
} catch (WakeupException e) {
  System.out.println("Consumer interrupted");
} catch (Exception e) {
  System.out.println("Error during poll");
}

medium

A. Error during poll

B. Consumer interrupted

C. No output

D. Exception stack trace

Solution

1. Step 1: Identify exception types caught

   WakeupException prints "Consumer interrupted"; all other exceptions print "Error during poll".

2. Step 2: Determine exception for network error

   Network errors throw general exceptions, not WakeupException.

3. Final Answer:

   Error during poll -> Option A

4. Quick Check:

   Network error triggers general Exception = Error during poll [OK]

Hint: General exceptions print "Error during poll" [OK]

Common Mistakes:

• Confusing WakeupException with network errors
• Expecting no output on error
• Assuming stack trace prints automatically

4. Identify the error in this Kafka producer error handling code snippet:

try {
  producer.send(record);
} catch (KafkaException e) {
  System.out.println("Error sending message");
}
producer.close();

medium

A. producer.send() does not throw exceptions

B. Catch block should catch Exception, not KafkaException

C. Missing return statement after catch

D. producer.close() should be inside finally block

Solution

1. Step 1: Check resource cleanup

   producer.close() must run regardless of exceptions to free resources.

2. Step 2: Use finally block for cleanup

   Placing close() inside finally ensures it always executes.

3. Final Answer:

   producer.close() should be inside finally block -> Option D

4. Quick Check:

   Always close resources in finally block [OK]

Hint: Put cleanup code in finally block to always run [OK]

Common Mistakes:

• Closing producer only on success
• Catching too broad or wrong exceptions
• Assuming send() never throws exceptions

5. You want to build a Kafka consumer that retries polling up to 3 times on recoverable errors, then logs and stops. Which approach correctly implements this error handling?

hard

A. Use a loop with try-catch around poll; on exception increment retry count; break after 3 retries; finally close consumer

B. Call consumer.poll() once; if exception, immediately close consumer without retry

C. Wrap poll in try-catch but ignore exceptions and continue polling forever

D. Use multiple nested try-catch blocks without retry logic

Solution

1. Step 1: Understand retry logic requirements

   Retry polling up to 3 times on recoverable errors, then stop and log.

2. Step 2: Identify correct control flow

   Loop with try-catch, count retries, break after 3 attempts, and close consumer in finally block is correct.

3. Final Answer:

   Use a loop with try-catch around poll; on exception increment retry count; break after 3 retries; finally close consumer -> Option A

4. Quick Check:

   Retry with loop and try-catch = Use a loop with try-catch around poll; on exception increment retry count; break after 3 retries; finally close consumer [OK]

Hint: Use loop with try-catch and retry count for controlled retries [OK]

Common Mistakes:

• Not retrying on recoverable errors
• Ignoring exceptions and continuing endlessly
• Closing consumer too early without retries