Consumer throughput optimization in Kafka - Mini Project: Build & Apply

Consumer throughput optimization

📖 Scenario: You are working with a Kafka consumer that reads messages from a topic. To improve the speed of processing messages, you want to optimize the consumer throughput by adjusting the batch size and processing messages in batches.

🎯 Goal: Build a Kafka consumer that reads messages in batches of a fixed size and processes them together to improve throughput.

📋 What You'll Learn

Create a list of messages simulating Kafka consumer records

Set a batch size configuration variable

Use a loop to process messages in batches of the given batch size

Print each batch of messages to show the grouped processing

💡 Why This Matters

🌍 Real World

Kafka consumers often process messages in batches to improve throughput and reduce overhead. This project simulates that by grouping messages into batches before processing.

💼 Career

Understanding how to optimize Kafka consumer throughput is important for backend developers and data engineers working with real-time data pipelines and streaming applications.

1

DATA SETUP: Create a list of messages

Create a list called messages with these exact string entries: 'msg1', 'msg2', 'msg3', 'msg4', 'msg5', 'msg6', 'msg7', 'msg8', 'msg9', 'msg10'.

Kafka

# Create the list of messages
# Your code here

Hint

Use square brackets [] to create a list and separate each message with commas.

2

CONFIGURATION: Set the batch size

Create a variable called batch_size and set it to the integer 3.

Kafka

messages = ['msg1', 'msg2', 'msg3', 'msg4', 'msg5', 'msg6', 'msg7', 'msg8', 'msg9', 'msg10']
# Set the batch size to 3
# Your code here

Hint

Use a simple assignment statement to set batch_size to 3.

3

CORE LOGIC: Process messages in batches

Use a for loop with variable i and range(0, len(messages), batch_size) to iterate over messages in steps of batch_size. Inside the loop, create a variable batch that slices messages from i to i + batch_size.

Kafka

messages = ['msg1', 'msg2', 'msg3', 'msg4', 'msg5', 'msg6', 'msg7', 'msg8', 'msg9', 'msg10']
batch_size = 3
# Loop over messages in batches and create batch slices
# Your code here

Hint

Use slicing messages[i:i + batch_size] to get each batch.

4

OUTPUT: Print each batch of messages

Inside the for loop, add a print(batch) statement to display each batch of messages.

Kafka

messages = ['msg1', 'msg2', 'msg3', 'msg4', 'msg5', 'msg6', 'msg7', 'msg8', 'msg9', 'msg10']
batch_size = 3
for i in range(0, len(messages), batch_size):
    batch = messages[i:i + batch_size]
    # Print the current batch
    # Your code here

Hint

Use print(batch) inside the loop to show each group of messages.

Practice

1. What is the primary purpose of increasing max.poll.records in a Kafka consumer configuration?

easy

A. To decrease the consumer's memory usage

B. To increase the number of records fetched in one poll, improving throughput

C. To reduce the number of partitions assigned to the consumer

D. To increase the consumer's connection timeout

Solution

1. Step 1: Understand max.poll.records role

   This setting controls how many records the consumer fetches in a single poll call.

2. Step 2: Effect on throughput

   Increasing this value allows the consumer to process more records per poll, which can improve throughput by reducing overhead.

3. Final Answer:

   To increase the number of records fetched in one poll, improving throughput -> Option B

4. Quick Check:

   max.poll.records controls batch size [OK]

Hint: More records per poll means higher throughput [OK]

Common Mistakes:

• Confusing max.poll.records with timeout settings
• Thinking it reduces memory usage
• Assuming it changes partition assignment

2. Which of the following is the correct way to set fetch.min.bytes to 50000 in a Kafka consumer properties file?

easy

A. fetch.min.bytes=50000

B. fetch.min.bytes: 50000

C. "fetch.min.bytes" = 50000

D. fetch_min_bytes=50000

Solution

1. Step 1: Identify Kafka properties file syntax

   Kafka consumer properties use key=value pairs without quotes or underscores.

2. Step 2: Match correct syntax

   fetch.min.bytes=50000 uses the correct key with dot notation and equals sign, matching Kafka config style.

3. Final Answer:

   fetch.min.bytes=50000 -> Option A

4. Quick Check:

   Kafka config uses key=value format [OK]

Hint: Use dot notation and equals sign for Kafka configs [OK]

Common Mistakes:

• Using colon instead of equals sign
• Adding quotes around keys
• Replacing dots with underscores

3. Given the following Kafka consumer configuration snippet:

props.put("max.poll.records", "1000");
props.put("fetch.min.bytes", "1048576");

What is the expected effect on consumer throughput and latency?

medium

A. Lower throughput and lower latency

B. Higher throughput and lower latency

C. No change in throughput or latency

D. Higher throughput with potentially increased latency

Solution

1. Step 1: Analyze max.poll.records setting

   Setting it to 1000 means the consumer fetches up to 1000 records per poll, increasing batch size and throughput.

2. Step 2: Analyze fetch.min.bytes setting

   Setting it to 1MB (1048576 bytes) means the consumer waits to fetch at least 1MB of data, which can increase latency waiting for enough data.

3. Final Answer:

   Higher throughput with potentially increased latency -> Option D

4. Quick Check:

   Large batch size = higher throughput, bigger fetch.min.bytes = higher latency [OK]

Hint: Large batch size boosts throughput but may add latency [OK]

Common Mistakes:

• Assuming latency always decreases with throughput
• Ignoring fetch.min.bytes impact
• Confusing max.poll.records with fetch.max.bytes

4. You notice your Kafka consumer is lagging and not processing messages fast enough. Which of the following configuration mistakes could cause this issue?

medium

A. Setting max.poll.records too low, like 1

B. Setting fetch.min.bytes to 1

C. Setting max.poll.interval.ms to a very high value

D. Setting enable.auto.commit to true

Solution

1. Step 1: Understand impact of low max.poll.records

   Setting it to 1 means the consumer fetches only one record per poll, severely limiting throughput and causing lag.

2. Step 2: Evaluate other options

   Setting fetch.min.bytes to 1 reduces wait time, high max.poll.interval.ms delays rebalance but doesn't reduce throughput, and auto commit true is normal behavior.

3. Final Answer:

   Setting max.poll.records too low, like 1 -> Option A

4. Quick Check:

   Too small batch size causes lag [OK]

Hint: Too small max.poll.records limits throughput [OK]

Common Mistakes:

• Confusing fetch.min.bytes effect
• Thinking auto commit affects speed directly
• Ignoring batch size impact

5. You want to optimize a Kafka consumer for high throughput but low latency. Which combination of settings is best?

hard

A. Set both max.poll.records and fetch.min.bytes to high values

B. Set max.poll.records low (e.g., 10) and fetch.min.bytes high (e.g., 1048576)

C. Set max.poll.records high (e.g., 1000) and fetch.min.bytes low (e.g., 1)

D. Set both max.poll.records and fetch.min.bytes to low values

Solution

1. Step 1: Understand throughput and latency tradeoff

   High max.poll.records allows processing many records per poll, increasing throughput.

2. Step 2: Understand fetch.min.bytes effect on latency

   Low fetch.min.bytes means the consumer fetches data quickly without waiting for large batches, reducing latency.

3. Step 3: Combine settings for desired effect

   High batch size with low fetch size balances high throughput and low latency.

4. Final Answer:

   Set max.poll.records high (e.g., 1000) and fetch.min.bytes low (e.g., 1) -> Option C

5. Quick Check:

   High batch + low fetch.min.bytes = throughput + low latency [OK]

Hint: High max.poll.records + low fetch.min.bytes = best balance [OK]

Common Mistakes:

• Setting both values high causes latency spikes
• Setting both low reduces throughput
• Ignoring fetch.min.bytes impact on latency