Consumer throughput optimization in Kafka - Mini Project: Build & Apply

Consumer throughput optimization

📖 Scenario: You are working with a Kafka consumer that reads messages from a topic. To improve the speed of processing messages, you want to optimize the consumer throughput by adjusting the batch size and processing messages in batches.

🎯 Goal: Build a Kafka consumer that reads messages in batches of a fixed size and processes them together to improve throughput.

📋 What You'll Learn

Create a list of messages simulating Kafka consumer records

Set a batch size configuration variable

Use a loop to process messages in batches of the given batch size

Print each batch of messages to show the grouped processing

💡 Why This Matters

🌍 Real World

Kafka consumers often process messages in batches to improve throughput and reduce overhead. This project simulates that by grouping messages into batches before processing.

💼 Career

Understanding how to optimize Kafka consumer throughput is important for backend developers and data engineers working with real-time data pipelines and streaming applications.

1

DATA SETUP: Create a list of messages

Create a list called messages with these exact string entries: 'msg1', 'msg2', 'msg3', 'msg4', 'msg5', 'msg6', 'msg7', 'msg8', 'msg9', 'msg10'.

Kafka

# Create the list of messages
# Your code here

Need a hint?

Use square brackets [] to create a list and separate each message with commas.

2

CONFIGURATION: Set the batch size

Create a variable called batch_size and set it to the integer 3.

Kafka

messages = ['msg1', 'msg2', 'msg3', 'msg4', 'msg5', 'msg6', 'msg7', 'msg8', 'msg9', 'msg10']
# Set the batch size to 3
# Your code here

Need a hint?

Use a simple assignment statement to set batch_size to 3.

3

CORE LOGIC: Process messages in batches

Use a for loop with variable i and range(0, len(messages), batch_size) to iterate over messages in steps of batch_size. Inside the loop, create a variable batch that slices messages from i to i + batch_size.

Kafka

messages = ['msg1', 'msg2', 'msg3', 'msg4', 'msg5', 'msg6', 'msg7', 'msg8', 'msg9', 'msg10']
batch_size = 3
# Loop over messages in batches and create batch slices
# Your code here

Need a hint?

Use slicing messages[i:i + batch_size] to get each batch.

4

OUTPUT: Print each batch of messages

Inside the for loop, add a print(batch) statement to display each batch of messages.

Kafka

messages = ['msg1', 'msg2', 'msg3', 'msg4', 'msg5', 'msg6', 'msg7', 'msg8', 'msg9', 'msg10']
batch_size = 3
for i in range(0, len(messages), batch_size):
    batch = messages[i:i + batch_size]
    # Print the current batch
    # Your code here

Need a hint?

Use print(batch) inside the loop to show each group of messages.