Why consumer groups enable parallel processing in Kafka - See It in Action

Why consumer groups enable parallel processing

📖 Scenario: Imagine you run a busy online store. Many orders come in every second. You want to process these orders quickly and smoothly without making customers wait.

🎯 Goal: You will learn how Kafka consumer groups help process many messages at the same time, making your order system faster and more efficient.

📋 What You'll Learn

Create a list of order messages

Set up a consumer group size variable

Assign orders to consumers using parallel processing logic

Print which consumer processes which orders

💡 Why This Matters

🌍 Real World

In real online stores or apps, many messages or tasks come in fast. Using consumer groups lets multiple workers handle these tasks at the same time, speeding up processing.

💼 Career

Understanding consumer groups and parallel processing is key for jobs in data engineering, backend development, and system design where Kafka or similar tools are used.

1

Create the list of orders

Create a list called orders with these exact strings: 'order1', 'order2', 'order3', 'order4', 'order5'.

Kafka

# Create the list of orders
# Your code here

Need a hint?

Use square brackets [] to make a list and put the order names inside quotes.

2

Set the consumer group size

Create a variable called consumer_group_size and set it to 2.

Kafka

orders = ['order1', 'order2', 'order3', 'order4', 'order5']
# Set the consumer group size to 2
# Your code here

Need a hint?

Just write the variable name, an equals sign, and the number 2.

3

Assign orders to consumers in parallel

Create a dictionary called assignments that assigns each order to a consumer number from 0 up to consumer_group_size - 1. Use a for loop with enumerate(orders) and assign consumers using index % consumer_group_size.

Kafka

orders = ['order1', 'order2', 'order3', 'order4', 'order5']
consumer_group_size = 2
# Assign each order to a consumer using a for loop
# Your code here

Need a hint?

Use enumerate to get the order position and use modulo % to cycle through consumers.

4

Print the consumer assignments

Use a for loop with order, consumer in assignments.items() to print lines like Order order1 is processed by consumer 0 for each order.

Kafka

orders = ['order1', 'order2', 'order3', 'order4', 'order5']
consumer_group_size = 2
assignments = {}
for index, order in enumerate(orders):
    consumer = index % consumer_group_size
    assignments[order] = consumer
# Print which consumer processes each order
# Your code here

Need a hint?

Use a for loop over assignments.items() and print with an f-string.