Why tuning handles production load in Kafka - See It in Action

Why tuning handles production load

📖 Scenario: You are managing a Kafka system that handles messages from many users in real time. Sometimes, the system slows down or crashes when many messages arrive at once. To keep the system fast and reliable, you need to understand how tuning Kafka settings helps handle heavy production loads.

🎯 Goal: Learn how to set up a simple Kafka producer and consumer, add a configuration for batch size, and see how tuning this setting affects message processing under load.

📋 What You'll Learn

Create a Kafka producer that sends messages

Create a Kafka consumer that receives messages

Add a configuration variable for batch size

Print the number of messages processed

💡 Why This Matters

🌍 Real World

Kafka is used in many companies to handle large streams of data in real time, such as logs, user activity, or financial transactions. Tuning Kafka settings helps keep these systems fast and reliable under heavy use.

💼 Career

Understanding Kafka tuning is important for roles like data engineers, backend developers, and system administrators who manage data pipelines and real-time processing systems.

1

Set up Kafka producer and consumer

Create a Kafka producer called producer and a Kafka consumer called consumer using the KafkaProducer and KafkaConsumer classes from the kafka library. Use the bootstrap server 'localhost:9092' and the topic 'test-topic'.

Kafka

# Your code here

Need a hint?

Use KafkaProducer(bootstrap_servers='localhost:9092') to create the producer and KafkaConsumer('test-topic', bootstrap_servers='localhost:9092') to create the consumer.

2

Add batch size configuration

Create a variable called batch_size and set it to 16384. Then, update the producer to use this batch_size by passing it as the batch_size parameter when creating the KafkaProducer.

Kafka

from kafka import KafkaProducer, KafkaConsumer

producer = KafkaProducer(bootstrap_servers='localhost:9092')
consumer = KafkaConsumer('test-topic', bootstrap_servers='localhost:9092')
# Your code here

Need a hint?

Set batch_size = 16384 and pass it to KafkaProducer(batch_size=batch_size).

3

Send and receive messages using batch size

Use a for loop with variable i in range(5) to send messages with the text f'Message {i}' using producer.send('test-topic', value=message.encode()). Then, use a for loop with variables message in consumer to receive messages and count them in a variable called count.

Kafka

from kafka import KafkaProducer, KafkaConsumer

batch_size = 16384
producer = KafkaProducer(bootstrap_servers='localhost:9092', batch_size=batch_size)
consumer = KafkaConsumer('test-topic', bootstrap_servers='localhost:9092')

# Your code here

Need a hint?

Send 5 messages in a loop, then receive messages in a loop and count them until 5 messages are received.

4

Print the number of messages processed

Write print(f"Processed {count} messages") to display how many messages were received by the consumer.

Kafka

from kafka import KafkaProducer, KafkaConsumer

batch_size = 16384
producer = KafkaProducer(bootstrap_servers='localhost:9092', batch_size=batch_size)
consumer = KafkaConsumer('test-topic', bootstrap_servers='localhost:9092')

for i in range(5):
    message = f'Message {i}'
    producer.send('test-topic', value=message.encode())
producer.flush()

count = 0
for message in consumer:
    count += 1
    if count == 5:
        break

# Your code here

Need a hint?

Use print(f"Processed {count} messages") to show the result.