Why tuning handles production load in Kafka - Why It Works

Introduction

When many users or systems send data to Kafka at the same time, it can slow down or crash if not set up right. Tuning Kafka means adjusting settings so it can handle lots of data smoothly without delays or errors.

When your Kafka cluster slows down during peak business hours and messages pile up.

When you notice consumers lagging behind producers and data is delayed.

When you want to avoid losing messages during sudden spikes in traffic.

When your Kafka brokers use too much CPU or memory under load.

When you want to improve throughput and reduce latency for real-time data processing.

Config File - server.properties

server.properties

broker.id=1
listeners=PLAINTEXT://:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs
num.partitions=3
default.replication.factor=2
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=localhost:2181
replica.fetch.max.bytes=1048576
message.max.bytes=1000012
fetch.max.bytes=52428800
replica.lag.time.max.ms=10000
replica.lag.max.messages=4000

broker.id: Unique ID for each Kafka broker.

listeners: Network address Kafka listens on.

num.network.threads and num.io.threads: Threads for handling network and disk I/O.

socket.send.buffer.bytes and socket.receive.buffer.bytes: Network buffer sizes to improve data flow.

num.partitions: Number of partitions per topic to allow parallel processing.

default.replication.factor: Number of copies of data for fault tolerance.

log.retention.hours: How long logs are kept before deletion.

replica.fetch.max.bytes: Max data size fetched by replicas to keep them in sync.

These settings help Kafka handle more data efficiently and stay reliable under heavy use.

Commands

Starts the Kafka broker using the tuned configuration file to handle production load.

Terminal

kafka-server-start.sh /usr/local/kafka/config/server.properties

Expected Output

[2024-06-01 10:00:00,000] INFO Kafka version : 3.5.0 (org.apache.kafka.common.utils.AppInfoParser)

[2024-06-01 10:00:00,001] INFO Kafka startTimeMs : 1685604000000 (org.apache.kafka.common.utils.AppInfoParser)

[2024-06-01 10:00:01,500] INFO [KafkaServer id=1] started (kafka.server.KafkaServer)

Creates a topic with multiple partitions and replication to distribute load and increase fault tolerance.

Terminal

kafka-topics.sh --create --topic example-topic --partitions 3 --replication-factor 2 --bootstrap-server localhost:9092

Expected Output

Created topic example-topic.

→

--partitions - Sets number of partitions for parallelism

→

--replication-factor - Sets number of copies for reliability

Checks consumer lag to see if consumers keep up with the production load.

Terminal

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group example-group

Expected Output

GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG  CONSUMER-ID  HOST  CLIENT-ID

example-group   example-topic   0          1000            1000            0    consumer-1   /127.0.0.1  consumer-1

example-group   example-topic   1          1000            1000            0    consumer-1   /127.0.0.1  consumer-1

example-group   example-topic   2          1000            1000            0    consumer-1   /127.0.0.1  consumer-1

→

--describe - Shows detailed consumer group info

Key Concept

If you remember nothing else from this pattern, remember: tuning Kafka’s configuration lets it handle many messages smoothly without delays or crashes.

Common Mistakes

Using only one partition for a high-traffic topic

Limits parallel processing and causes bottlenecks under load

Create topics with multiple partitions to spread the load

Setting replication factor to 1 in production

No data copies means risk of data loss if a broker fails

Set replication factor to at least 2 for fault tolerance

Not monitoring consumer lag regularly

Consumers falling behind causes delayed processing and stale data

Use kafka-consumer-groups.sh to check lag and adjust consumers

Summary

Start Kafka broker with tuned server.properties to optimize resource use.

Create topics with multiple partitions and replication for load distribution and reliability.

Monitor consumer lag to ensure consumers keep up with production load.

Practice

1. Why is tuning Kafka settings important for handling production load?

easy

A. It automatically fixes all network issues.

B. It changes the Kafka source code to add new features.

C. It reduces the number of Kafka brokers needed.

D. It helps Kafka process messages more efficiently under heavy load.

Solution

1. Step 1: Understand Kafka's role in message processing

   Kafka handles streams of messages, so efficient processing is key to avoid delays or failures.

2. Step 2: Recognize tuning impact on performance

   Tuning settings like batch size and timeouts helps Kafka manage more messages smoothly during high load.

3. Final Answer:

   It helps Kafka process messages more efficiently under heavy load. -> Option D

4. Quick Check:

   Tuning improves message handling = D [OK]

Hint: Tuning improves performance under load, not code or hardware [OK]

Common Mistakes:

• Thinking tuning changes Kafka code
• Assuming tuning reduces brokers automatically
• Believing tuning fixes network problems alone

2. Which Kafka configuration setting directly affects how many messages are sent in one batch?

easy

A. batch.size

B. max.poll.records

C. auto.offset.reset

D. log.retention.hours

Solution

1. Step 1: Identify batch-related settings

   Among the options, batch.size controls the number of bytes sent in one batch.

2. Step 2: Confirm other options' roles

   max.poll.records limits records fetched per poll, auto.offset.reset controls offset behavior, and log.retention.hours sets log retention time.

3. Final Answer:

   batch.size -> Option A

4. Quick Check:

   Batch size controls message batch = B [OK]

Hint: Batch size controls message grouping, not offsets or retention [OK]

Common Mistakes:

• Confusing batch.size with max.poll.records
• Mixing offset reset with batch size
• Thinking retention affects batch size

3. Given this Kafka producer config snippet:

{"batch.size": 16384, "linger.ms": 5}

What is the effect of setting linger.ms to 5?

medium

A. Kafka delays sending messages by 5 seconds.

B. Kafka waits up to 5 ms to batch more messages before sending.

C. Kafka sends messages immediately without waiting.

D. Kafka ignores batch size and sends single messages.

Solution

1. Step 1: Understand linger.ms meaning

   linger.ms sets how long Kafka waits to batch messages before sending.

2. Step 2: Interpret the value 5

   Setting it to 5 means Kafka waits up to 5 milliseconds to collect more messages for batching.

3. Final Answer:

   Kafka waits up to 5 ms to batch more messages before sending. -> Option B

4. Quick Check:

   linger.ms controls wait time = C [OK]

Hint: linger.ms sets wait time in milliseconds before sending batch [OK]

Common Mistakes:

• Thinking linger.ms is seconds, not milliseconds
• Assuming immediate send ignores linger.ms
• Confusing linger.ms with batch.size

4. You set num.network.threads to 1 on a Kafka broker but notice slow message processing under load. What is the likely issue?

medium

A. The batch size is too large.

B. Too many network threads cause CPU overload.

C. Too few network threads limit concurrent connections.

D. The retention period is too short.

Solution

1. Step 1: Understand num.network.threads role

   This setting controls how many threads handle network requests concurrently.

2. Step 2: Analyze effect of setting it to 1

   With only 1 thread, Kafka cannot handle many connections well, causing slow processing under load.

3. Final Answer:

   Too few network threads limit concurrent connections. -> Option C

4. Quick Check:

   Low network threads cause slow processing = A [OK]

Hint: Low network threads limit concurrency, causing slow processing [OK]

Common Mistakes:

• Blaming batch size or retention instead of threads
• Assuming more threads always cause overload
• Ignoring network thread impact on load

5. You want to improve Kafka producer throughput for a high-volume system. Which combination of tuning changes is best?

hard

A. Increase batch.size, increase linger.ms, and increase num.io.threads

B. Decrease batch.size, decrease linger.ms, and decrease num.io.threads

C. Increase batch.size, decrease linger.ms, and decrease num.io.threads

D. Decrease batch.size, increase linger.ms, and increase num.io.threads

Solution

1. Step 1: Understand batch.size and linger.ms effects

   Larger batch.size and higher linger.ms allow Kafka to batch more messages, improving throughput.

2. Step 2: Consider num.io.threads impact

   Increasing num.io.threads allows more parallel I/O operations, helping handle higher load.

3. Step 3: Evaluate options

   Increase batch.size, increase linger.ms, and increase num.io.threads increases all three to improve throughput; others reduce some settings, which lowers throughput.

4. Final Answer:

   Increase batch.size, increase linger.ms, and increase num.io.threads -> Option A

5. Quick Check:

   Increase batch, linger, and I/O threads = A [OK]

Hint: Increase batch size, linger time, and I/O threads for throughput [OK]

Common Mistakes:

• Reducing batch size or linger.ms when throughput is needed
• Decreasing I/O threads limits parallelism
• Mixing increase and decrease incorrectly