Producer throughput optimization in Kafka - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output of this Kafka producer throughput calculation?

Given the following Kafka producer configuration snippet, what is the effective throughput in messages per second if the batch.size is 16384 bytes, linger.ms is 5 ms, and the average message size is 512 bytes?

Kafka

batch_size = 16384
linger_ms = 5
message_size = 512
messages_per_batch = batch_size // message_size
effective_throughput = messages_per_batch / (linger_ms / 1000)
print(int(effective_throughput))

A 1024

B 6400

C 1638

D 8192

💡 Hint

Calculate how many messages fit in one batch, then divide by linger time in seconds.

The batch size is 16384 bytes, each message is 512 bytes, so 16384 // 512 = 32 messages per batch. The linger time is 5 ms, which is 0.005 seconds. Throughput = 32 / 0.005 = 6400 messages per second. The code computes this exactly and prints 6400.

❓ Predict Output

intermediate

What error does this Kafka producer config cause?

What error will this Kafka producer configuration cause when trying to send messages? Properties: acks=all batch.size=0 linger.ms=10

A No error, messages sent immediately

B TimeoutException due to linger.ms too high

C IllegalArgumentException: batch.size must be > 0

D SerializationException due to missing serializer

💡 Hint

Check the valid range for batch.size in Kafka producer configs.

batch.size=0 is valid in Kafka and disables batching. The producer sends each message as an individual request immediately. No exception is thrown during producer initialization or message sending.

🚀 Application

advanced

Optimize Kafka producer throughput with compression and batching

You want to optimize Kafka producer throughput for a high-volume topic. Which combination of settings will most likely increase throughput without significantly increasing latency?

A Set compression.type=none, batch.size=16384, linger.ms=0

B Set compression.type=gzip, batch.size=1024, linger.ms=50

C Set compression.type=snappy, batch.size=65536, linger.ms=20

D Set compression.type=lz4, batch.size=512, linger.ms=100

💡 Hint

Higher batch size and moderate linger time with fast compression helps throughput.

Using snappy compression is fast and reduces data size. Increasing batch.size to 65536 bytes allows more messages per batch. Setting linger.ms to 20 ms allows batching without too much delay. This combination improves throughput while keeping latency reasonable.

🔧 Debug

advanced

Identify the cause of low Kafka producer throughput

A Kafka producer is configured with batch.size=32768 and linger.ms=100, but throughput is unexpectedly low. Which of the following is the most likely cause?

A batch.size is too small causing frequent sends

B linger.ms is too high causing excessive delay

C acks=all causing slow acknowledgments

D max.in.flight.requests.per.connection=1 limiting parallelism

💡 Hint

Check settings that limit concurrency in sending requests.

Setting max.in.flight.requests.per.connection=1 limits the number of unacknowledged requests to one, reducing parallelism and throughput. This is often the cause of low throughput despite large batch size and linger time.

🧠 Conceptual

expert

Which Kafka producer setting most directly controls the trade-off between latency and throughput?

In Kafka producer configuration, which setting most directly controls the trade-off between latency and throughput by delaying sending to batch more messages?

A linger.ms

B batch.size

C compression.type

D acks

💡 Hint

This setting defines how long the producer waits before sending a batch.

linger.ms defines the maximum time to wait for additional messages before sending a batch. Increasing linger.ms increases batching and throughput but adds latency. It directly controls the latency-throughput trade-off.

Practice

1. What is the main benefit of increasing the batch.size setting in a Kafka producer?

easy

A. It guarantees message delivery to all replicas.

B. It compresses messages to save disk space.

C. It allows sending more messages together, improving throughput.

D. It reduces the number of partitions in a topic.

Solution

1. Step 1: Understand batch.size role

   The batch.size controls how many messages the producer groups before sending.

2. Step 2: Effect on throughput

   Grouping more messages reduces network calls, increasing throughput.

3. Final Answer:

   It allows sending more messages together, improving throughput. -> Option C

4. Quick Check:

   batch.size increases throughput [OK]

Hint: More batch size means fewer sends, better throughput [OK]

Common Mistakes:

• Confusing batch.size with message compression
• Thinking batch.size controls message durability
• Assuming batch.size changes topic partitions

2. Which of the following is the correct way to set the linger time to 100 milliseconds in Kafka producer configuration?

easy

A. "linger.ms = 100"

B. "linger_time = 100"

C. "lingerMs: 100"

D. "linger = 100ms"

Solution

1. Step 1: Identify correct property name

   The Kafka producer uses linger.ms to set linger time in milliseconds.

2. Step 2: Confirm correct syntax

   The correct syntax is linger.ms = 100, using dot notation and milliseconds.

3. Final Answer:

   "linger.ms = 100" -> Option A

4. Quick Check:

   linger.ms sets linger time [OK]

Hint: Use exact property names with dots, like linger.ms [OK]

Common Mistakes:

• Using underscores instead of dots
• Adding units like 'ms' in the value
• Using camelCase or colon syntax incorrectly

3. Given this Kafka producer config snippet:

props.put("batch.size", 16384);
props.put("linger.ms", 5);
props.put("acks", "all");

What is the expected effect on message sending?

medium

A. Messages batch up to 16KB or wait 5ms before sending, with full acknowledgment.

B. Messages are compressed automatically before sending.

C. Messages are sent immediately without batching.

D. Producer waits for only one replica acknowledgment.

Solution

1. Step 1: Analyze batch.size and linger.ms

   Batch size 16384 bytes means messages batch up to 16KB or linger 5ms before sending.

2. Step 2: Understand acks setting

   acks=all means producer waits for all replicas to confirm, ensuring durability.

3. Final Answer:

   Messages batch up to 16KB or wait 5ms before sending, with full acknowledgment. -> Option A

4. Quick Check:

   batch.size + linger.ms + acks=all = Messages batch up to 16KB or wait 5ms before sending, with full acknowledgment. [OK]

Hint: Batch size + linger.ms controls send timing, acks controls durability [OK]

Common Mistakes:

• Assuming messages send immediately ignoring linger.ms
• Confusing acks=all with acks=1
• Thinking compression happens automatically without config

4. You have this Kafka producer config:

props.put("batch.size", 32768);
props.put("linger.ms", 0);
props.put("acks", "0");

What is the main problem with this setup if you want high throughput and reliability?

medium

A. linger.ms=0 disables batching, reducing throughput.

B. acks=0 means no acknowledgment, risking message loss.

C. batch.size is too large and causes errors.

D. The config syntax is invalid and will cause exceptions.

Solution

1. Step 1: Check acks setting impact

   acks=0 means producer does not wait for any acknowledgment, risking message loss.

2. Step 2: Evaluate throughput vs reliability

   While batch.size is large and linger.ms=0 disables waiting, the main reliability issue is acks=0.

3. Final Answer:

   acks=0 means no acknowledgment, risking message loss. -> Option B

4. Quick Check:

   acks=0 risks loss despite throughput [OK]

Hint: acks=0 disables reliability, not good for safe throughput [OK]

Common Mistakes:

• Thinking linger.ms=0 disables batching completely
• Assuming batch.size too large causes errors
• Confusing config syntax as invalid

5. You want to optimize a Kafka producer for maximum throughput but still ensure messages are not lost. Which combination of settings is best?

hard

A. batch.size=512, linger.ms=50, acks=0, compression.type=lz4

B. batch.size=1024, linger.ms=0, acks=0, compression.type=none

C. batch.size=16384, linger.ms=0, acks=1, compression.type=gzip

D. batch.size=65536, linger.ms=20, acks=all, compression.type=snappy

Solution

1. Step 1: Maximize throughput with batching and linger

   Large batch.size (65536) and linger.ms (20ms) allow grouping many messages before sending.

2. Step 2: Ensure reliability and compression

   acks=all ensures all replicas confirm, preventing loss. Compression (snappy) reduces data size, improving throughput.

3. Final Answer:

   batch.size=65536, linger.ms=20, acks=all, compression.type=snappy -> Option D

4. Quick Check:

   Large batch + linger + acks=all + compression = batch.size=65536, linger.ms=20, acks=all, compression.type=snappy [OK]

Hint: Big batch + linger + acks=all + compression = best throughput + safety [OK]

Common Mistakes:

• Choosing acks=0 losing reliability
• Using too small batch sizes
• Ignoring compression benefits