Consumer throughput optimization in Kafka - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output of this Kafka consumer throughput calculation?

Consider a Kafka consumer configured with the following settings:

max.poll.records=500
fetch.min.bytes=1MB
fetch.max.wait.ms=500

If the consumer processes 1000 messages per second, what is the expected throughput in MB per second assuming each message is 1KB?

A Approximately 500 MB/s

B Approximately 0.5 MB/s

C Approximately 2 MB/s

D Approximately 1 MB/s

💡 Hint

Think about how message size and processing rate relate to throughput.

Each message is 1KB, so 1000 messages equal 1000 KB = 1 MB per second throughput.

🧠 Conceptual

intermediate

Which Kafka consumer setting most directly improves throughput by increasing batch size?

Which configuration option should you increase to allow the Kafka consumer to fetch more data per request and improve throughput?

A auto.offset.reset

B session.timeout.ms

C max.poll.records

D enable.auto.commit

💡 Hint

Look for the setting that controls how many records are fetched in one poll.

max.poll.records controls the maximum number of records returned in a single poll, increasing it can improve throughput by processing larger batches.

🔧 Debug

advanced

Why does this Kafka consumer code cause low throughput?

Review the following Kafka consumer code snippet:

while (true) {
  ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
  for (ConsumerRecord<String, String> record : records) {
    processRecord(record);
    consumer.commitSync();
  }
}

What is the main reason this code causes low throughput?

A Not setting max.poll.records limits batch size to 1

B Calling commitSync() inside the loop causes frequent commits, slowing processing

C processRecord() is called outside the poll loop causing missed messages

D Polling with Duration.ofMillis(100) is too long and delays fetching

💡 Hint

Think about how committing offsets affects throughput.

Calling commitSync() inside the loop commits after every message, causing overhead and reducing throughput. Committing once per batch is better.

📝 Syntax

advanced

Which Kafka consumer configuration snippet correctly sets fetch size to optimize throughput?

Choose the correct Java code snippet to set the fetch.min.bytes to 1MB and fetch.max.wait.ms to 500ms in Kafka consumer properties.

A props.put("fetch.min.bytes", "1048576"); props.put("fetch.max.wait.ms", "500");

B props.put(fetch.min.bytes, 1048576); props.put(fetch.max.wait.ms, 500);

C props.put("fetch.min.bytes", 1048576); props.put("fetch.max.wait.ms", 500);

D props.put("fetch.min.bytes", 1_048_576); props.put("fetch.max.wait.ms", 500L);

💡 Hint

Kafka consumer properties expect string values for configuration.

Kafka consumer configs require string values, so numbers must be passed as strings.

🚀 Application

expert

How many messages will be processed per poll with these settings?

A Kafka consumer is configured with max.poll.records=1000 and fetch.min.bytes=500000. The producer sends messages of size 1KB each. If the broker has 2000 messages ready, how many messages will the consumer receive in one poll?

A 1000 messages

B 500 messages

C 2000 messages

D Cannot determine without fetch.max.wait.ms

💡 Hint

Consider max.poll.records as the upper limit per poll.

The consumer will receive up to max.poll.records (1000) messages per poll, even if more are available.

Practice

1. What is the primary purpose of increasing max.poll.records in a Kafka consumer configuration?

easy

A. To decrease the consumer's memory usage

B. To increase the number of records fetched in one poll, improving throughput

C. To reduce the number of partitions assigned to the consumer

D. To increase the consumer's connection timeout

Solution

1. Step 1: Understand max.poll.records role

   This setting controls how many records the consumer fetches in a single poll call.

2. Step 2: Effect on throughput

   Increasing this value allows the consumer to process more records per poll, which can improve throughput by reducing overhead.

3. Final Answer:

   To increase the number of records fetched in one poll, improving throughput -> Option B

4. Quick Check:

   max.poll.records controls batch size [OK]

Hint: More records per poll means higher throughput [OK]

Common Mistakes:

• Confusing max.poll.records with timeout settings
• Thinking it reduces memory usage
• Assuming it changes partition assignment

2. Which of the following is the correct way to set fetch.min.bytes to 50000 in a Kafka consumer properties file?

easy

A. fetch.min.bytes=50000

B. fetch.min.bytes: 50000

C. "fetch.min.bytes" = 50000

D. fetch_min_bytes=50000

Solution

1. Step 1: Identify Kafka properties file syntax

   Kafka consumer properties use key=value pairs without quotes or underscores.

2. Step 2: Match correct syntax

   fetch.min.bytes=50000 uses the correct key with dot notation and equals sign, matching Kafka config style.

3. Final Answer:

   fetch.min.bytes=50000 -> Option A

4. Quick Check:

   Kafka config uses key=value format [OK]

Hint: Use dot notation and equals sign for Kafka configs [OK]

Common Mistakes:

• Using colon instead of equals sign
• Adding quotes around keys
• Replacing dots with underscores

3. Given the following Kafka consumer configuration snippet:

props.put("max.poll.records", "1000");
props.put("fetch.min.bytes", "1048576");

What is the expected effect on consumer throughput and latency?

medium

A. Lower throughput and lower latency

B. Higher throughput and lower latency

C. No change in throughput or latency

D. Higher throughput with potentially increased latency

Solution

1. Step 1: Analyze max.poll.records setting

   Setting it to 1000 means the consumer fetches up to 1000 records per poll, increasing batch size and throughput.

2. Step 2: Analyze fetch.min.bytes setting

   Setting it to 1MB (1048576 bytes) means the consumer waits to fetch at least 1MB of data, which can increase latency waiting for enough data.

3. Final Answer:

   Higher throughput with potentially increased latency -> Option D

4. Quick Check:

   Large batch size = higher throughput, bigger fetch.min.bytes = higher latency [OK]

Hint: Large batch size boosts throughput but may add latency [OK]

Common Mistakes:

• Assuming latency always decreases with throughput
• Ignoring fetch.min.bytes impact
• Confusing max.poll.records with fetch.max.bytes

4. You notice your Kafka consumer is lagging and not processing messages fast enough. Which of the following configuration mistakes could cause this issue?

medium

A. Setting max.poll.records too low, like 1

B. Setting fetch.min.bytes to 1

C. Setting max.poll.interval.ms to a very high value

D. Setting enable.auto.commit to true

Solution

1. Step 1: Understand impact of low max.poll.records

   Setting it to 1 means the consumer fetches only one record per poll, severely limiting throughput and causing lag.

2. Step 2: Evaluate other options

   Setting fetch.min.bytes to 1 reduces wait time, high max.poll.interval.ms delays rebalance but doesn't reduce throughput, and auto commit true is normal behavior.

3. Final Answer:

   Setting max.poll.records too low, like 1 -> Option A

4. Quick Check:

   Too small batch size causes lag [OK]

Hint: Too small max.poll.records limits throughput [OK]

Common Mistakes:

• Confusing fetch.min.bytes effect
• Thinking auto commit affects speed directly
• Ignoring batch size impact

5. You want to optimize a Kafka consumer for high throughput but low latency. Which combination of settings is best?

hard

A. Set both max.poll.records and fetch.min.bytes to high values

B. Set max.poll.records low (e.g., 10) and fetch.min.bytes high (e.g., 1048576)

C. Set max.poll.records high (e.g., 1000) and fetch.min.bytes low (e.g., 1)

D. Set both max.poll.records and fetch.min.bytes to low values

Solution

1. Step 1: Understand throughput and latency tradeoff

   High max.poll.records allows processing many records per poll, increasing throughput.

2. Step 2: Understand fetch.min.bytes effect on latency

   Low fetch.min.bytes means the consumer fetches data quickly without waiting for large batches, reducing latency.

3. Step 3: Combine settings for desired effect

   High batch size with low fetch size balances high throughput and low latency.

4. Final Answer:

   Set max.poll.records high (e.g., 1000) and fetch.min.bytes low (e.g., 1) -> Option C

5. Quick Check:

   High batch + low fetch.min.bytes = throughput + low latency [OK]

Hint: High max.poll.records + low fetch.min.bytes = best balance [OK]

Common Mistakes:

• Setting both values high causes latency spikes
• Setting both low reduces throughput
• Ignoring fetch.min.bytes impact on latency