Consumer throughput optimization in Kafka - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to set the maximum poll records for higher throughput.

Kafka

props.put("max.poll.records", [1]);

Drag options to blanks, or click blank then click option'

A500

B1000

C10

D1

Attempts:

3 left

💡 Hint

Common Mistakes

Setting max.poll.records too low limits throughput.

Using a value too high can cause long processing delays.

Setting max.poll.records to 500 allows the consumer to fetch up to 500 records per poll, improving throughput.

2fill in blank

medium

Complete the code to set the fetch size for the consumer to optimize throughput.

Kafka

props.put("fetch.min.bytes", [1]);

Drag options to blanks, or click blank then click option'

A1

B50000

C1000

D100

Attempts:

3 left

💡 Hint

Common Mistakes

Setting fetch.min.bytes too low causes many small fetches.

Too high a value can increase latency.

Setting fetch.min.bytes to 50000 means the consumer waits to fetch at least 50KB of data, improving throughput by reducing fetch frequency.

3fill in blank

hard

Fix the error in the code to properly commit offsets asynchronously for better throughput.

Kafka

consumer.[1]((offsets, exception) -> {});

Drag options to blanks, or click blank then click option'

Acommit

BcommitOffset

CcommitAsync

DcommitSync

Attempts:

3 left

💡 Hint

Common Mistakes

Using commitSync causes blocking and reduces throughput.

Using a non-existent commit method causes runtime errors.

Using commitAsync() commits offsets asynchronously, which improves throughput by not blocking the consumer thread.

4fill in blank

hard

Fill both blanks to configure the consumer for better throughput by adjusting session timeout and heartbeat interval.

Kafka

props.put("session.timeout.ms", [1]);
props.put("heartbeat.interval.ms", [2]);

Drag options to blanks, or click blank then click option'

A30000

B10000

C3000

D1000

Attempts:

3 left

💡 Hint

Common Mistakes

Setting heartbeat interval higher than session timeout causes consumer to be considered dead.

Too low session timeout causes frequent rebalances.

Setting session.timeout.ms to 30000 ms and heartbeat.interval.ms to 3000 ms balances consumer liveness detection and heartbeat frequency for throughput optimization.

5fill in blank

hard

Fill all three blanks to create a consumer poll loop that maximizes throughput by adjusting poll timeout, processing batch size, and commit frequency.

Kafka

while (true) {
    ConsumerRecords<String, String> records = consumer.poll([1]);
    for (ConsumerRecord<String, String> record : records) {
        process(record);
    }
    if (records.count() >= [2]) {
        consumer.[3]((offsets, exception) -> {});
    }
}

Drag options to blanks, or click blank then click option'

A1000

B500

CcommitAsync

DcommitSync

Attempts:

3 left

💡 Hint

Common Mistakes

Using commitSync blocks the consumer and reduces throughput.

Polling with too low timeout fetches fewer records, reducing throughput.

Polling with a 1000 ms timeout allows batch fetching, processing batches of 500 records before committing asynchronously improves throughput by reducing commit overhead.

Practice

1. What is the primary purpose of increasing max.poll.records in a Kafka consumer configuration?

easy

A. To decrease the consumer's memory usage

B. To increase the number of records fetched in one poll, improving throughput

C. To reduce the number of partitions assigned to the consumer

D. To increase the consumer's connection timeout

Solution

1. Step 1: Understand max.poll.records role

   This setting controls how many records the consumer fetches in a single poll call.

2. Step 2: Effect on throughput

   Increasing this value allows the consumer to process more records per poll, which can improve throughput by reducing overhead.

3. Final Answer:

   To increase the number of records fetched in one poll, improving throughput -> Option B

4. Quick Check:

   max.poll.records controls batch size [OK]

Hint: More records per poll means higher throughput [OK]

Common Mistakes:

• Confusing max.poll.records with timeout settings
• Thinking it reduces memory usage
• Assuming it changes partition assignment

2. Which of the following is the correct way to set fetch.min.bytes to 50000 in a Kafka consumer properties file?

easy

A. fetch.min.bytes=50000

B. fetch.min.bytes: 50000

C. "fetch.min.bytes" = 50000

D. fetch_min_bytes=50000

Solution

1. Step 1: Identify Kafka properties file syntax

   Kafka consumer properties use key=value pairs without quotes or underscores.

2. Step 2: Match correct syntax

   fetch.min.bytes=50000 uses the correct key with dot notation and equals sign, matching Kafka config style.

3. Final Answer:

   fetch.min.bytes=50000 -> Option A

4. Quick Check:

   Kafka config uses key=value format [OK]

Hint: Use dot notation and equals sign for Kafka configs [OK]

Common Mistakes:

• Using colon instead of equals sign
• Adding quotes around keys
• Replacing dots with underscores

3. Given the following Kafka consumer configuration snippet:

props.put("max.poll.records", "1000");
props.put("fetch.min.bytes", "1048576");

What is the expected effect on consumer throughput and latency?

medium

A. Lower throughput and lower latency

B. Higher throughput and lower latency

C. No change in throughput or latency

D. Higher throughput with potentially increased latency

Solution

1. Step 1: Analyze max.poll.records setting

   Setting it to 1000 means the consumer fetches up to 1000 records per poll, increasing batch size and throughput.

2. Step 2: Analyze fetch.min.bytes setting

   Setting it to 1MB (1048576 bytes) means the consumer waits to fetch at least 1MB of data, which can increase latency waiting for enough data.

3. Final Answer:

   Higher throughput with potentially increased latency -> Option D

4. Quick Check:

   Large batch size = higher throughput, bigger fetch.min.bytes = higher latency [OK]

Hint: Large batch size boosts throughput but may add latency [OK]

Common Mistakes:

• Assuming latency always decreases with throughput
• Ignoring fetch.min.bytes impact
• Confusing max.poll.records with fetch.max.bytes

4. You notice your Kafka consumer is lagging and not processing messages fast enough. Which of the following configuration mistakes could cause this issue?

medium

A. Setting max.poll.records too low, like 1

B. Setting fetch.min.bytes to 1

C. Setting max.poll.interval.ms to a very high value

D. Setting enable.auto.commit to true

Solution

1. Step 1: Understand impact of low max.poll.records

   Setting it to 1 means the consumer fetches only one record per poll, severely limiting throughput and causing lag.

2. Step 2: Evaluate other options

   Setting fetch.min.bytes to 1 reduces wait time, high max.poll.interval.ms delays rebalance but doesn't reduce throughput, and auto commit true is normal behavior.

3. Final Answer:

   Setting max.poll.records too low, like 1 -> Option A

4. Quick Check:

   Too small batch size causes lag [OK]

Hint: Too small max.poll.records limits throughput [OK]

Common Mistakes:

• Confusing fetch.min.bytes effect
• Thinking auto commit affects speed directly
• Ignoring batch size impact

5. You want to optimize a Kafka consumer for high throughput but low latency. Which combination of settings is best?

hard

A. Set both max.poll.records and fetch.min.bytes to high values

B. Set max.poll.records low (e.g., 10) and fetch.min.bytes high (e.g., 1048576)

C. Set max.poll.records high (e.g., 1000) and fetch.min.bytes low (e.g., 1)

D. Set both max.poll.records and fetch.min.bytes to low values

Solution

1. Step 1: Understand throughput and latency tradeoff

   High max.poll.records allows processing many records per poll, increasing throughput.

2. Step 2: Understand fetch.min.bytes effect on latency

   Low fetch.min.bytes means the consumer fetches data quickly without waiting for large batches, reducing latency.

3. Step 3: Combine settings for desired effect

   High batch size with low fetch size balances high throughput and low latency.

4. Final Answer:

   Set max.poll.records high (e.g., 1000) and fetch.min.bytes low (e.g., 1) -> Option C

5. Quick Check:

   High batch + low fetch.min.bytes = throughput + low latency [OK]

Hint: High max.poll.records + low fetch.min.bytes = best balance [OK]

Common Mistakes:

• Setting both values high causes latency spikes
• Setting both low reduces throughput
• Ignoring fetch.min.bytes impact on latency