Memory and buffer configuration in Kafka - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output of the producer buffer size setting?

Consider a Kafka producer configured with buffer.memory=33554432 (32 MB). What is the maximum amount of memory the producer can use to buffer records waiting to be sent?

Kafka

Properties props = new Properties();
props.put("buffer.memory", "33554432");
KafkaProducer<String, String> producer = new KafkaProducer<>(props);
System.out.println(producer.metrics().get("bufferpool-total-bytes").metricValue());

A 0

B 33554432

C 67108864

D 1048576

💡 Hint

The buffer.memory property sets the total bytes of memory the producer can use to buffer records.

The buffer.memory setting controls the total bytes of memory the producer can use to buffer records before sending. Setting it to 33554432 means 32 MB of buffer memory.

❓ Predict Output

intermediate

What happens if fetch.min.bytes is set too high?

In a Kafka consumer configuration, if fetch.min.bytes is set to 1048576 (1 MB), what is the expected behavior when the broker has less data available?

Kafka

Properties props = new Properties();
props.put("fetch.min.bytes", "1048576");
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
System.out.println(records.count());

A Consumer waits until 1 MB of data is available before returning any records.

B Consumer returns zero records and closes the connection.

C Consumer throws an exception if less than 1 MB is available.

D Consumer immediately returns any available data regardless of size.

💡 Hint

fetch.min.bytes controls the minimum amount of data the broker returns to the consumer.

Setting fetch.min.bytes to 1 MB means the consumer will wait until at least 1 MB of data is available before returning records, which can increase latency but improve throughput.

🔧 Debug

advanced

Why does the producer throw a BufferExhaustedException?

A Kafka producer configured with buffer.memory=16384 (16 KB) and batch.size=32768 (32 KB) throws a BufferExhaustedException during high load. Why?

Kafka

Properties props = new Properties();
props.put("buffer.memory", "16384");
props.put("batch.size", "32768");
KafkaProducer<String, String> producer = new KafkaProducer<>(props);
// High volume sending loop
for (int i = 0; i < 100000; i++) {
    producer.send(new ProducerRecord<>("topic", "key", "value"));
}

A The producer is missing the linger.ms setting, causing immediate sends.

B batch.size is smaller than buffer.memory, causing buffer overflow.

C buffer.memory is smaller than batch.size, so the producer cannot allocate enough buffer for a batch.

D The producer's max.request.size is too large.

💡 Hint

buffer.memory must be at least as large as batch.size to hold a batch in memory.

The producer's buffer.memory is 16 KB but batch.size is 32 KB, so the producer cannot allocate enough memory for a batch, causing BufferExhaustedException.

📝 Syntax

advanced

Identify the syntax error in this Kafka consumer buffer configuration

Which option contains a syntax error in setting the consumer's fetch.max.bytes property in Java?

Kafka

Properties props = new Properties();

A ;)"00882425" ,"setyb.xam.hctef"(tup.sporp

B props.put("fetch.max.bytes", 52428800);

C props.put("fetch.max.bytes", "52428800");

D props.put(fetch.max.bytes, "52428800");

💡 Hint

Property keys must be strings in quotes.

Option D misses quotes around the property key, causing a compile-time error because fetch.max.bytes is not defined as a variable.

🚀 Application

expert

Calculate total memory usage for a Kafka producer with multiple buffers

A Kafka producer is configured with buffer.memory=67108864 (64 MB), batch.size=16384 (16 KB), and max.in.flight.requests.per.connection=5. What is the minimum total memory the producer needs to handle 5 in-flight batches without blocking?

A 81920 bytes plus overhead, but less than 67108864 bytes

B 67108864 bytes

C 81920 bytes

D More than 67108864 bytes

💡 Hint

Total memory must cover all in-flight batches but cannot exceed buffer.memory.

Each batch is 16384 bytes. For 5 in-flight requests, minimum memory is 5 * 16384 = 81920 bytes. This is less than buffer.memory (64 MB), so the producer can handle it without blocking. Actual usage includes overhead, so total is slightly more than 81920 bytes but less than 67108864 bytes.