Disk I/O optimization in Kafka - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Understanding Kafka Disk Flush Behavior

What is the output of the following Kafka producer configuration snippet regarding disk flush frequency?

Kafka

props.put("linger.ms", "500");
props.put("batch.size", "16384");
props.put("acks", "all");
// Producer sends 10 messages rapidly
for (int i = 0; i < 10; i++) {
    producer.send(new ProducerRecord<>("topic", Integer.toString(i), "message" + i));
}
producer.flush();
System.out.println("Messages sent and flushed.");

A Messages sent and flushed. Producer never flushes automatically; manual flush is required only.

B Messages sent and flushed. Producer flushes after every single message immediately.

C Messages sent and flushed. Producer batches messages and flushes every 500ms or when batch size is reached.

D Messages sent and flushed. Producer flushes only after 10 seconds regardless of batch size.

💡 Hint

Think about how linger.ms and batch.size affect when Kafka flushes data to disk.

The linger.ms setting controls how long the producer waits before sending a batch, allowing more messages to accumulate. The batch.size sets the max batch size. The producer flushes when either condition is met. So, messages are batched and flushed every 500ms or when batch size is reached.

🧠 Conceptual

intermediate

Effect of Kafka Log Segment Size on Disk I/O

How does increasing the Kafka log segment size affect disk I/O performance?

A Log segment size has no effect on disk I/O performance in Kafka.

B Larger log segments reduce the frequency of disk flushes, improving sequential disk writes and reducing I/O overhead.

C Larger log segments increase random disk writes, causing more disk I/O overhead.

D Smaller log segments always improve disk I/O by reducing disk seek times.

💡 Hint

Think about how disk writes benefit from sequential large writes versus many small writes.

Increasing log segment size means Kafka writes larger chunks sequentially to disk less often, which reduces disk seeks and overhead, improving disk I/O performance.

🔧 Debug

advanced

Identifying Disk I/O Bottleneck in Kafka Consumer

Given this Kafka consumer code snippet, which option correctly identifies the cause of high disk I/O latency?

Kafka

consumer.subscribe(Collections.singletonList("topic"));
while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
    for (ConsumerRecord<String, String> record : records) {
        process(record.value());
        consumer.commitSync();
    }
}

A Subscribing to a single topic causes disk I/O bottleneck due to lack of parallelism.

B Polling every 100ms causes excessive disk I/O and should be increased to 1000ms.

C Not using commitAsync() causes disk I/O latency due to synchronous commits.

D Calling commitSync() inside the for-loop causes frequent disk writes, increasing disk I/O latency.

💡 Hint

Consider how often commitSync() writes to disk and its impact on performance.

Calling commitSync() inside the loop commits offsets after every record, causing many small disk writes and increasing disk I/O latency. Committing offsets less frequently reduces disk I/O overhead.

📝 Syntax

advanced

Correct Kafka Producer Configuration for Disk I/O Optimization

Which option shows the correct Java code snippet to configure a Kafka producer for optimized disk I/O with compression and batching?

A

props.put("compression.type", "snappy");
props.put("batch.size", 32768);
props.put("linger.ms", 100);

B

props.put("compression", "snappy");
props.put("batchSize", "32768");
props.put("linger", 100);

C

props.put("compression.type", snappy);
props.put("batch.size", "32768");
props.put("linger.ms", "100");

D

props.put("compression.type", "gzip");
props.put("batch.size", 100);
props.put("linger.ms", 1000);

💡 Hint

Check property names and value types carefully.

Option A uses correct property names and appropriate value types for compression.type, batch.size, and linger.ms. Other options have incorrect property names, missing quotes, or inappropriate values.

🚀 Application

expert

Optimizing Kafka Broker Disk Usage for High Throughput

You manage a Kafka cluster with high disk I/O load. Which configuration change will most effectively reduce disk I/O pressure on brokers while maintaining high throughput?

A Increase log.segment.bytes to a larger size and enable log compaction to reduce disk writes.

B Decrease log.retention.hours to delete old logs faster and reduce disk usage.

C Set num.io.threads to 1 to serialize disk access and reduce contention.

D Disable log.cleaner to avoid extra disk I/O from compaction processes.

💡 Hint

Think about how log segment size and compaction affect disk write patterns.

Increasing log segment size reduces the frequency of disk flushes, improving sequential writes. Enabling log compaction removes redundant data, reducing disk usage and I/O. Other options either reduce data retention without improving throughput or reduce parallelism causing bottlenecks.