Disk I/O optimization in Kafka - Commands & Configuration

Introduction

Disk input/output speed affects how fast Kafka reads and writes messages. Optimizing disk I/O helps Kafka handle more data smoothly and reduces delays.

When Kafka brokers experience slow message processing due to disk bottlenecks.

When you want to improve Kafka throughput for high-volume data streams.

When monitoring shows high disk wait times affecting Kafka performance.

When setting up Kafka on servers with multiple disks and you want to balance load.

When tuning Kafka for better durability and faster recovery after crashes.

Config File - server.properties

server.properties

log.dirs=/var/lib/kafka-logs-1,/var/lib/kafka-logs-2
num.io.threads=8
num.replica.fetchers=4
log.flush.interval.messages=10000
log.flush.interval.ms=1000
log.segment.bytes=1073741824
log.retention.hours=168
log.retention.bytes=10737418240

log.dirs: Lists multiple directories on different disks to spread Kafka logs and reduce disk I/O bottlenecks.

num.io.threads: Number of threads Kafka uses for disk I/O operations, increasing parallelism.

num.replica.fetchers: Threads fetching data from leader replicas, affecting disk read load.

log.flush.interval.messages and log.flush.interval.ms: Control how often Kafka flushes data to disk, balancing durability and performance.

log.segment.bytes: Size of log segments to optimize disk writes.

log.retention.hours and log.retention.bytes: Manage how long and how much data Kafka keeps on disk to avoid disk full issues.

Commands

Starts the Kafka broker using the optimized disk I/O settings from the configuration file.

Terminal

kafka-server-start.sh /opt/kafka/config/server.properties

Expected Output

[2024-06-01 12:00:00,000] INFO Kafka version : 3.5.0 (org.apache.kafka.common.utils.AppInfoParser)

[2024-06-01 12:00:00,001] INFO Kafka startTimeMs : 1685611200000 (org.apache.kafka.common.utils.AppInfoParser)

[2024-06-01 12:00:05,000] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)

Checks disk I/O statistics every 2 seconds, 3 times, to monitor disk usage and identify bottlenecks.

Terminal

iostat -dx 2 3

Expected Output

Linux 5.15.0-60-generic (kafka-server)  06/01/2024  _x86_64_  (4 CPU)



Device            r/s     w/s     rkB/s   wkB/s  rrqm/s  wrqm/s  r_await  w_await  svctm  %util

sda               50.00   30.00   1024.00 512.00  0.00    0.00    1.00     2.00     0.50   4.00

sdb               70.00   40.00   2048.00 1024.00 0.00    0.00    1.50     1.80     0.40   5.00

Shows details about the topic partitions and replicas to verify data distribution and replication which affect disk I/O.

Terminal

kafka-topics.sh --describe --topic example-topic --bootstrap-server localhost:9092

Expected Output

Topic: example-topic  PartitionCount: 3  ReplicationFactor: 2  Configs:

  Topic: example-topic  Partition: 0  Leader: 1  Replicas: 1,2  Isr: 1,2

  Topic: example-topic  Partition: 1  Leader: 2  Replicas: 2,1  Isr: 2,1

  Topic: example-topic  Partition: 2  Leader: 1  Replicas: 1,2  Isr: 1,2

→

--describe - Shows detailed information about the topic

→

--bootstrap-server - Specifies the Kafka server to connect to

Key Concept

If you remember nothing else from disk I/O optimization, remember: spreading Kafka logs across multiple disks and tuning I/O threads improves performance and reduces bottlenecks.

Common Mistakes

Setting log.dirs to only one directory on a single disk.

This causes all disk I/O to hit one disk, creating a bottleneck and slowing Kafka.

Use multiple directories on separate disks in log.dirs to distribute load.

Using too few I/O threads with num.io.threads.

Kafka cannot perform enough parallel disk operations, limiting throughput.

Increase num.io.threads to match the number of disks and CPU cores for better parallelism.

Setting log.flush.interval.ms or log.flush.interval.messages too low.

Causes excessive disk flushes, increasing disk I/O and reducing performance.

Set flush intervals to reasonable values balancing durability and performance.

Summary

Configure multiple log directories on separate disks to spread disk I/O load.

Adjust num.io.threads to increase parallel disk operations.

Monitor disk usage with iostat to identify bottlenecks and tune settings accordingly.

Practice

1. Which Kafka configuration setting primarily affects how many threads handle disk read and write operations?

easy

A. num.io.threads

B. log.retention.hours

C. zookeeper.connect

D. auto.create.topics.enable

Solution

1. Step 1: Identify the setting related to disk I/O threads

   The setting num.io.threads controls the number of threads Kafka uses for disk input/output operations.

2. Step 2: Compare other options for relevance

   Other options like log.retention.hours relate to data retention, zookeeper.connect is for cluster coordination, and auto.create.topics.enable controls topic creation, none affect disk I/O threads.

3. Final Answer:

   num.io.threads -> Option A

4. Quick Check:

   Disk I/O threads = num.io.threads [OK]

Hint: Look for settings with 'io.threads' for disk I/O control [OK]

Common Mistakes:

• Confusing retention settings with I/O threads
• Choosing zookeeper settings instead of I/O
• Selecting topic creation options

2. Which of the following is the correct syntax to set the log segment size to 1 GB in Kafka's server.properties file?

easy

A. log.segment.bytes=1GB

B. log.segment.size=1GB

C. segment.log.bytes=1024

D. log.segment.bytes=1073741824

Solution

1. Step 1: Understand Kafka's configuration syntax for sizes

   Kafka expects byte values as integers for size settings, so 1 GB must be expressed in bytes (1 GB = 1073741824 bytes).

2. Step 2: Evaluate each option's correctness

   log.segment.bytes=1073741824 uses the correct key log.segment.bytes and the correct byte value. log.segment.size=1GB uses an invalid key and unit. segment.log.bytes=1024 uses wrong key and too small value. log.segment.bytes=1GB uses correct key but invalid unit '1GB'.

3. Final Answer:

   log.segment.bytes=1073741824 -> Option D

4. Quick Check:

   Size in bytes = 1073741824 [OK]

Hint: Use bytes as integer for size settings, no units like GB [OK]

Common Mistakes:

• Using units like GB instead of bytes
• Incorrect property names
• Using too small or wrong numeric values

3. Given the following Kafka configuration snippet:

num.io.threads=4
log.segment.bytes=10485760
log.dirs=/kafka-logs

What is the effect of setting num.io.threads to 4?

medium

A. Kafka limits the number of topics to 4.

B. Kafka will create 4 log directories.

C. Kafka will use 4 threads to handle disk I/O operations, improving parallelism.

D. Kafka will retain logs for 4 hours.

Solution

1. Step 1: Understand the role of num.io.threads

   This setting controls how many threads Kafka uses for disk input/output operations, allowing parallel reads and writes.

2. Step 2: Check other options for relevance

   Options about limiting topics, creating directories, or retention hours do not relate to num.io.threads.

3. Final Answer:

   Kafka will use 4 threads to handle disk I/O operations, improving parallelism. -> Option C

4. Quick Check:

   num.io.threads = 4 threads for disk I/O [OK]

Hint: num.io.threads sets disk I/O thread count, not topics or retention [OK]

Common Mistakes:

• Confusing thread count with topic count
• Assuming it controls log retention
• Thinking it creates multiple directories

4. You have set log.segment.bytes=10485760 but Kafka still creates very small log segments. What is the most likely cause?

medium

A. The log.roll.ms setting causes segments to roll before reaching the size limit.

B. The num.io.threads is too low.

C. The log.dirs path is incorrect.

D. Kafka does not support segment size configuration.

Solution

1. Step 1: Understand segment rolling behavior

   Kafka rolls log segments based on size or time. If log.roll.ms is set low, segments roll by time before reaching the size limit.

2. Step 2: Evaluate other options

   Low num.io.threads affects performance but not segment size. Incorrect log.dirs causes errors but not small segments. Kafka supports segment size configuration.

3. Final Answer:

   The log.roll.ms setting causes segments to roll before reaching the size limit. -> Option A

4. Quick Check:

   Segment rolling time overrides size limit [OK]

Hint: Check log.roll.ms if segments are smaller than expected [OK]

Common Mistakes:

• Blaming I/O threads for segment size
• Ignoring time-based rolling settings
• Assuming Kafka ignores segment size config

5. You want to optimize Kafka disk I/O for a high-throughput system. Which combination of settings is best to improve write performance without increasing disk usage too much?

hard

A. Decrease num.io.threads, set very large log.segment.bytes, and disable flushing

B. Increase num.io.threads, set moderate log.segment.bytes, and tune log.flush.interval.messages

C. Set num.io.threads to 1, use small log.segment.bytes, and flush after every message

D. Ignore num.io.threads, set log.segment.bytes to default, and increase retention time

Solution

1. Step 1: Understand the impact of num.io.threads and segment size

   Increasing num.io.threads allows more parallel disk operations. Moderate segment size balances write efficiency and disk usage.

2. Step 2: Consider flushing settings

   Tuning log.flush.interval.messages controls how often data is flushed to disk, improving performance without excessive disk writes.

3. Step 3: Evaluate other options

   Decreasing threads or disabling flushing harms performance or data safety. Small segments and frequent flushes increase disk I/O. Ignoring threads or retention does not optimize writes.

4. Final Answer:

   Increase num.io.threads, set moderate log.segment.bytes, and tune log.flush.interval.messages -> Option B

5. Quick Check:

   Balanced threads, segment size, and flush tuning optimize disk I/O [OK]

Hint: Balance threads, segment size, and flush interval for best write speed [OK]

Common Mistakes:

• Setting too large segments causing disk bloat
• Disabling flushing risking data loss
• Using too few I/O threads limiting throughput