How to Configure JVM for Kafka: Simple Guide

To configure the JVM for Kafka, set JVM options in the KAFKA_HEAP_OPTS and KAFKA_JVM_PERFORMANCE_OPTS environment variables or in the Kafka startup scripts. These options control memory size, garbage collection, and other JVM behaviors to optimize Kafka's performance.

Syntax

Kafka JVM configuration is done by setting environment variables or editing startup scripts with JVM options:

• KAFKA_HEAP_OPTS: Sets heap memory size (e.g., -Xms1G -Xmx1G).
• KAFKA_JVM_PERFORMANCE_OPTS: Sets JVM tuning options like garbage collection.
• These variables are used in Kafka's kafka-server-start.sh script.

bash

export KAFKA_HEAP_OPTS="-Xms1G -Xmx1G"
export KAFKA_JVM_PERFORMANCE_OPTS="-server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent"

bin/kafka-server-start.sh config/server.properties

Example

This example shows how to configure Kafka JVM heap size to 2GB and enable G1 garbage collector with tuned pause time.

bash

export KAFKA_HEAP_OPTS="-Xms2G -Xmx2G"
export KAFKA_JVM_PERFORMANCE_OPTS="-server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent"

bin/kafka-server-start.sh config/server.properties

Output

Starting Kafka server with heap size 2GB and G1GC enabled
[2024-06-01 12:00:00,000] INFO Kafka version : 3.5.0 (org.apache.kafka.common.utils.AppInfoParser)
[2024-06-01 12:00:00,001] INFO Kafka startTimeMs : 1685611200000 (org.apache.kafka.common.utils.AppInfoParser)
[2024-06-01 12:00:01,000] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)

Common Pitfalls

Common mistakes when configuring JVM for Kafka include:

• Setting heap size too large or too small causing out-of-memory or poor performance.
• Not tuning garbage collection leading to long pause times.
• Forgetting to export environment variables before starting Kafka.
• Using deprecated JVM options incompatible with the JVM version.

bash

## Wrong: No heap size set (default small heap)
export KAFKA_HEAP_OPTS=""

## Right: Set heap size explicitly
export KAFKA_HEAP_OPTS="-Xms4G -Xmx4G"

Quick Reference

Summary of key JVM options for Kafka:

Option	Description	Example
-Xms	Initial heap size	-Xms2G
-Xmx	Maximum heap size	-Xmx2G
-XX:+UseG1GC	Enable G1 Garbage Collector	-XX:+UseG1GC
-XX:MaxGCPauseMillis	Target max GC pause time	-XX:MaxGCPauseMillis=20
-XX:InitiatingHeapOccupancyPercent	GC trigger threshold	-XX:InitiatingHeapOccupancyPercent=35
-XX:+ExplicitGCInvokesConcurrent	Make explicit GC concurrent	-XX:+ExplicitGCInvokesConcurrent

Key Takeaways

Set heap size explicitly using KAFKA_HEAP_OPTS to control Kafka memory.

Use KAFKA_JVM_PERFORMANCE_OPTS to tune garbage collection for better latency.

Always export JVM options before starting Kafka to apply settings.

Avoid too large heap sizes that cause long GC pauses or too small that cause out-of-memory.

Use modern JVM options compatible with your JVM version for best results.