SSL/TLS encryption in Kafka - Commands & Configuration

Introduction

SSL/TLS encryption protects data sent between Kafka clients and brokers by making it unreadable to outsiders. It solves the problem of sensitive information being intercepted during communication.

When you want to secure messages between Kafka producers and brokers to prevent eavesdropping

When Kafka consumers need to connect securely to brokers over public or untrusted networks

When compliance rules require encrypted communication for data in transit

When you want to authenticate clients and brokers to each other using certificates

When preventing man-in-the-middle attacks on Kafka connections is important

Config File - server.properties

server.properties

listeners=SSL://:9093
advertised.listeners=SSL://localhost:9093
ssl.keystore.location=/var/private/ssl/kafka.server.keystore.jks
ssl.keystore.password=keystorepassword
ssl.key.password=keypassword
ssl.truststore.location=/var/private/ssl/kafka.server.truststore.jks
ssl.truststore.password=truststorepassword
security.inter.broker.protocol=SSL
ssl.client.auth=required

listeners: Defines Kafka to listen on SSL port 9093.

advertised.listeners: How Kafka advertises its SSL address to clients.

ssl.keystore.location: Path to the broker's keystore file containing its private key and certificate.

ssl.keystore.password and ssl.key.password: Passwords to access the keystore and key.

ssl.truststore.location: Path to truststore containing trusted CA certificates.

ssl.truststore.password: Password for the truststore.

security.inter.broker.protocol: Protocol used between brokers, set to SSL for encrypted communication.

ssl.client.auth: Requires clients to present certificates for authentication.

Commands

Starts the Kafka broker with SSL/TLS enabled using the configured server.properties file.

Terminal

kafka-server-start.sh /opt/kafka/config/server.properties

Expected Output

[2024-06-01 12:00:00,000] INFO Kafka version : 3.5.0 (org.apache.kafka.common.utils.AppInfoParser)

[2024-06-01 12:00:00,001] INFO Kafka startTimeMs : 1685611200000 (org.apache.kafka.common.utils.AppInfoParser)

[2024-06-01 12:00:01,500] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)

Starts a Kafka producer that connects securely over SSL to send messages to 'my-topic'. The client-ssl.properties file contains SSL client settings.

Terminal

kafka-console-producer.sh --broker-list localhost:9093 --topic my-topic --producer.config client-ssl.properties

Expected Output

> This is a test message

→

--broker-list - Specifies the Kafka broker address with SSL port

→

--producer.config - Points to SSL configuration for the producer client

Starts a Kafka consumer that connects securely over SSL to read messages from 'my-topic' from the beginning.

Terminal

kafka-console-consumer.sh --bootstrap-server localhost:9093 --topic my-topic --from-beginning --consumer.config client-ssl.properties

Expected Output

This is a test message

→

--bootstrap-server - Specifies the Kafka broker address with SSL port

→

--consumer.config - Points to SSL configuration for the consumer client

Key Concept

If you remember nothing else from this pattern, remember: SSL/TLS encrypts Kafka traffic and authenticates clients and brokers using certificates to keep data safe in transit.

Common Mistakes

Not setting the correct SSL port in listeners and advertised.listeners

Clients will fail to connect because Kafka is not listening on the expected SSL port.

Set listeners and advertised.listeners to the SSL port, e.g., SSL://:9093

Using wrong or missing keystore/truststore passwords

Kafka broker or clients cannot access certificates, causing SSL handshake failures.

Ensure keystore and truststore passwords are correct and match the files.

Forgetting to configure client SSL properties for producers and consumers

Clients cannot authenticate or encrypt connections, leading to connection errors.

Create and use client-ssl.properties with proper SSL settings for clients.

Summary

Configure Kafka broker with SSL settings in server.properties to enable encrypted communication.

Start Kafka broker and verify it listens on the SSL port.

Use SSL-configured client properties to securely produce and consume messages.