SSL/TLS encryption in Kafka - Commands & Configuration

Introduction

SSL/TLS encryption protects data sent between Kafka clients and brokers by making it unreadable to outsiders. It solves the problem of sensitive information being intercepted during communication.

When you want to secure messages between Kafka producers and brokers to prevent eavesdropping

When Kafka consumers need to connect securely to brokers over public or untrusted networks

When compliance rules require encrypted communication for data in transit

When you want to authenticate clients and brokers to each other using certificates

When preventing man-in-the-middle attacks on Kafka connections is important

Config File - server.properties

server.properties

listeners=SSL://:9093
advertised.listeners=SSL://localhost:9093
ssl.keystore.location=/var/private/ssl/kafka.server.keystore.jks
ssl.keystore.password=keystorepassword
ssl.key.password=keypassword
ssl.truststore.location=/var/private/ssl/kafka.server.truststore.jks
ssl.truststore.password=truststorepassword
security.inter.broker.protocol=SSL
ssl.client.auth=required

listeners: Defines Kafka to listen on SSL port 9093.

advertised.listeners: How Kafka advertises its SSL address to clients.

ssl.keystore.location: Path to the broker's keystore file containing its private key and certificate.

ssl.keystore.password and ssl.key.password: Passwords to access the keystore and key.

ssl.truststore.location: Path to truststore containing trusted CA certificates.

ssl.truststore.password: Password for the truststore.

security.inter.broker.protocol: Protocol used between brokers, set to SSL for encrypted communication.

ssl.client.auth: Requires clients to present certificates for authentication.

Commands

Starts the Kafka broker with SSL/TLS enabled using the configured server.properties file.

Terminal

kafka-server-start.sh /opt/kafka/config/server.properties

Expected Output

[2024-06-01 12:00:00,000] INFO Kafka version : 3.5.0 (org.apache.kafka.common.utils.AppInfoParser)

[2024-06-01 12:00:00,001] INFO Kafka startTimeMs : 1685611200000 (org.apache.kafka.common.utils.AppInfoParser)

[2024-06-01 12:00:01,500] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)

Starts a Kafka producer that connects securely over SSL to send messages to 'my-topic'. The client-ssl.properties file contains SSL client settings.

Terminal

kafka-console-producer.sh --broker-list localhost:9093 --topic my-topic --producer.config client-ssl.properties

Expected Output

> This is a test message

→

--broker-list - Specifies the Kafka broker address with SSL port

→

--producer.config - Points to SSL configuration for the producer client

Starts a Kafka consumer that connects securely over SSL to read messages from 'my-topic' from the beginning.

Terminal

kafka-console-consumer.sh --bootstrap-server localhost:9093 --topic my-topic --from-beginning --consumer.config client-ssl.properties

Expected Output

This is a test message

→

--bootstrap-server - Specifies the Kafka broker address with SSL port

→

--consumer.config - Points to SSL configuration for the consumer client

Key Concept

If you remember nothing else from this pattern, remember: SSL/TLS encrypts Kafka traffic and authenticates clients and brokers using certificates to keep data safe in transit.

Common Mistakes

Not setting the correct SSL port in listeners and advertised.listeners

Clients will fail to connect because Kafka is not listening on the expected SSL port.

Set listeners and advertised.listeners to the SSL port, e.g., SSL://:9093

Using wrong or missing keystore/truststore passwords

Kafka broker or clients cannot access certificates, causing SSL handshake failures.

Ensure keystore and truststore passwords are correct and match the files.

Forgetting to configure client SSL properties for producers and consumers

Clients cannot authenticate or encrypt connections, leading to connection errors.

Create and use client-ssl.properties with proper SSL settings for clients.

Summary

Configure Kafka broker with SSL settings in server.properties to enable encrypted communication.

Start Kafka broker and verify it listens on the SSL port.

Use SSL-configured client properties to securely produce and consume messages.

Practice

1. What is the main purpose of enabling SSL/TLS encryption in Kafka?

easy

A. To protect data from being read or altered by unauthorized parties

B. To increase Kafka message throughput

C. To reduce Kafka broker memory usage

D. To enable Kafka to run without a network connection

Solution

1. Step 1: Understand SSL/TLS role in Kafka

   SSL/TLS encryption is used to secure data by encrypting communication between Kafka clients and brokers.

2. Step 2: Identify the main benefit

   This encryption prevents unauthorized reading or tampering of data during transmission.

3. Final Answer:

   To protect data from being read or altered by unauthorized parties -> Option A

4. Quick Check:

   SSL/TLS protects data confidentiality and integrity [OK]

Hint: SSL/TLS means secure data transfer, so it protects data [OK]

Common Mistakes:

• Thinking SSL/TLS improves performance
• Confusing SSL/TLS with offline mode
• Assuming SSL/TLS reduces memory use

2. Which Kafka configuration property specifies the use of SSL for communication?

easy

A. security.protocol

B. ssl.enabled

C. encryption.method

D. kafka.ssl

Solution

1. Step 1: Recall Kafka SSL configuration

   Kafka uses the property security.protocol to define the communication protocol, including SSL.

2. Step 2: Verify other options

   Properties like ssl.enabled, encryption.method, and kafka.ssl are not valid Kafka config keys.

3. Final Answer:

   security.protocol -> Option A

4. Quick Check:

   security.protocol sets SSL usage [OK]

Hint: Remember Kafka uses security.protocol for SSL setup [OK]

Common Mistakes:

• Using non-existent properties like ssl.enabled
• Confusing encryption.method with Kafka config
• Assuming kafka.ssl is a valid property

3. Given this Kafka client config snippet:

security.protocol=SSL
ssl.keystore.location=/path/client.keystore.jks
ssl.truststore.location=/path/client.truststore.jks

What will happen if the truststore file path is incorrect?

medium

A. The client will connect but with a warning logged

B. The client will connect but without encryption

C. The client will ignore the truststore and use default certificates

D. The client will fail to connect due to truststore loading error

Solution

1. Step 1: Understand truststore role

   The truststore contains certificates to verify the broker's identity. If missing or incorrect, verification fails.

2. Step 2: Predict client behavior on truststore error

   Kafka client will fail to establish SSL connection if truststore cannot be loaded or is invalid.

3. Final Answer:

   The client will fail to connect due to truststore loading error -> Option D

4. Quick Check:

   Incorrect truststore causes connection failure [OK]

Hint: Truststore errors block SSL connection, no fallback [OK]

Common Mistakes:

• Assuming client connects without encryption
• Thinking client uses default certificates automatically
• Believing connection succeeds with warnings

4. You configured Kafka broker with SSL but clients still connect without encryption. What is the most likely mistake?

medium

A. Broker keystore file is missing

B. Clients did not set security.protocol to SSL

C. Broker truststore file is invalid

D. Clients set ssl.keystore.location incorrectly

Solution

1. Step 1: Check client SSL configuration

   Even if broker is configured for SSL, clients must also set security.protocol=SSL to use SSL.

2. Step 2: Understand effect of missing client SSL setting

   If clients do not specify SSL, they connect using plaintext, ignoring broker SSL setup.

3. Final Answer:

   Clients did not set security.protocol to SSL -> Option B

4. Quick Check:

   Client SSL setting required for encrypted connection [OK]

Hint: Both client and broker must enable SSL explicitly [OK]

Common Mistakes:

• Assuming broker config alone enforces SSL
• Blaming missing broker keystore without checking client
• Confusing truststore errors with client SSL setting

5. You want to secure Kafka communication with SSL and also require client authentication. Which configuration combination is correct?

hard

A. Broker: security.protocol=SSL; Client: security.protocol=PLAINTEXT

B. Broker: security.protocol=PLAINTEXT; Client: security.protocol=SSL

C. Broker: security.protocol=SSL, ssl.client.auth=required; Client: security.protocol=SSL, ssl.keystore.location set

D. Broker: security.protocol=SSL, ssl.client.auth=none; Client: security.protocol=SSL

Solution

1. Step 1: Understand client authentication in Kafka SSL

   To require client authentication, broker must set ssl.client.auth=required and clients must provide keystore with their certificates.

2. Step 2: Verify correct protocol settings

   Both broker and client must use security.protocol=SSL to enable SSL communication.

3. Step 3: Eliminate incorrect options

   Options with PLAINTEXT protocol or ssl.client.auth=none do not enforce client authentication.

4. Final Answer:

   Broker: security.protocol=SSL, ssl.client.auth=required; Client: security.protocol=SSL, ssl.keystore.location set -> Option C

5. Quick Check:

   Client auth requires ssl.client.auth=required and client keystore [OK]

Hint: Client auth needs broker ssl.client.auth=required and client keystore [OK]

Common Mistakes:

• Setting broker to PLAINTEXT while client uses SSL
• Not setting ssl.client.auth to required on broker
• Forgetting client keystore for authentication