Client authentication configuration in Kafka - Commands & Configuration

Introduction

Client authentication configuration in Kafka ensures that only trusted users or applications can connect to the Kafka server. It protects your data by requiring clients to prove their identity before sending or receiving messages.

When you want to prevent unauthorized applications from reading or writing Kafka topics.

When you need to secure communication between Kafka clients and brokers in a multi-tenant environment.

When you want to comply with security policies that require encrypted and authenticated connections.

When running Kafka in a cloud or public network where anyone could try to connect.

When you want to audit and control which clients access your Kafka cluster.

Config File - server.properties

server.properties

listeners=SASL_PLAINTEXT://:9092
advertised.listeners=SASL_PLAINTEXT://localhost:9092
security.inter.broker.protocol=SASL_PLAINTEXT
sasl.enabled.mechanisms=PLAIN
sasl.mechanism.inter.broker.protocol=PLAIN
listener.name.sasl_plaintext.plain.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="admin-secret";

listeners: Defines the protocol and port Kafka listens on with SASL_PLAINTEXT for authentication without encryption.

advertised.listeners: The address clients use to connect to Kafka.

security.inter.broker.protocol: Protocol used between Kafka brokers, here also SASL_PLAINTEXT.

sasl.enabled.mechanisms: Enables the PLAIN mechanism for username/password authentication.

sasl.mechanism.inter.broker.protocol: Mechanism used between brokers.

listener.name.sasl_plaintext.plain.sasl.jaas.config: Configures the JAAS login module with username and password for authentication.

Commands

Starts the Kafka broker using the configuration file that includes client authentication settings.

Terminal

kafka-server-start.sh server.properties

Expected Output

[2024-06-01 12:00:00,000] INFO Kafka version : 3.5.0 (org.apache.kafka.common.utils.AppInfoParser)

[2024-06-01 12:00:00,001] INFO Kafka startTimeMs : 1685611200000 (org.apache.kafka.common.utils.AppInfoParser)

[2024-06-01 12:00:05,000] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)

Sends messages to Kafka topic 'test-topic' using SASL PLAIN authentication with username and password.

Terminal

kafka-console-producer.sh --broker-list localhost:9092 --topic test-topic --producer-property sasl.mechanism=PLAIN --producer-property security.protocol=SASL_PLAINTEXT --producer-property sasl.jaas.config='org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="admin-secret";'

Expected Output

No output (command runs silently)

→

--producer-property sasl.mechanism=PLAIN - Specifies the SASL mechanism for authentication.

→

--producer-property security.protocol=SASL_PLAINTEXT - Defines the security protocol to use SASL without encryption.

Reads messages from 'test-topic' using SASL PLAIN authentication with the same username and password.

Terminal

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test-topic --from-beginning --consumer-property sasl.mechanism=PLAIN --consumer-property security.protocol=SASL_PLAINTEXT --consumer-property sasl.jaas.config='org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="admin-secret";'

Expected Output

Hello Kafka

This is a secure message

→

--consumer-property sasl.mechanism=PLAIN - Specifies the SASL mechanism for authentication.

→

--consumer-property security.protocol=SASL_PLAINTEXT - Defines the security protocol to use SASL without encryption.

Key Concept

If you remember nothing else from this pattern, remember: client authentication in Kafka uses SASL mechanisms configured on both broker and client sides to verify identities before allowing access.

Common Mistakes

Not setting the sasl.jaas.config property on the client side.

Without this, the client cannot provide credentials and authentication will fail.

Always include sasl.jaas.config with username and password in client commands or config.

Using PLAINTEXT protocol without SASL enabled in broker config.

Clients will try to authenticate but broker won't accept SASL connections, causing connection errors.

Enable SASL mechanisms and set security.inter.broker.protocol to SASL_PLAINTEXT or SASL_SSL in broker config.

Mismatched usernames or passwords between broker and client.

Authentication will fail because credentials do not match what broker expects.

Ensure usernames and passwords in broker JAAS config and client commands are identical.

Summary

Configure Kafka broker with SASL_PLAINTEXT listener and JAAS login module for client authentication.

Start Kafka broker with the authentication configuration applied.

Use client commands with sasl.jaas.config property to authenticate with username and password when producing or consuming messages.