Client authentication configuration in Kafka - Commands & Configuration

Introduction

Client authentication configuration in Kafka ensures that only trusted users or applications can connect to the Kafka server. It protects your data by requiring clients to prove their identity before sending or receiving messages.

When you want to prevent unauthorized applications from reading or writing Kafka topics.

When you need to secure communication between Kafka clients and brokers in a multi-tenant environment.

When you want to comply with security policies that require encrypted and authenticated connections.

When running Kafka in a cloud or public network where anyone could try to connect.

When you want to audit and control which clients access your Kafka cluster.

Config File - server.properties

server.properties

listeners=SASL_PLAINTEXT://:9092
advertised.listeners=SASL_PLAINTEXT://localhost:9092
security.inter.broker.protocol=SASL_PLAINTEXT
sasl.enabled.mechanisms=PLAIN
sasl.mechanism.inter.broker.protocol=PLAIN
listener.name.sasl_plaintext.plain.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="admin-secret";

listeners: Defines the protocol and port Kafka listens on with SASL_PLAINTEXT for authentication without encryption.

advertised.listeners: The address clients use to connect to Kafka.

security.inter.broker.protocol: Protocol used between Kafka brokers, here also SASL_PLAINTEXT.

sasl.enabled.mechanisms: Enables the PLAIN mechanism for username/password authentication.

sasl.mechanism.inter.broker.protocol: Mechanism used between brokers.

listener.name.sasl_plaintext.plain.sasl.jaas.config: Configures the JAAS login module with username and password for authentication.

Commands

Starts the Kafka broker using the configuration file that includes client authentication settings.

Terminal

kafka-server-start.sh server.properties

Expected Output

[2024-06-01 12:00:00,000] INFO Kafka version : 3.5.0 (org.apache.kafka.common.utils.AppInfoParser)

[2024-06-01 12:00:00,001] INFO Kafka startTimeMs : 1685611200000 (org.apache.kafka.common.utils.AppInfoParser)

[2024-06-01 12:00:05,000] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)

Sends messages to Kafka topic 'test-topic' using SASL PLAIN authentication with username and password.

Terminal

kafka-console-producer.sh --broker-list localhost:9092 --topic test-topic --producer-property sasl.mechanism=PLAIN --producer-property security.protocol=SASL_PLAINTEXT --producer-property sasl.jaas.config='org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="admin-secret";'

Expected Output

No output (command runs silently)

→

--producer-property sasl.mechanism=PLAIN - Specifies the SASL mechanism for authentication.

→

--producer-property security.protocol=SASL_PLAINTEXT - Defines the security protocol to use SASL without encryption.

Reads messages from 'test-topic' using SASL PLAIN authentication with the same username and password.

Terminal

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test-topic --from-beginning --consumer-property sasl.mechanism=PLAIN --consumer-property security.protocol=SASL_PLAINTEXT --consumer-property sasl.jaas.config='org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="admin-secret";'

Expected Output

Hello Kafka

This is a secure message

→

--consumer-property sasl.mechanism=PLAIN - Specifies the SASL mechanism for authentication.

→

--consumer-property security.protocol=SASL_PLAINTEXT - Defines the security protocol to use SASL without encryption.

Key Concept

If you remember nothing else from this pattern, remember: client authentication in Kafka uses SASL mechanisms configured on both broker and client sides to verify identities before allowing access.

Common Mistakes

Not setting the sasl.jaas.config property on the client side.

Without this, the client cannot provide credentials and authentication will fail.

Always include sasl.jaas.config with username and password in client commands or config.

Using PLAINTEXT protocol without SASL enabled in broker config.

Clients will try to authenticate but broker won't accept SASL connections, causing connection errors.

Enable SASL mechanisms and set security.inter.broker.protocol to SASL_PLAINTEXT or SASL_SSL in broker config.

Mismatched usernames or passwords between broker and client.

Authentication will fail because credentials do not match what broker expects.

Ensure usernames and passwords in broker JAAS config and client commands are identical.

Summary

Configure Kafka broker with SASL_PLAINTEXT listener and JAAS login module for client authentication.

Start Kafka broker with the authentication configuration applied.

Use client commands with sasl.jaas.config property to authenticate with username and password when producing or consuming messages.

Practice

1. What is the main purpose of client authentication in Kafka?

easy

A. To verify the identity of clients connecting to the Kafka broker

B. To increase the speed of message delivery

C. To compress messages before sending

D. To store messages permanently on the client side

Solution

1. Step 1: Understand client authentication

   Client authentication is about confirming who is connecting to Kafka to ensure security.

2. Step 2: Identify the purpose in Kafka context

   Kafka uses client authentication to verify client identity before allowing access.

3. Final Answer:

   To verify the identity of clients connecting to the Kafka broker -> Option A

4. Quick Check:

   Client authentication = verify identity [OK]

Hint: Authentication means verifying who you are [OK]

Common Mistakes:

• Confusing authentication with message compression
• Thinking authentication speeds up delivery
• Mixing authentication with data storage

2. Which Kafka client configuration property is used to enable SSL authentication?

easy

A. enable.ssl.auth=true

B. ssl.enabled=true

C. security.protocol=SSL

D. auth.method=ssl

Solution

1. Step 1: Recall Kafka SSL config property

   The correct property to enable SSL is 'security.protocol' set to 'SSL'.

2. Step 2: Check other options for correctness

   Other options are not valid Kafka config keys for SSL authentication.

3. Final Answer:

   security.protocol=SSL -> Option C

4. Quick Check:

   SSL enabled by security.protocol=SSL [OK]

Hint: Use security.protocol=SSL to enable SSL [OK]

Common Mistakes:

• Using non-existent properties like ssl.enabled
• Confusing property names with auth.method
• Adding extra words like enable.ssl.auth

3. Given this Kafka client config snippet:

security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="user" password="pass";

What type of client authentication is configured?

medium

A. SSL without SASL

B. SASL with PLAIN mechanism over SSL

C. SASL with SCRAM mechanism

D. No authentication

Solution

1. Step 1: Analyze security.protocol and sasl.mechanism

   security.protocol=SASL_SSL means SASL authentication over SSL. sasl.mechanism=PLAIN means using PLAIN SASL mechanism.

2. Step 2: Confirm authentication type

   The config uses SASL PLAIN mechanism secured by SSL transport.

3. Final Answer:

   SASL with PLAIN mechanism over SSL -> Option B

4. Quick Check:

   SASL_SSL + PLAIN = SASL PLAIN over SSL [OK]

Hint: SASL_SSL means SASL over SSL, check sasl.mechanism [OK]

Common Mistakes:

• Ignoring sasl.mechanism and assuming only SSL
• Confusing PLAIN with SCRAM mechanism
• Thinking no authentication is set

4. You have this Kafka client config:

security.protocol=SSL
ssl.keystore.location=/path/to/keystore
ssl.keystore.password=wrongpass

The client fails to authenticate. What is the most likely cause?

medium

A. Incorrect keystore password

B. Missing sasl.mechanism property

C. security.protocol should be PLAINTEXT

D. ssl.keystore.location path is not needed

Solution

1. Step 1: Check SSL keystore config

   The keystore password is set to 'wrongpass', which likely does not match the actual password.

2. Step 2: Evaluate other options

   Missing sasl.mechanism is irrelevant for SSL-only config. PLAINTEXT disables security. Keystore location is required.

3. Final Answer:

   Incorrect keystore password -> Option A

4. Quick Check:

   Wrong password causes SSL auth failure [OK]

Hint: Check passwords carefully for SSL keystore [OK]

Common Mistakes:

• Assuming sasl.mechanism is needed for SSL-only
• Changing security.protocol to PLAINTEXT by mistake
• Ignoring keystore location requirement

5. You want to configure a Kafka client to authenticate using SASL SCRAM-SHA-512 over SSL. Which configuration snippet is correct?

hard

A. security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="user" password="pass";

B. security.protocol=SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="user" password="pass";

C. security.protocol=SASL_PLAINTEXT
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="user" password="pass";

D. security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="user" password="pass";

Solution

1. Step 1: Identify correct security.protocol for SASL SCRAM over SSL

   It must be 'SASL_SSL' to use SASL authentication over SSL.

2. Step 2: Match sasl.mechanism and jaas config

   For SCRAM-SHA-512, sasl.mechanism must be 'SCRAM-SHA-512' and jaas config must use ScramLoginModule.

3. Step 3: Verify options

   security.protocol=SASL_SSL sasl.mechanism=SCRAM-SHA-512 sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="user" password="pass"; matches all requirements. security.protocol=SSL sasl.mechanism=SCRAM-SHA-512 sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="user" password="pass"; misses SASL in protocol. security.protocol=SASL_PLAINTEXT sasl.mechanism=SCRAM-SHA-512 sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="user" password="pass"; uses PLAINTEXT transport. security.protocol=SASL_SSL sasl.mechanism=PLAIN sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="user" password="pass"; uses PLAIN mechanism, not SCRAM.

4. Final Answer:

   security.protocol=SASL_SSL sasl.mechanism=SCRAM-SHA-512 sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="user" password="pass"; -> Option D

5. Quick Check:

   SASL_SSL + SCRAM-SHA-512 + ScramLoginModule = security.protocol=SASL_SSL sasl.mechanism=SCRAM-SHA-512 sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="user" password="pass"; [OK]

Hint: Use SASL_SSL with matching SCRAM mechanism and login module [OK]

Common Mistakes:

• Using SSL without SASL for SCRAM
• Mixing PLAIN mechanism with SCRAM config
• Setting security.protocol to PLAINTEXT