Encryption at rest in Kafka - Mini Project: Build & Apply

Encryption at rest with Kafka

📖 Scenario: You are setting up a Kafka topic to securely store messages by enabling encryption at rest. This means the data saved on disk will be encrypted to protect it from unauthorized access.

🎯 Goal: You will configure a Kafka topic with encryption at rest enabled by setting the correct configuration properties.

📋 What You'll Learn

Create a Kafka topic configuration dictionary with the exact name topic_config and include the encryption settings.

Add a variable called encryption_algorithm with the value "AES/GCM/NoPadding".

Use the topic_config dictionary to set the encryption algorithm and enable encryption.

Print the topic_config dictionary to show the final configuration.

💡 Why This Matters

🌍 Real World

Encrypting data at rest in Kafka protects sensitive information stored on disk from unauthorized access or theft.

💼 Career

Understanding how to configure encryption at rest is important for roles in data engineering, security, and system administration working with Kafka.

1

Create Kafka topic configuration dictionary

Create a dictionary called topic_config with these exact entries: "encryption.enabled": "true" and "encryption.key.provider": "local".

Kafka

# Create the topic_config dictionary with encryption enabled and key provider
# Your code here

Hint

Use curly braces {} to create a dictionary with the exact keys and values.

2

Add encryption algorithm variable

Add a variable called encryption_algorithm and set it to the string "AES/GCM/NoPadding".

Kafka

topic_config = {"encryption.enabled": "true", "encryption.key.provider": "local"}
# Add the encryption_algorithm variable
# Your code here

Hint

Use a simple assignment statement to create the variable with the exact string value.

3

Add encryption algorithm to topic configuration

Add the encryption_algorithm variable value to the topic_config dictionary with the key "encryption.algorithm".

Kafka

topic_config = {"encryption.enabled": "true", "encryption.key.provider": "local"}
encryption_algorithm = "AES/GCM/NoPadding"
# Add encryption_algorithm to topic_config with key "encryption.algorithm"
# Your code here

Hint

Use the dictionary key assignment syntax to add the new key and value.

4

Print the final topic configuration

Print the topic_config dictionary to display the final encryption configuration.

Kafka

topic_config = {"encryption.enabled": "true", "encryption.key.provider": "local"}
encryption_algorithm = "AES/GCM/NoPadding"
topic_config["encryption.algorithm"] = encryption_algorithm
# Print the topic_config dictionary
# Your code here

Hint

Use print(topic_config) to show the dictionary.

Practice

1. What does encryption at rest mean in the context of Kafka?

easy

A. Encrypting data while it is being sent between Kafka brokers

B. Protecting stored Kafka data by making it unreadable without a key

C. Compressing Kafka messages to save disk space

D. Authenticating users before they access Kafka topics

Solution

1. Step 1: Understand encryption at rest

   Encryption at rest means data stored on disk is encrypted to prevent unauthorized reading.

2. Step 2: Differentiate from other security concepts

   Encryption in transit protects data moving between systems, not stored data.

3. Final Answer:

   Protecting stored Kafka data by making it unreadable without a key -> Option B

4. Quick Check:

   Encryption at rest = Protect stored data [OK]

Hint: Encryption at rest means stored data is unreadable without keys [OK]

Common Mistakes:

• Confusing encryption at rest with encryption in transit
• Thinking compression is encryption
• Mixing authentication with encryption

2. Which of the following is the correct way to enable encryption at rest in Kafka?

easy

A. Use disk-level encryption or external tools since Kafka does not encrypt data on disk by default

B. Add encryption.at.rest=true in the Kafka topic configuration

C. Set ssl.enabled=true in the Kafka broker configuration

D. Configure kafka.security.encryption=true in the producer settings

Solution

1. Step 1: Check Kafka's default encryption behavior

   Kafka does not encrypt data on disk by default; it requires external disk encryption or tools.

2. Step 2: Review configuration options

   Options A, B, and D are not valid Kafka configurations for encryption at rest.

3. Final Answer:

   Use disk-level encryption or external tools since Kafka does not encrypt data on disk by default -> Option A

4. Quick Check:

   Kafka disk encryption = external tools [OK]

Hint: Kafka needs external disk encryption for data at rest [OK]

Common Mistakes:

• Assuming SSL enables encryption at rest
• Trying to set non-existent Kafka config keys
• Confusing topic or producer configs with disk encryption

3. Given the following Kafka broker configuration snippet, what is the effect on data security?

listeners=PLAINTEXT://:9092
ssl.enabled.protocols=TLSv1.2,TLSv1.3
ssl.keystore.location=/var/private/keystore.jks
ssl.keystore.password=changeit

medium

A. Data is encrypted both on disk and during transmission

B. Data is encrypted on disk but not during transmission

C. Data is encrypted during transmission but not on disk

D. No encryption is enabled

Solution

1. Step 1: Analyze the configuration

   The listeners are set to PLAINTEXT://:9092, which means connections are not encrypted (no SSL/TLS in transit). The SSL keystore settings are present but not activated for this listener.

2. Step 2: Confirm no encryption at rest

   There are no settings for disk encryption; data on disk is unencrypted.

3. Final Answer:

   Data is encrypted during transmission but not on disk -> Option C

4. Quick Check:

   PLAINTEXT listener + no disk encryption = no encryption at rest, but SSL settings indicate encryption possible if listener changed [OK]

Hint: PLAINTEXT listener means no encryption in transit or at rest [OK]

Common Mistakes:

• Assuming SSL encrypts data on disk
• Ignoring the PLAINTEXT listener setting
• Confusing keystore with disk encryption

4. You want to enable encryption at rest for Kafka data. You added the following to your broker config:

encryption.at.rest=true

But it does not work. What is the likely problem?

medium

A. You need to restart the Kafka producer, not the broker

B. The config key is case sensitive and should be Encryption.At.Rest

C. You must set ssl.enabled=true instead

D. Kafka does not support encryption.at.rest config; disk encryption must be external

Solution

1. Step 1: Verify Kafka config support

   Kafka does not have a built-in encryption.at.rest configuration option.

2. Step 2: Understand how to enable encryption at rest

   Encryption at rest requires external disk encryption tools or file system encryption.

3. Final Answer:

   Kafka does not support encryption.at.rest config; disk encryption must be external -> Option D

4. Quick Check:

   No native encryption at rest config in Kafka [OK]

Hint: Kafka needs external disk encryption, no native config key [OK]

Common Mistakes:

• Trying to use non-existent Kafka config keys
• Restarting wrong components
• Confusing SSL with encryption at rest

5. You want to ensure Kafka data is encrypted at rest on a Linux server. Which approach correctly applies encryption at rest while keeping Kafka running smoothly?

hard

A. Use Linux LUKS disk encryption on the Kafka data directory and configure Kafka normally

B. Enable SSL in Kafka broker config to encrypt data on disk

C. Set Kafka topic config encryption.enabled=true to encrypt data at rest

D. Use Kafka producer encryption settings to encrypt data before writing to disk

Solution

1. Step 1: Identify how to encrypt data at rest on Linux

   Linux LUKS provides full disk encryption, protecting Kafka data on disk transparently.

2. Step 2: Confirm Kafka compatibility

   Kafka works normally on encrypted disks without special config; SSL and topic configs do not encrypt at rest.

3. Final Answer:

   Use Linux LUKS disk encryption on the Kafka data directory and configure Kafka normally -> Option A

4. Quick Check:

   Disk encryption tool + normal Kafka = encryption at rest [OK]

Hint: Use OS-level disk encryption for Kafka data at rest [OK]

Common Mistakes:

• Thinking SSL encrypts data on disk
• Expecting Kafka topic configs to encrypt data at rest
• Trying to encrypt data only at producer level