Encryption at rest in Kafka - Commands & Configuration

Introduction

Encryption at rest protects your data stored on disk by converting it into a secret code. This stops unauthorized people from reading your data even if they get access to the storage.

When you want to protect sensitive messages stored in Kafka topics from being read by unauthorized users.

When your company policy requires all stored data to be encrypted for compliance reasons.

When you want to secure Kafka logs and data files on disk in case of server theft or hacking.

When running Kafka on shared infrastructure and you want to isolate your data from other tenants.

When you want to add an extra layer of security beyond network encryption for Kafka data.

Config File - server.properties

server.properties

log.dirs=/var/lib/kafka/data

# Enable encryption at rest using the file system encryption
# Kafka itself does not provide built-in encryption at rest,
# so this example assumes using OS-level encryption like dm-crypt or LUKS.

# Example configuration for SSL encryption for data in transit (not at rest):
ssl.keystore.location=/var/private/ssl/kafka.server.keystore.jks
ssl.keystore.password=changeit
ssl.key.password=changeit
ssl.truststore.location=/var/private/ssl/kafka.server.truststore.jks
ssl.truststore.password=changeit

# To ensure data is encrypted at rest, configure your disk or volume with encryption tools
# outside Kafka, such as LUKS on Linux or BitLocker on Windows.

# This file shows Kafka server properties; encryption at rest is handled by the OS or storage layer.

This configuration file sets Kafka server properties.

Kafka does not have built-in encryption at rest, so you must encrypt the disk or volume where Kafka stores data.

The file shows SSL settings for encrypting data in transit, which is different from encryption at rest.

To encrypt at rest, use operating system or hardware encryption tools on the Kafka data directory.

Commands

Check the block devices and their mount points to identify the disk where Kafka data is stored.

Terminal

lsblk -o NAME,MOUNTPOINT,TYPE,UUID

Expected Output

NAME   MOUNTPOINT          TYPE UUID

sda                        disk 1234-5678

├─sda1 /                    part 1234-5678

└─sda2 /var/lib/kafka/data  part 8765-4321

Encrypt the Kafka data partition using LUKS to enable encryption at rest on the disk.

Terminal

sudo cryptsetup luksFormat /dev/sda2

Expected Output

WARNING: Device /dev/sda2 already contains a filesystem (ext4).

Are you sure you want to overwrite it? (Type uppercase YES): YES

Enter passphrase for /dev/sda2:

Verify passphrase:

Open the encrypted partition to make it accessible as a decrypted device.

Terminal

sudo cryptsetup luksOpen /dev/sda2 kafka_data_enc

Expected Output

No output (command runs silently)

Format the decrypted device with ext4 filesystem to store Kafka data.

Terminal

sudo mkfs.ext4 /dev/mapper/kafka_data_enc

Expected Output

mke2fs 1.45.6 (20-Mar-2020)

Creating filesystem with 1310720 4k blocks and 327680 inodes

Filesystem UUID: 123e4567-e89b-12d3-a456-426614174000

Superblock backups stored on blocks: 

	32768, 98304, 163840, 229376, 294912, 819200, 884736



Allocating group tables: done

Writing inode tables: done

Creating journal (16384 blocks): done

Writing superblocks and filesystem accounting information: done

Mount the encrypted and formatted device to Kafka's data directory so Kafka stores data encrypted on disk.

Terminal

sudo mount /dev/mapper/kafka_data_enc /var/lib/kafka/data

Expected Output

No output (command runs silently)

Restart Kafka to ensure it uses the encrypted data directory for storing data.

Terminal

systemctl restart kafka

Expected Output

No output (command runs silently)

Key Concept

Encryption at rest means protecting stored data by encrypting the disk or volume where Kafka saves its data, usually handled outside Kafka itself.

Common Mistakes

Trying to enable encryption at rest only by configuring Kafka server properties.

Kafka does not provide built-in encryption at rest; it only supports encryption in transit via SSL/TLS.

Use operating system or hardware disk encryption tools like LUKS or BitLocker to encrypt the Kafka data directory.

Encrypting the disk but not mounting it to Kafka's data directory.

Kafka will continue writing to the unencrypted directory, leaving data unprotected.

After encrypting and formatting the disk, mount it to the Kafka data directory before starting Kafka.

Not restarting Kafka after changing the data directory mount.

Kafka may not recognize the new encrypted storage and continue using old paths.

Always restart Kafka after mounting the encrypted volume to ensure it uses the encrypted storage.

Summary

Kafka does not have built-in encryption at rest; use OS-level disk encryption to protect stored data.

Encrypt the disk or partition where Kafka stores data using tools like LUKS, then mount it to Kafka's data directory.

Restart Kafka after mounting the encrypted disk to ensure data is stored securely.