HDFS encryption at rest in Hadoop

Introduction

HDFS encryption at rest keeps your data safe by making sure it is stored in a locked, unreadable form on disk. This protects your data from being stolen or seen by unauthorized people.

You want to protect sensitive files like personal information or financial data stored in HDFS.

Your company rules require all stored data to be encrypted for security.

You want to prevent data theft if someone gets physical access to the storage disks.

You need to comply with laws that require data encryption for stored data.

You want to add an extra layer of security beyond user access controls.

Syntax

Hadoop

1. Define encryption zones in HDFS using hdfs crypto commands.
2. Use a Key Management Server (KMS) to manage encryption keys.
3. Configure HDFS to enable encryption at rest in hdfs-site.xml.

Example commands:

# Create an encryption key
hdfs key create my_key

# Create an encryption zone
hdfs crypto -createZone -keyName my_key -path /encrypted_zone

# List encryption zones
hdfs crypto -listZones

# Put files into the encryption zone
hdfs dfs -put localfile /encrypted_zone/

# Read files normally; HDFS decrypts automatically
hdfs dfs -cat /encrypted_zone/localfile

Encryption zones are special directories where files are encrypted automatically.

The Key Management Server (KMS) securely stores and manages encryption keys.

Examples

This example shows creating a key and encryption zone, then storing and reading a file securely.

Hadoop

# Create a key named 'finance_key'
hdfs key create finance_key

# Create an encryption zone at /finance_data using 'finance_key'
hdfs crypto -createZone -keyName finance_key -path /finance_data

# Put a file into the encrypted zone
hdfs dfs -put report.csv /finance_data/

# Read the file normally
hdfs dfs -cat /finance_data/report.csv

If you put files outside an encryption zone, they are not encrypted at rest.

Hadoop

# What if the encryption zone does not exist?
# Trying to put a file into a non-encrypted directory stores it unencrypted.
hdfs dfs -put data.txt /non_encrypted_dir/

# To encrypt, create an encryption zone first.
hdfs crypto -createZone -keyName finance_key -path /non_encrypted_dir

Encryption zones require a valid key. Create the key before the zone.

Hadoop

# What if the key does not exist?
# Creating an encryption zone with a missing key will fail.
hdfs crypto -createZone -keyName missing_key -path /new_zone

# You must create the key first:
hdfs key create missing_key
hdfs crypto -createZone -keyName missing_key -path /new_zone

Encryption zones can exist without files; they encrypt files when added.

Hadoop

# What if the encryption zone is empty?
# An empty encryption zone is allowed and ready to store encrypted files.
hdfs crypto -listZones
# Shows the zone even if no files are inside yet.

Sample Program

This script creates a key and encryption zone, uploads a file, lists the zone, and reads the file to show encryption at rest in action.

Hadoop

# This is a shell script example to demonstrate HDFS encryption at rest

# Step 1: Create an encryption key
hdfs key create test_key

# Step 2: Create an encryption zone
hdfs crypto -createZone -keyName test_key -path /test_encrypted_zone

# Step 3: Show existing encryption zones
hdfs crypto -listZones

# Step 4: Put a file into the encryption zone
echo "Hello, encrypted HDFS!" > localfile.txt
hdfs dfs -put localfile.txt /test_encrypted_zone/

# Step 5: List files in the encryption zone
hdfs dfs -ls /test_encrypted_zone

# Step 6: Read the file (HDFS decrypts automatically)
hdfs dfs -cat /test_encrypted_zone/localfile.txt

Important Notes

Encryption at rest adds some overhead but protects data if disks are stolen.

Time complexity: Encryption/decryption happens automatically and efficiently during file read/write.

Common mistake: Forgetting to create encryption zones means files are stored unencrypted.

Use encryption zones when you want automatic encryption for all files in a directory. Use file-level encryption for more control but more complexity.

Summary

HDFS encryption at rest protects stored data by encrypting files on disk.

Encryption zones are special directories that automatically encrypt files inside them.

Keys are managed by a Key Management Server and must be created before zones.