HDFS encryption at rest in Hadoop - Mini Project: Build & Apply

HDFS Encryption at Rest Setup and Verification

📖 Scenario: You work in a company that stores sensitive data on Hadoop Distributed File System (HDFS). To protect this data, you need to enable encryption at rest. This means the data files on disk are encrypted, so even if someone accesses the storage directly, they cannot read the data without the encryption keys.In this project, you will set up a simple HDFS encryption zone, configure the key management, and verify that files stored in the encryption zone are encrypted.

🎯 Goal: Set up HDFS encryption at rest by creating a key in the Key Management Server (KMS), creating an encryption zone in HDFS, and verifying that files stored in this zone are encrypted.

📋 What You'll Learn

Create an encryption key named my_key in the KMS

Create an HDFS directory /encrypted_zone

Create an encryption zone on /encrypted_zone using my_key

Copy a test file into the encryption zone

Verify the file is encrypted by checking its encryption status

💡 Why This Matters

🌍 Real World

Many companies store sensitive data in Hadoop clusters. Encrypting data at rest protects it from unauthorized access if disks are stolen or accessed directly.

💼 Career

Data engineers and Hadoop administrators need to know how to configure and verify HDFS encryption to meet security compliance requirements.

1

Create an encryption key in KMS

Use the hadoop key command to create an encryption key called my_key in the Key Management Server (KMS). This key will be used to encrypt files in HDFS.

Hadoop

# Create an encryption key named my_key
# Your code here

Need a hint?

Use the hadoop key create command followed by the key name.

2

Create an HDFS directory for the encryption zone

Create a directory in HDFS at /encrypted_zone using the hdfs dfs -mkdir command. This directory will be used as the encryption zone.

Hadoop

hadoop key create my_key
# Create the HDFS directory /encrypted_zone
# Your code here

Need a hint?

Use hdfs dfs -mkdir /encrypted_zone to create the directory.

3

Create the encryption zone on the directory

Use the hdfs crypto -createZone command to create an encryption zone on /encrypted_zone using the key my_key.

Hadoop

hadoop key create my_key
hdfs dfs -mkdir /encrypted_zone
# Create encryption zone on /encrypted_zone with key my_key
# Your code here

Need a hint?

Use hdfs crypto -createZone -keyName my_key -path /encrypted_zone to create the encryption zone.

4

Copy a test file and verify encryption

Copy a local file named testfile.txt into /encrypted_zone using hdfs dfs -put. Then, use hdfs crypto -listZones and hdfs dfs -ls to verify the file is inside the encryption zone and encrypted.

Hadoop

hadoop key create my_key
hdfs dfs -mkdir /encrypted_zone
hdfs crypto -createZone -keyName my_key -path /encrypted_zone
# Copy testfile.txt to /encrypted_zone and verify encryption
# Your code here

Need a hint?

Use hdfs dfs -put to copy the file, then hdfs crypto -listZones and hdfs dfs -ls to check encryption.