HDFS encryption at rest in Hadoop - Mini Project: Build & Apply

HDFS Encryption at Rest Setup and Verification

📖 Scenario: You work in a company that stores sensitive data on Hadoop Distributed File System (HDFS). To protect this data, you need to enable encryption at rest. This means the data files on disk are encrypted, so even if someone accesses the storage directly, they cannot read the data without the encryption keys.In this project, you will set up a simple HDFS encryption zone, configure the key management, and verify that files stored in the encryption zone are encrypted.

🎯 Goal: Set up HDFS encryption at rest by creating a key in the Key Management Server (KMS), creating an encryption zone in HDFS, and verifying that files stored in this zone are encrypted.

📋 What You'll Learn

Create an encryption key named my_key in the KMS

Create an HDFS directory /encrypted_zone

Create an encryption zone on /encrypted_zone using my_key

Copy a test file into the encryption zone

Verify the file is encrypted by checking its encryption status

💡 Why This Matters

🌍 Real World

Many companies store sensitive data in Hadoop clusters. Encrypting data at rest protects it from unauthorized access if disks are stolen or accessed directly.

💼 Career

Data engineers and Hadoop administrators need to know how to configure and verify HDFS encryption to meet security compliance requirements.

1

Create an encryption key in KMS

Use the hadoop key command to create an encryption key called my_key in the Key Management Server (KMS). This key will be used to encrypt files in HDFS.

Hadoop

# Create an encryption key named my_key
# Your code here

Hint

Use the hadoop key create command followed by the key name.

2

Create an HDFS directory for the encryption zone

Create a directory in HDFS at /encrypted_zone using the hdfs dfs -mkdir command. This directory will be used as the encryption zone.

Hadoop

hadoop key create my_key
# Create the HDFS directory /encrypted_zone
# Your code here

Hint

Use hdfs dfs -mkdir /encrypted_zone to create the directory.

3

Create the encryption zone on the directory

Use the hdfs crypto -createZone command to create an encryption zone on /encrypted_zone using the key my_key.

Hadoop

hadoop key create my_key
hdfs dfs -mkdir /encrypted_zone
# Create encryption zone on /encrypted_zone with key my_key
# Your code here

Hint

Use hdfs crypto -createZone -keyName my_key -path /encrypted_zone to create the encryption zone.

4

Copy a test file and verify encryption

Copy a local file named testfile.txt into /encrypted_zone using hdfs dfs -put. Then, use hdfs crypto -listZones and hdfs dfs -ls to verify the file is inside the encryption zone and encrypted.

Hadoop

hadoop key create my_key
hdfs dfs -mkdir /encrypted_zone
hdfs crypto -createZone -keyName my_key -path /encrypted_zone
# Copy testfile.txt to /encrypted_zone and verify encryption
# Your code here

Hint

Use hdfs dfs -put to copy the file, then hdfs crypto -listZones and hdfs dfs -ls to check encryption.

Practice

1. What is the main purpose of HDFS encryption at rest?

easy

A. To speed up data processing in Hadoop

B. To protect stored data by encrypting files on disk

C. To compress files for storage efficiency

D. To replicate data across nodes for fault tolerance

Solution

1. Step 1: Understand the concept of encryption at rest

   Encryption at rest means protecting data stored on disk by converting it into a secure format.

2. Step 2: Identify the purpose of HDFS encryption at rest

   HDFS encryption at rest specifically encrypts files on disk to prevent unauthorized access.

3. Final Answer:

   To protect stored data by encrypting files on disk -> Option B

4. Quick Check:

   Encryption at rest = Protect data on disk [OK]

Hint: Encryption at rest means data is safe on disk [OK]

Common Mistakes:

• Confusing encryption with compression
• Thinking encryption speeds up processing
• Mixing encryption with data replication

2. Which of the following is the correct command to create an encryption zone in HDFS?

easy

A. hdfs dfsadmin -createEncryptionZone -keyName mykey -path /zone

B. hdfs dfs -createEncryptionZone -keyName mykey -path /zone

C. hdfs crypto -createZone -keyName mykey -path /zone

D. hdfs crypto -createEncryptionZone -keyName mykey -path /zone

Solution

1. Step 1: Recall the correct syntax for creating encryption zones

   The command to create an encryption zone uses 'hdfs dfsadmin' with the '-createEncryptionZone' option.

2. Step 2: Match the command options

   hdfs dfsadmin -createEncryptionZone -keyName mykey -path /zone correctly uses 'hdfs dfsadmin -createEncryptionZone -keyName mykey -path /zone'. Other options use incorrect commands or flags.

3. Final Answer:

   hdfs dfsadmin -createEncryptionZone -keyName mykey -path /zone -> Option A

4. Quick Check:

   Encryption zone creation uses dfsadmin command [OK]

Hint: Use 'hdfs dfsadmin -createEncryptionZone' to create zones [OK]

Common Mistakes:

• Using 'hdfs dfs' instead of 'hdfs dfsadmin'
• Incorrect flag names like '-createZone'
• Mixing crypto commands with dfsadmin commands

3. Given the following commands executed in order:
hdfs key create mykey
hdfs dfsadmin -createEncryptionZone -keyName mykey -path /secure
What will happen if you try to write a file to /secure?

medium

A. The file will be stored encrypted automatically

B. The file will be stored unencrypted

C. An error will occur because the key does not exist

D. The file will be compressed but not encrypted

Solution

1. Step 1: Understand key creation and encryption zone setup

   The key 'mykey' is created first, then an encryption zone is created at '/secure' using that key.

2. Step 2: Analyze file write behavior in encryption zones

   Files written inside an encryption zone are automatically encrypted using the zone's key.

3. Final Answer:

   The file will be stored encrypted automatically -> Option A

4. Quick Check:

   Encryption zone + key = automatic encryption [OK]

Hint: Files in encryption zones encrypt automatically [OK]

Common Mistakes:

• Assuming files need manual encryption
• Thinking key creation is after zone creation
• Confusing compression with encryption

4. You run the command hdfs dfsadmin -createEncryptionZone -keyName unknownkey -path /data but get an error. What is the most likely cause?

medium

A. The encryption zone path /data does not exist

B. The command syntax is incorrect

C. The key 'unknownkey' has not been created in the Key Management Server

D. Encryption zones cannot be created on /data directory

Solution

1. Step 1: Check prerequisites for creating encryption zones

   An encryption zone requires a valid key that must be created beforehand in the Key Management Server.

2. Step 2: Identify cause of error

   The error likely occurs because 'unknownkey' does not exist, so the zone cannot be created with a missing key.

3. Final Answer:

   The key 'unknownkey' has not been created in the Key Management Server -> Option C

4. Quick Check:

   Missing key causes encryption zone creation error [OK]

Hint: Keys must exist before creating encryption zones [OK]

Common Mistakes:

• Assuming path existence causes error
• Ignoring key creation step
• Thinking command syntax is wrong

5. You want to secure sensitive data in HDFS by creating an encryption zone. Which sequence of steps is correct?

hard

A. Write files, create encryption zone, then create key

B. Write files, create key, then create encryption zone

C. Create encryption zone, then create key, then write files

D. Create key, create encryption zone, then write files

Solution

1. Step 1: Understand the order of operations for encryption setup

   First, a key must be created in the Key Management Server to be used for encryption.

2. Step 2: Create the encryption zone using the created key

   After the key exists, create the encryption zone specifying that key.

3. Step 3: Write files inside the encryption zone to have them encrypted automatically

   Files written after zone creation will be encrypted at rest.

4. Final Answer:

   Create key, create encryption zone, then write files -> Option D

5. Quick Check:

   Key -> Zone -> Files = Correct sequence [OK]

Hint: Create key before zone, then write files inside zone [OK]

Common Mistakes:

• Creating zone before key
• Writing files before zone creation
• Assuming files encrypt without zone