HDFS encryption at rest in Hadoop - Step-by-Step Execution

Concept Flow - HDFS encryption at rest

Start: Data write request

↓

Client requests encryption key

↓

Key Management Server (KMS) provides key

↓

Data encrypted by client using key

↓

Encrypted data written to HDFS storage

↓

Data stored securely on disk

↓

Read request

↓

Client requests decryption key from KMS

↓

Data decrypted on client side

↓

Data delivered to user/application

Data is encrypted before storing on disk using keys from a Key Management Server, ensuring data is secure at rest and decrypted only when accessed.

Execution Sample

Hadoop

# Pseudocode for HDFS encryption at rest
key = KMS.getEncryptionKey()
encrypted_data = encrypt(data, key)
HDFS.write(encrypted_data)

read_key = KMS.getDecryptionKey()
decrypted_data = decrypt(HDFS.read(), read_key)

This code shows how data is encrypted before writing to HDFS and decrypted after reading, using keys from the KMS.

Execution Table

Step	Action	Key Used	Data State	Storage State
1	Request encryption key from KMS	None	Plain data	Empty
2	Encrypt data with key	Encryption Key	Encrypted data	Empty
3	Write encrypted data to HDFS	Encryption Key	Encrypted data	Encrypted data stored
4	Request decryption key from KMS	None	Encrypted data	Encrypted data stored
5	Read encrypted data from HDFS	None	Encrypted data	Encrypted data stored
6	Decrypt data with key	Decryption Key	Plain data	Encrypted data stored
7	Deliver plain data to user	Decryption Key	Plain data	Encrypted data stored

💡 Data is securely stored encrypted on disk; decrypted only when accessed with proper key.

Variable Tracker

Variable	Start	After Step 2	After Step 3	After Step 6	Final
key	None	Encryption Key	Encryption Key	Decryption Key	Decryption Key
data	Plain data	Encrypted data	Encrypted data	Plain data	Plain data
storage	Empty	Empty	Encrypted data stored	Encrypted data stored	Encrypted data stored

Key Moments - 3 Insights

Why is the data encrypted before writing to HDFS storage?

Encrypting data before writing ensures that data at rest on disk is protected from unauthorized access, as shown in execution_table step 3 where encrypted data is stored.

How does the client get the encryption and decryption keys?

The client requests keys from the Key Management Server (KMS) as shown in steps 1 and 4 in the execution_table, ensuring keys are centrally managed and secure.

Is the data decrypted on the HDFS storage node?

No, data remains encrypted on storage. Decryption happens on the client side after reading, as shown in steps 5 and 6 in the execution_table.

Visual Quiz - 3 Questions

Look at the execution_table, at which step is the data actually stored encrypted on disk?

A Step 5

B Step 2

C Step 3

D Step 6

💡 Hint

Check the 'Storage State' column in execution_table rows.

According to variable_tracker, what is the state of 'data' after Step 6?

A Encrypted data

B Plain data

C None

D Partially encrypted

💡 Hint

Look at the 'data' row under 'After Step 6' in variable_tracker.

If the client did not request the decryption key from KMS, what would happen at Step 6?

A Data would remain encrypted and unusable

B Data would be lost

C Data would be decrypted anyway

D Data would be stored unencrypted

💡 Hint

Refer to the role of keys in execution_table steps 4 and 6.

Concept Snapshot

HDFS encryption at rest protects data by encrypting it before storage.
Keys are managed by a Key Management Server (KMS).
Data is encrypted on the client side before writing.
Encrypted data is stored on disk securely.
Decryption happens on the client side after reading.
This ensures data is safe even if storage is compromised.

Full Transcript

HDFS encryption at rest means data is encrypted before it is saved on disk. The client asks the Key Management Server for an encryption key. Using this key, the client encrypts the data. Then, the encrypted data is written to HDFS storage. When reading, the client requests the decryption key from the KMS. The encrypted data is read from storage and decrypted on the client side. This process keeps data safe on disk and only readable by authorized users with keys.

Practice

1. What is the main purpose of HDFS encryption at rest?

easy

A. To speed up data processing in Hadoop

B. To protect stored data by encrypting files on disk

C. To compress files for storage efficiency

D. To replicate data across nodes for fault tolerance

Solution

1. Step 1: Understand the concept of encryption at rest

   Encryption at rest means protecting data stored on disk by converting it into a secure format.

2. Step 2: Identify the purpose of HDFS encryption at rest

   HDFS encryption at rest specifically encrypts files on disk to prevent unauthorized access.

3. Final Answer:

   To protect stored data by encrypting files on disk -> Option B

4. Quick Check:

   Encryption at rest = Protect data on disk [OK]

Hint: Encryption at rest means data is safe on disk [OK]

Common Mistakes:

• Confusing encryption with compression
• Thinking encryption speeds up processing
• Mixing encryption with data replication

2. Which of the following is the correct command to create an encryption zone in HDFS?

easy

A. hdfs dfsadmin -createEncryptionZone -keyName mykey -path /zone

B. hdfs dfs -createEncryptionZone -keyName mykey -path /zone

C. hdfs crypto -createZone -keyName mykey -path /zone

D. hdfs crypto -createEncryptionZone -keyName mykey -path /zone

Solution

1. Step 1: Recall the correct syntax for creating encryption zones

   The command to create an encryption zone uses 'hdfs dfsadmin' with the '-createEncryptionZone' option.

2. Step 2: Match the command options

   hdfs dfsadmin -createEncryptionZone -keyName mykey -path /zone correctly uses 'hdfs dfsadmin -createEncryptionZone -keyName mykey -path /zone'. Other options use incorrect commands or flags.

3. Final Answer:

   hdfs dfsadmin -createEncryptionZone -keyName mykey -path /zone -> Option A

4. Quick Check:

   Encryption zone creation uses dfsadmin command [OK]

Hint: Use 'hdfs dfsadmin -createEncryptionZone' to create zones [OK]

Common Mistakes:

• Using 'hdfs dfs' instead of 'hdfs dfsadmin'
• Incorrect flag names like '-createZone'
• Mixing crypto commands with dfsadmin commands

3. Given the following commands executed in order:
hdfs key create mykey
hdfs dfsadmin -createEncryptionZone -keyName mykey -path /secure
What will happen if you try to write a file to /secure?

medium

A. The file will be stored encrypted automatically

B. The file will be stored unencrypted

C. An error will occur because the key does not exist

D. The file will be compressed but not encrypted

Solution

1. Step 1: Understand key creation and encryption zone setup

   The key 'mykey' is created first, then an encryption zone is created at '/secure' using that key.

2. Step 2: Analyze file write behavior in encryption zones

   Files written inside an encryption zone are automatically encrypted using the zone's key.

3. Final Answer:

   The file will be stored encrypted automatically -> Option A

4. Quick Check:

   Encryption zone + key = automatic encryption [OK]

Hint: Files in encryption zones encrypt automatically [OK]

Common Mistakes:

• Assuming files need manual encryption
• Thinking key creation is after zone creation
• Confusing compression with encryption

4. You run the command hdfs dfsadmin -createEncryptionZone -keyName unknownkey -path /data but get an error. What is the most likely cause?

medium

A. The encryption zone path /data does not exist

B. The command syntax is incorrect

C. The key 'unknownkey' has not been created in the Key Management Server

D. Encryption zones cannot be created on /data directory

Solution

1. Step 1: Check prerequisites for creating encryption zones

   An encryption zone requires a valid key that must be created beforehand in the Key Management Server.

2. Step 2: Identify cause of error

   The error likely occurs because 'unknownkey' does not exist, so the zone cannot be created with a missing key.

3. Final Answer:

   The key 'unknownkey' has not been created in the Key Management Server -> Option C

4. Quick Check:

   Missing key causes encryption zone creation error [OK]

Hint: Keys must exist before creating encryption zones [OK]

Common Mistakes:

• Assuming path existence causes error
• Ignoring key creation step
• Thinking command syntax is wrong

5. You want to secure sensitive data in HDFS by creating an encryption zone. Which sequence of steps is correct?

hard

A. Write files, create encryption zone, then create key

B. Write files, create key, then create encryption zone

C. Create encryption zone, then create key, then write files

D. Create key, create encryption zone, then write files

Solution

1. Step 1: Understand the order of operations for encryption setup

   First, a key must be created in the Key Management Server to be used for encryption.

2. Step 2: Create the encryption zone using the created key

   After the key exists, create the encryption zone specifying that key.

3. Step 3: Write files inside the encryption zone to have them encrypted automatically

   Files written after zone creation will be encrypted at rest.

4. Final Answer:

   Create key, create encryption zone, then write files -> Option D

5. Quick Check:

   Key -> Zone -> Files = Correct sequence [OK]

Hint: Create key before zone, then write files inside zone [OK]

Common Mistakes:

• Creating zone before key
• Writing files before zone creation
• Assuming files encrypt without zone