HDFS encryption at rest in Hadoop - Step-by-Step Execution

Concept Flow - HDFS encryption at rest

Start: Data write request

↓

Client requests encryption key

↓

Key Management Server (KMS) provides key

↓

Data encrypted by client using key

↓

Encrypted data written to HDFS storage

↓

Data stored securely on disk

↓

Read request

↓

Client requests decryption key from KMS

↓

Data decrypted on client side

↓

Data delivered to user/application

Data is encrypted before storing on disk using keys from a Key Management Server, ensuring data is secure at rest and decrypted only when accessed.

Execution Sample

Hadoop

# Pseudocode for HDFS encryption at rest
key = KMS.getEncryptionKey()
encrypted_data = encrypt(data, key)
HDFS.write(encrypted_data)

read_key = KMS.getDecryptionKey()
decrypted_data = decrypt(HDFS.read(), read_key)

This code shows how data is encrypted before writing to HDFS and decrypted after reading, using keys from the KMS.

Execution Table

Step	Action	Key Used	Data State	Storage State
1	Request encryption key from KMS	None	Plain data	Empty
2	Encrypt data with key	Encryption Key	Encrypted data	Empty
3	Write encrypted data to HDFS	Encryption Key	Encrypted data	Encrypted data stored
4	Request decryption key from KMS	None	Encrypted data	Encrypted data stored
5	Read encrypted data from HDFS	None	Encrypted data	Encrypted data stored
6	Decrypt data with key	Decryption Key	Plain data	Encrypted data stored
7	Deliver plain data to user	Decryption Key	Plain data	Encrypted data stored

💡 Data is securely stored encrypted on disk; decrypted only when accessed with proper key.

Variable Tracker

Variable	Start	After Step 2	After Step 3	After Step 6	Final
key	None	Encryption Key	Encryption Key	Decryption Key	Decryption Key
data	Plain data	Encrypted data	Encrypted data	Plain data	Plain data
storage	Empty	Empty	Encrypted data stored	Encrypted data stored	Encrypted data stored

Key Moments - 3 Insights

Why is the data encrypted before writing to HDFS storage?

Encrypting data before writing ensures that data at rest on disk is protected from unauthorized access, as shown in execution_table step 3 where encrypted data is stored.

How does the client get the encryption and decryption keys?

The client requests keys from the Key Management Server (KMS) as shown in steps 1 and 4 in the execution_table, ensuring keys are centrally managed and secure.

Is the data decrypted on the HDFS storage node?

No, data remains encrypted on storage. Decryption happens on the client side after reading, as shown in steps 5 and 6 in the execution_table.

Visual Quiz - 3 Questions

Look at the execution_table, at which step is the data actually stored encrypted on disk?

A Step 5

B Step 2

C Step 3

D Step 6

💡 Hint

Check the 'Storage State' column in execution_table rows.

According to variable_tracker, what is the state of 'data' after Step 6?

A Encrypted data

B Plain data

C None

D Partially encrypted

💡 Hint

Look at the 'data' row under 'After Step 6' in variable_tracker.

If the client did not request the decryption key from KMS, what would happen at Step 6?

A Data would remain encrypted and unusable

B Data would be lost

C Data would be decrypted anyway

D Data would be stored unencrypted

💡 Hint

Refer to the role of keys in execution_table steps 4 and 6.

Concept Snapshot

HDFS encryption at rest protects data by encrypting it before storage.
Keys are managed by a Key Management Server (KMS).
Data is encrypted on the client side before writing.
Encrypted data is stored on disk securely.
Decryption happens on the client side after reading.
This ensures data is safe even if storage is compromised.

Full Transcript

HDFS encryption at rest means data is encrypted before it is saved on disk. The client asks the Key Management Server for an encryption key. Using this key, the client encrypts the data. Then, the encrypted data is written to HDFS storage. When reading, the client requests the decryption key from the KMS. The encrypted data is read from storage and decrypted on the client side. This process keeps data safe on disk and only readable by authorized users with keys.