Wire encryption for data in transit in Hadoop - Mini Project: Build & Apply

Wire Encryption for Data in Transit in Hadoop

📖 Scenario: You work as a data engineer managing a Hadoop cluster. To protect sensitive data moving between nodes, you need to enable wire encryption. This ensures data is encrypted while traveling across the network, preventing eavesdropping.

🎯 Goal: Enable wire encryption for Hadoop data in transit by configuring the core-site.xml file with the correct properties.

📋 What You'll Learn

Create a dictionary called hadoop_config with initial Hadoop configuration properties.

Add a configuration variable enable_wire_encryption set to true to control encryption activation.

Use a comprehension to add wire encryption properties to hadoop_config only if enable_wire_encryption is true.

Print the final hadoop_config dictionary showing all properties.

💡 Why This Matters

🌍 Real World

Wire encryption protects data moving between Hadoop nodes from being intercepted or read by unauthorized users on the network.

💼 Career

Data engineers and Hadoop administrators must configure secure data transfer to comply with data privacy and security standards.

1

Create initial Hadoop configuration dictionary

Create a dictionary called hadoop_config with these exact entries: 'fs.defaultFS': 'hdfs://localhost:9000', 'hadoop.security.authentication': 'simple', and 'dfs.encrypt.data.transfer': 'false'.

Hadoop

# Create the hadoop_config dictionary with initial properties
# Your code here

Need a hint?

Use curly braces {} to create a dictionary with the exact keys and values.

2

Add encryption enable flag

Create a variable called enable_wire_encryption and set it to true.

Hadoop

hadoop_config = {
    'fs.defaultFS': 'hdfs://localhost:9000',
    'hadoop.security.authentication': 'simple',
    'dfs.encrypt.data.transfer': 'false'
}
# Create enable_wire_encryption variable and set to true
# Your code here

Need a hint?

Just assign true to the variable enable_wire_encryption.

3

Add wire encryption properties conditionally

Use an if statement to check if enable_wire_encryption is true. Inside it, update hadoop_config with these exact entries: 'dfs.encrypt.data.transfer': 'true', 'dfs.encrypt.data.transfer.algorithm': 'AES/CTR/NoPadding', and 'dfs.encrypt.data.transfer.cipher.suites': 'AES/CTR/NoPadding'.

Hadoop

hadoop_config = {
    'fs.defaultFS': 'hdfs://localhost:9000',
    'hadoop.security.authentication': 'simple',
    'dfs.encrypt.data.transfer': 'false'
}
enable_wire_encryption = true
# Add wire encryption properties to hadoop_config if enable_wire_encryption is true
# Your code here

Need a hint?

Use hadoop_config.update() inside the if block to add the encryption properties.

4

Print the final Hadoop configuration

Write a print statement to display the hadoop_config dictionary.

Hadoop

hadoop_config = {
    'fs.defaultFS': 'hdfs://localhost:9000',
    'hadoop.security.authentication': 'simple',
    'dfs.encrypt.data.transfer': 'false'
}
enable_wire_encryption = true

if enable_wire_encryption:
    hadoop_config.update({
        'dfs.encrypt.data.transfer': 'true',
        'dfs.encrypt.data.transfer.algorithm': 'AES/CTR/NoPadding',
        'dfs.encrypt.data.transfer.cipher.suites': 'AES/CTR/NoPadding'
    })
# Print the hadoop_config dictionary
# Your code here

Need a hint?

Use print(hadoop_config) to show the final dictionary.