AWS EMR setup in Apache Spark - Mini Project: Build & Apply

AWS EMR Setup with Apache Spark

📖 Scenario: You are working as a data engineer who needs to process large datasets using Apache Spark on AWS EMR (Elastic MapReduce). Setting up the EMR cluster correctly is the first step before running any Spark jobs.

🎯 Goal: Set up an AWS EMR cluster configuration using Apache Spark. You will create the initial cluster configuration, add necessary settings, apply the core Spark configuration, and finally output the cluster setup details.

📋 What You'll Learn

Create a dictionary with the initial EMR cluster configuration

Add a configuration variable for the Spark version

Apply the Spark core configuration to the cluster setup

Print the final EMR cluster configuration dictionary

💡 Why This Matters

🌍 Real World

Setting up AWS EMR clusters is a common task for data engineers to run big data processing jobs using Apache Spark.

💼 Career

Understanding EMR cluster configuration helps in managing cloud resources efficiently and running scalable data pipelines.

1

Create initial EMR cluster configuration

Create a dictionary called emr_cluster with these exact entries: 'Name': 'TestCluster', 'ReleaseLabel': 'emr-6.7.0', and 'Instances': {'InstanceGroups': []}.

Apache Spark

# Create the initial EMR cluster configuration dictionary
# Your code here

Need a hint?

Use a dictionary with keys 'Name', 'ReleaseLabel', and 'Instances'. The 'Instances' key should have a nested dictionary with 'InstanceGroups' as an empty list.

2

Add Spark version configuration

Create a variable called spark_version and set it to the string '3.3.1'.

Apache Spark

emr_cluster = {
    'Name': 'TestCluster',
    'ReleaseLabel': 'emr-6.7.0',
    'Instances': {'InstanceGroups': []}
}
# Create a variable called spark_version and set it to '3.3.1'
# Your code here

Need a hint?

Assign the string '3.3.1' to the variable named spark_version.

3

Add Spark core configuration to EMR cluster

Add a key 'Configurations' to the emr_cluster dictionary. Set its value to a list containing one dictionary with 'Classification': 'spark' and 'Properties': {'spark.version': spark_version}.

Apache Spark

emr_cluster = {
    'Name': 'TestCluster',
    'ReleaseLabel': 'emr-6.7.0',
    'Instances': {'InstanceGroups': []}
}
spark_version = '3.3.1'
# Add the 'Configurations' key to emr_cluster with Spark settings
# Your code here

Need a hint?

Use a list with one dictionary inside for the 'Configurations' key. The dictionary should have 'Classification' and 'Properties' keys.

4

Print the final EMR cluster configuration

Write a print statement to display the emr_cluster dictionary.

Apache Spark

emr_cluster = {
    'Name': 'TestCluster',
    'ReleaseLabel': 'emr-6.7.0',
    'Instances': {'InstanceGroups': []},
    'Configurations': [
        {
            'Classification': 'spark',
            'Properties': {'spark.version': spark_version}
        }
    ]
}
spark_version = '3.3.1'
# Print the emr_cluster dictionary
# Your code here

Need a hint?

Use print(emr_cluster) to show the final dictionary.