AWS EMR setup in Apache Spark - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Architecture

intermediate

Choosing the Right EMR Cluster Configuration

You want to run a Spark job on AWS EMR that processes large datasets with high fault tolerance. Which cluster configuration best fits this need?

A Multiple master nodes with no core or task nodes.

B A single master node with multiple core nodes and no task nodes.

C A single master node, multiple core nodes, and multiple task nodes.

D Multiple task nodes only, no master or core nodes.

💡 Hint

Think about fault tolerance and workload distribution in EMR clusters.

Option C provides a master node to manage the cluster, core nodes to store data and run tasks, and task nodes to handle additional processing. This setup offers fault tolerance and scalability.

❓ security

intermediate

Securing EMR Cluster Access

You want to restrict SSH access to your EMR cluster only from your office IP address. Which AWS feature should you configure?

A Enable EMR cluster encryption at rest.

B Configure EMR bootstrap actions to disable SSH.

C Use IAM roles to restrict access to the EMR cluster.

D Modify the EMR cluster's security group to allow SSH only from your office IP.

💡 Hint

Think about network-level access control.

Security groups act like virtual firewalls controlling inbound and outbound traffic. Restricting SSH access by IP is done via security group rules.

❓ service_behavior

advanced

EMR Cluster Auto Scaling Behavior

Given an EMR cluster with auto scaling enabled, what happens when the workload decreases significantly?

A The cluster automatically terminates task nodes but keeps core nodes running.

B The cluster does not change node count automatically.

C The cluster shuts down the master node to save costs.

D The cluster automatically terminates core nodes to save costs.

💡 Hint

Consider which nodes store data and which nodes are ephemeral.

Core nodes store data in HDFS and are essential for cluster stability, so they are not terminated automatically. Task nodes are ephemeral and can be scaled down to save costs.

❓ Configuration

advanced

Configuring EMR to Use Spot Instances

You want to reduce EMR cluster costs by using Spot Instances for worker nodes. Which configuration is correct?

A Set task nodes as Spot Instances and core nodes as On-Demand.

B Set master node as Spot Instance and core nodes as On-Demand.

C Set all nodes as Spot Instances including master node.

D Set core nodes as Spot Instances and master node as On-Demand.

💡 Hint

Think about which nodes are critical for cluster stability.

The master and core nodes are critical for cluster stability and data storage, so they should be On-Demand. Task nodes can be Spot Instances to save costs as they are replaceable.

🧠 Conceptual

expert

EMR Cluster Data Persistence After Termination

What happens to the data stored on HDFS of EMR core nodes when the cluster is terminated?

A Data on HDFS remains available after cluster termination for 7 days.

B Data on HDFS is lost unless backed up to S3 before termination.

C Data on HDFS is automatically saved to S3 by EMR on termination.

D Data on HDFS is replicated to other clusters automatically.

💡 Hint

Consider the nature of ephemeral storage on EMR core nodes.

HDFS storage on EMR core nodes is ephemeral and tied to the cluster lifecycle. Data must be saved to durable storage like S3 before cluster termination to avoid loss.

Practice

1. What is the primary purpose of setting up an AWS EMR cluster?

easy

A. To create a ready-to-use environment for big data processing

B. To host a website with static content

C. To store small files for backup

D. To run simple desktop applications remotely

Solution

1. Step 1: Understand AWS EMR's role

   AWS EMR is designed to process large data sets using big data tools like Apache Spark.

2. Step 2: Compare options

   Options B, C, and D describe unrelated uses like hosting websites or backups, which are not EMR's main purpose.

3. Final Answer:

   To create a ready-to-use environment for big data processing -> Option A

4. Quick Check:

   EMR = Big Data Processing [OK]

Hint: EMR is for big data, not simple hosting or backups [OK]

Common Mistakes:

• Confusing EMR with simple storage services
• Thinking EMR hosts websites
• Assuming EMR runs desktop apps

2. Which of the following is the correct way to specify the EMR version in a cluster setup?

easy

A. ReleaseLabel: 'emr-6.7.0'

B. Version: '6.7.0'

C. EMRVersion: 'v6.7'

D. ClusterVersion: '6.7'

Solution

1. Step 1: Identify correct EMR version key

   The AWS EMR API uses ReleaseLabel to specify the EMR version, e.g., 'emr-6.7.0'.

2. Step 2: Validate other options

   Options A, B, and D use incorrect keys or formats not recognized by AWS EMR.

3. Final Answer:

   <code>ReleaseLabel: 'emr-6.7.0'</code> -> Option A

4. Quick Check:

   EMR version key = ReleaseLabel [OK]

Hint: EMR version uses 'ReleaseLabel' with 'emr-' prefix [OK]

Common Mistakes:

• Using 'Version' or 'EMRVersion' instead of 'ReleaseLabel'
• Omitting the 'emr-' prefix
• Using incomplete version strings

3. Given this EMR cluster configuration snippet:

{"Instances": {"InstanceGroups": [{"InstanceCount": 3, "InstanceType": "m5.xlarge", "Name": "Core"}]}}

How many core nodes will the cluster have?

medium

A. 1

B. Depends on the master node count

C. 0

D. 3

Solution

1. Step 1: Locate InstanceCount for Core nodes

   The configuration shows InstanceCount: 3 for the Core instance group.

2. Step 2: Understand Core node count meaning

   This means the cluster will have 3 core nodes of type m5.xlarge.

3. Final Answer:

   3 -> Option D

4. Quick Check:

   Core nodes = InstanceCount = 3 [OK]

Hint: InstanceCount under Core group = number of core nodes [OK]

Common Mistakes:

• Confusing core nodes with master nodes
• Assuming default counts if not specified
• Ignoring InstanceCount value

4. You tried to create an EMR cluster but got a permissions error. Which is the most likely cause?

medium

A. You specified an invalid EMR version

B. You did not assign the default EMR roles for EC2 and EMR service

C. You chose an unsupported instance type

D. You forgot to specify the cluster name

Solution

1. Step 1: Identify cause of permissions error

   Permissions errors usually happen when IAM roles are missing or incorrect.

2. Step 2: Check EMR role requirements

   EMR requires default roles for EC2 instances and EMR service to access resources properly.

3. Final Answer:

   You did not assign the default EMR roles for EC2 and EMR service -> Option B

4. Quick Check:

   Permissions error = missing default roles [OK]

Hint: Always assign default EMR roles to avoid permission errors [OK]

Common Mistakes:

• Blaming version or instance type for permission issues
• Ignoring IAM role setup
• Assuming cluster name affects permissions

5. You want to run Apache Spark on an EMR cluster with 1 master node and 4 core nodes. Which configuration snippet correctly sets this up?

hard

A. {"Instances": {"InstanceGroups": [{"InstanceCount": 2, "InstanceType": "m5.xlarge", "Name": "Master"}, {"InstanceCount": 4, "InstanceType": "m5.xlarge", "Name": "Core"}]}}

B. {"Instances": {"InstanceGroups": [{"InstanceCount": 2, "InstanceType": "m5.xlarge", "Name": "Core"}, {"InstanceCount": 4, "InstanceType": "m5.xlarge", "Name": "Master"}]}}

C. {"Instances": {"InstanceGroups": [{"InstanceCount": 1, "InstanceType": "m5.xlarge", "Name": "Master"}, {"InstanceCount": 4, "InstanceType": "m5.xlarge", "Name": "Core"}]}}

D. {"Instances": {"InstanceGroups": [{"InstanceCount": 4, "InstanceType": "m5.xlarge", "Name": "Master"}, {"InstanceCount": 2, "InstanceType": "m5.xlarge", "Name": "Core"}]}}

Solution

1. Step 1: Identify master and core node counts

   The requirement is 1 master node and 4 core nodes.

2. Step 2: Match counts and names in configuration

   {"Instances": {"InstanceGroups": [{"InstanceCount": 1, "InstanceType": "m5.xlarge", "Name": "Master"}, {"InstanceCount": 4, "InstanceType": "m5.xlarge", "Name": "Core"}]}}

   correctly assigns 1 to Master and 4 to Core nodes with proper names.

3. Step 3: Check other options

   Options A, C, and D mix counts or swap Master and Core names incorrectly.

4. Final Answer:

   Configuration with 1 Master and 4 Core nodes correctly named -> Option C

5. Quick Check:

   Master=1, Core=4 with correct names [OK]

Hint: Master and Core counts must match names exactly [OK]

Common Mistakes:

• Swapping Master and Core node counts
• Using wrong instance group names
• Assigning incorrect counts