AWS EMR setup in Apache Spark - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Architecture

intermediate

Choosing the Right EMR Cluster Configuration

You want to run a Spark job on AWS EMR that processes large datasets with high fault tolerance. Which cluster configuration best fits this need?

A Multiple master nodes with no core or task nodes.

B A single master node with multiple core nodes and no task nodes.

C A single master node, multiple core nodes, and multiple task nodes.

D Multiple task nodes only, no master or core nodes.

💡 Hint

Think about fault tolerance and workload distribution in EMR clusters.

Option C provides a master node to manage the cluster, core nodes to store data and run tasks, and task nodes to handle additional processing. This setup offers fault tolerance and scalability.

❓ security

intermediate

Securing EMR Cluster Access

You want to restrict SSH access to your EMR cluster only from your office IP address. Which AWS feature should you configure?

A Enable EMR cluster encryption at rest.

B Configure EMR bootstrap actions to disable SSH.

C Use IAM roles to restrict access to the EMR cluster.

D Modify the EMR cluster's security group to allow SSH only from your office IP.

💡 Hint

Think about network-level access control.

Security groups act like virtual firewalls controlling inbound and outbound traffic. Restricting SSH access by IP is done via security group rules.

❓ service_behavior

advanced

EMR Cluster Auto Scaling Behavior

Given an EMR cluster with auto scaling enabled, what happens when the workload decreases significantly?

A The cluster automatically terminates task nodes but keeps core nodes running.

B The cluster does not change node count automatically.

C The cluster shuts down the master node to save costs.

D The cluster automatically terminates core nodes to save costs.

💡 Hint

Consider which nodes store data and which nodes are ephemeral.

Core nodes store data in HDFS and are essential for cluster stability, so they are not terminated automatically. Task nodes are ephemeral and can be scaled down to save costs.

❓ Configuration

advanced

Configuring EMR to Use Spot Instances

You want to reduce EMR cluster costs by using Spot Instances for worker nodes. Which configuration is correct?

A Set task nodes as Spot Instances and core nodes as On-Demand.

B Set master node as Spot Instance and core nodes as On-Demand.

C Set all nodes as Spot Instances including master node.

D Set core nodes as Spot Instances and master node as On-Demand.

💡 Hint

Think about which nodes are critical for cluster stability.

The master and core nodes are critical for cluster stability and data storage, so they should be On-Demand. Task nodes can be Spot Instances to save costs as they are replaceable.

🧠 Conceptual

expert

EMR Cluster Data Persistence After Termination

What happens to the data stored on HDFS of EMR core nodes when the cluster is terminated?

A Data on HDFS remains available after cluster termination for 7 days.

B Data on HDFS is lost unless backed up to S3 before termination.

C Data on HDFS is automatically saved to S3 by EMR on termination.

D Data on HDFS is replicated to other clusters automatically.

💡 Hint

Consider the nature of ephemeral storage on EMR core nodes.

HDFS storage on EMR core nodes is ephemeral and tied to the cluster lifecycle. Data must be saved to durable storage like S3 before cluster termination to avoid loss.