AWS EMR setup in Apache Spark - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to specify the EMR cluster's release label.

Apache Spark

emr_cluster = client.run_job_flow(ReleaseLabel='[1]')

Drag options to blanks, or click blank then click option'

Aemr-5.30.0

Bemr-6.7.0

Cemr-3.11.0

Demr-4.0.0

Attempts:

3 left

💡 Hint

Common Mistakes

Using an outdated EMR release label that lacks Spark support.

Misspelling the release label string.

The release label emr-6.7.0 specifies a recent stable EMR version with Apache Spark support.

2fill in blank

medium

Complete the code to set the instance type for the master node in the EMR cluster.

Apache Spark

Instances={'MasterInstanceType': '[1]', 'InstanceCount': 3}

Drag options to blanks, or click blank then click option'

Am5.xlarge

Bt2.micro

Cc4.large

Dr3.2xlarge

Attempts:

3 left

💡 Hint

Common Mistakes

Choosing an instance type too small for the master node.

Confusing master and core node instance types.

The m5.xlarge instance type is a balanced choice for the master node in EMR clusters.

3fill in blank

hard

Fix the error in the bootstrap action configuration to install a custom package.

Apache Spark

BootstrapActions=[{'Name': 'InstallPackages', 'ScriptBootstrapAction': {'Path': '[1]'}}]

Drag options to blanks, or click blank then click option'

Ahttp://mybucket/scripts/install.sh

Bgs://mybucket/scripts/install.sh

C/local/scripts/install.sh

Ds3://mybucket/scripts/install.sh

Attempts:

3 left

💡 Hint

Common Mistakes

Using local file paths instead of S3 URLs.

Using incorrect cloud storage prefixes like gs:// (Google Cloud Storage).

Bootstrap scripts must be stored in an S3 bucket accessible by EMR, so the path must start with s3://.

4fill in blank

hard

Fill both blanks to configure the EMR cluster to use Spot instances for core nodes with a bid price.

Apache Spark

Instances={'CoreInstanceType': '[1]', 'CoreInstanceMarket': '[2]', 'CoreBidPrice': '0.10', 'InstanceCount': 5}

Drag options to blanks, or click blank then click option'

Am5.xlarge

BON_DEMAND

CSPOT

Dc5.large

Attempts:

3 left

💡 Hint

Common Mistakes

Setting CoreInstanceMarket to 'ON_DEMAND' when Spot is intended.

Choosing an instance type not supported for Spot.

Core nodes use m5.xlarge instances and the market type SPOT enables Spot instances for cost savings.

5fill in blank

hard

Fill all three blanks to define an EMR step that runs a Spark application stored in S3.

Apache Spark

Steps=[{'Name': 'SparkApp', 'ActionOnFailure': '[1]', 'HadoopJarStep': {'Jar': '[2]', 'Args': ['spark-submit', '[3]']}}]

Drag options to blanks, or click blank then click option'

ACONTINUE

Bcommand-runner.jar

Cs3://mybucket/apps/my_spark_app.py

DTERMINATE_CLUSTER

Attempts:

3 left

💡 Hint

Common Mistakes

Using TERMINATE_CLUSTER which stops the cluster on failure.

Incorrect Jar name or missing Spark app path.

The step continues on failure (CONTINUE), uses command-runner.jar to run commands, and runs the Spark app located at s3://mybucket/apps/my_spark_app.py.

Practice

1. What is the primary purpose of setting up an AWS EMR cluster?

easy

A. To create a ready-to-use environment for big data processing

B. To host a website with static content

C. To store small files for backup

D. To run simple desktop applications remotely

Solution

1. Step 1: Understand AWS EMR's role

   AWS EMR is designed to process large data sets using big data tools like Apache Spark.

2. Step 2: Compare options

   Options B, C, and D describe unrelated uses like hosting websites or backups, which are not EMR's main purpose.

3. Final Answer:

   To create a ready-to-use environment for big data processing -> Option A

4. Quick Check:

   EMR = Big Data Processing [OK]

Hint: EMR is for big data, not simple hosting or backups [OK]

Common Mistakes:

• Confusing EMR with simple storage services
• Thinking EMR hosts websites
• Assuming EMR runs desktop apps

2. Which of the following is the correct way to specify the EMR version in a cluster setup?

easy

A. ReleaseLabel: 'emr-6.7.0'

B. Version: '6.7.0'

C. EMRVersion: 'v6.7'

D. ClusterVersion: '6.7'

Solution

1. Step 1: Identify correct EMR version key

   The AWS EMR API uses ReleaseLabel to specify the EMR version, e.g., 'emr-6.7.0'.

2. Step 2: Validate other options

   Options A, B, and D use incorrect keys or formats not recognized by AWS EMR.

3. Final Answer:

   <code>ReleaseLabel: 'emr-6.7.0'</code> -> Option A

4. Quick Check:

   EMR version key = ReleaseLabel [OK]

Hint: EMR version uses 'ReleaseLabel' with 'emr-' prefix [OK]

Common Mistakes:

• Using 'Version' or 'EMRVersion' instead of 'ReleaseLabel'
• Omitting the 'emr-' prefix
• Using incomplete version strings

3. Given this EMR cluster configuration snippet:

{"Instances": {"InstanceGroups": [{"InstanceCount": 3, "InstanceType": "m5.xlarge", "Name": "Core"}]}}

How many core nodes will the cluster have?

medium

A. 1

B. Depends on the master node count

C. 0

D. 3

Solution

1. Step 1: Locate InstanceCount for Core nodes

   The configuration shows InstanceCount: 3 for the Core instance group.

2. Step 2: Understand Core node count meaning

   This means the cluster will have 3 core nodes of type m5.xlarge.

3. Final Answer:

   3 -> Option D

4. Quick Check:

   Core nodes = InstanceCount = 3 [OK]

Hint: InstanceCount under Core group = number of core nodes [OK]

Common Mistakes:

• Confusing core nodes with master nodes
• Assuming default counts if not specified
• Ignoring InstanceCount value

4. You tried to create an EMR cluster but got a permissions error. Which is the most likely cause?

medium

A. You specified an invalid EMR version

B. You did not assign the default EMR roles for EC2 and EMR service

C. You chose an unsupported instance type

D. You forgot to specify the cluster name

Solution

1. Step 1: Identify cause of permissions error

   Permissions errors usually happen when IAM roles are missing or incorrect.

2. Step 2: Check EMR role requirements

   EMR requires default roles for EC2 instances and EMR service to access resources properly.

3. Final Answer:

   You did not assign the default EMR roles for EC2 and EMR service -> Option B

4. Quick Check:

   Permissions error = missing default roles [OK]

Hint: Always assign default EMR roles to avoid permission errors [OK]

Common Mistakes:

• Blaming version or instance type for permission issues
• Ignoring IAM role setup
• Assuming cluster name affects permissions

5. You want to run Apache Spark on an EMR cluster with 1 master node and 4 core nodes. Which configuration snippet correctly sets this up?

hard

A. {"Instances": {"InstanceGroups": [{"InstanceCount": 2, "InstanceType": "m5.xlarge", "Name": "Master"}, {"InstanceCount": 4, "InstanceType": "m5.xlarge", "Name": "Core"}]}}

B. {"Instances": {"InstanceGroups": [{"InstanceCount": 2, "InstanceType": "m5.xlarge", "Name": "Core"}, {"InstanceCount": 4, "InstanceType": "m5.xlarge", "Name": "Master"}]}}

C. {"Instances": {"InstanceGroups": [{"InstanceCount": 1, "InstanceType": "m5.xlarge", "Name": "Master"}, {"InstanceCount": 4, "InstanceType": "m5.xlarge", "Name": "Core"}]}}

D. {"Instances": {"InstanceGroups": [{"InstanceCount": 4, "InstanceType": "m5.xlarge", "Name": "Master"}, {"InstanceCount": 2, "InstanceType": "m5.xlarge", "Name": "Core"}]}}

Solution

1. Step 1: Identify master and core node counts

   The requirement is 1 master node and 4 core nodes.

2. Step 2: Match counts and names in configuration

   {"Instances": {"InstanceGroups": [{"InstanceCount": 1, "InstanceType": "m5.xlarge", "Name": "Master"}, {"InstanceCount": 4, "InstanceType": "m5.xlarge", "Name": "Core"}]}}

   correctly assigns 1 to Master and 4 to Core nodes with proper names.

3. Step 3: Check other options

   Options A, C, and D mix counts or swap Master and Core names incorrectly.

4. Final Answer:

   Configuration with 1 Master and 4 Core nodes correctly named -> Option C

5. Quick Check:

   Master=1, Core=4 with correct names [OK]

Hint: Master and Core counts must match names exactly [OK]

Common Mistakes:

• Swapping Master and Core node counts
• Using wrong instance group names
• Assigning incorrect counts