Dataproc for Spark/Hadoop in GCP - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to create a Dataproc cluster with the default image version.

GCP

gcloud dataproc clusters create my-cluster --region=us-central1 --image-version=[1]

Drag options to blanks, or click blank then click option'

A2.0-debian10

B1.5-debian10

C1.3-ubuntu18

D3.0-centos7

Attempts:

3 left

💡 Hint

Common Mistakes

Using an outdated or unsupported image version.

Omitting the --image-version flag which defaults to the latest but is better to specify.

The default and recommended Dataproc image version is 2.0-debian10 which supports the latest features.

2fill in blank

medium

Complete the code to submit a Spark job to the Dataproc cluster.

GCP

gcloud dataproc jobs submit spark --cluster=my-cluster --region=us-central1 --class=[1] --jars=gs://my-bucket/my-job.jar -- my-args

Drag options to blanks, or click blank then click option'

Aorg.apache.hadoop.mapreduce.Job

Borg.apache.spark.examples.SparkPi

Ccom.google.cloud.dataproc.Main

Dorg.apache.hadoop.examples.WordCount

Attempts:

3 left

💡 Hint

Common Mistakes

Using a Hadoop class instead of a Spark class.

Using a non-existent or incorrect class name.

The SparkPi example class org.apache.spark.examples.SparkPi is commonly used to test Spark jobs on Dataproc.

3fill in blank

hard

Fix the error in the command to create a Dataproc cluster with 3 worker nodes.

GCP

gcloud dataproc clusters create my-cluster --region=us-central1 --num-workers=[1]

Drag options to blanks, or click blank then click option'

A3-workers

Bworker-count=3

C--num-workers=3

D3

Attempts:

3 left

💡 Hint

Common Mistakes

Including the flag name in the value.

Using incorrect syntax like '3-workers'.

The correct flag to specify the number of worker nodes is --num-workers followed by the number without extra syntax.

4fill in blank

hard

Fill both blanks to configure a Dataproc cluster with autoscaling enabled and specify the autoscaling policy.

GCP

gcloud dataproc clusters create my-cluster --region=us-central1 --enable-autoscaling --autoscaling-policy=[1] --num-workers=[2]

Drag options to blanks, or click blank then click option'

Amy-autoscale-policy

Bdefault-policy

C3

D5

Attempts:

3 left

💡 Hint

Common Mistakes

Confusing the autoscaling policy name with the number of workers.

Using default-policy when a custom policy is required.

Use your autoscaling policy name (e.g., my-autoscale-policy) and set the initial number of workers (e.g., 3).

5fill in blank

hard

Fill all three blanks to submit a Hadoop MapReduce job with a main class and specify the input and output paths.

GCP

gcloud dataproc jobs submit hadoop --cluster=my-cluster --region=us-central1 --class=[1] --jars=gs://my-bucket/my-hadoop-job.jar -- [2] [3]

Drag options to blanks, or click blank then click option'

Aorg.apache.hadoop.examples.WordCount

Bgs://my-bucket/input-data

Cgs://my-bucket/output-data

Dorg.apache.spark.examples.SparkPi

Attempts:

3 left

💡 Hint

Common Mistakes

Using a Spark class for a Hadoop job.

Swapping input and output paths.

The Hadoop job uses the WordCount class, with input and output paths specified as GCS URIs.

Practice

1. What is the main purpose of Google Cloud Dataproc when working with Spark and Hadoop?

easy

A. To manage and simplify running Spark and Hadoop clusters

B. To replace Spark and Hadoop with a new technology

C. To store data permanently like a database

D. To create virtual machines for web hosting

Solution

1. Step 1: Understand Dataproc's role

   Dataproc is a managed service that helps run Spark and Hadoop without manual setup.

2. Step 2: Compare options

   The other options describe replacing Spark and Hadoop, storing data permanently like a database, or creating VMs for web hosting, none of which is Dataproc's main purpose.

3. Final Answer:

   To manage and simplify running Spark and Hadoop clusters -> Option A

4. Quick Check:

   Dataproc manages Spark/Hadoop clusters = A [OK]

Hint: Dataproc manages clusters, not replaces or stores data [OK]

Common Mistakes:

• Thinking Dataproc replaces Spark/Hadoop
• Confusing Dataproc with storage services
• Assuming Dataproc creates VMs for web hosting

2. Which of the following is the correct gcloud command to create a Dataproc cluster named my-cluster in the us-central1 region?

easy

A. gcloud dataproc create cluster my-cluster --region us-central1

B. gcloud dataproc cluster create my-cluster --zone us-central1

C. gcloud dataproc clusters new my-cluster region=us-central1

D. gcloud dataproc clusters create my-cluster --region=us-central1

Solution

1. Step 1: Recall correct gcloud syntax

   The correct command uses 'clusters create' followed by the cluster name and '--region=' flag.

2. Step 2: Check each option

   'gcloud dataproc clusters create my-cluster --region=us-central1' matches the correct syntax. The other options have incorrect order, flags, or keywords.

3. Final Answer:

   gcloud dataproc clusters create my-cluster --region=us-central1 -> Option D

4. Quick Check:

   Correct gcloud cluster create syntax = B [OK]

Hint: Use 'clusters create' and '--region=' flag for cluster creation [OK]

Common Mistakes:

• Using 'create cluster' instead of 'clusters create'
• Missing equal sign in '--region='
• Confusing zone and region flags

3. Given the following command, what will be the result?

gcloud dataproc clusters create test-cluster --region=us-east1 --num-workers=2 --worker-machine-type=n1-standard-2

medium

A. Creates a cluster but in the default region ignoring '--region' flag

B. Creates a cluster named 'test-cluster' with 2 worker nodes of type n1-standard-2 in us-east1

C. Fails because '--num-workers' is not a valid flag

D. Creates a cluster with default worker count and machine type ignoring specified flags

Solution

1. Step 1: Analyze command flags

   The command specifies cluster name, region, number of workers, and worker machine type correctly.

2. Step 2: Understand Dataproc behavior

   Dataproc creates the cluster with given parameters if flags are valid and syntax is correct.

3. Final Answer:

   Creates a cluster named 'test-cluster' with 2 worker nodes of type n1-standard-2 in us-east1 -> Option B

4. Quick Check:

   Flags set cluster size and type correctly = D [OK]

Hint: Flags set cluster size and type; valid flags create cluster as specified [OK]

Common Mistakes:

• Assuming '--num-workers' is invalid
• Ignoring region flag effects
• Thinking default values override flags

4. You run this command to create a Dataproc cluster but get an error:

gcloud dataproc clusters create mycluster --num-worker=3 --region=us-west1

What is the likely cause of the error?

medium

A. Cluster name 'mycluster' is invalid because it has no hyphen

B. The region 'us-west1' is not supported by Dataproc

C. The flag '--num-worker' is incorrect; it should be '--num-workers'

D. Missing machine type flag causes the error

Solution

1. Step 1: Check flag correctness

   The correct flag for worker count is '--num-workers' (plural), not '--num-worker'.

2. Step 2: Validate other inputs

   Region 'us-west1' is valid, cluster name format is acceptable, and machine type is optional with defaults.

3. Final Answer:

   The flag '--num-worker' is incorrect; it should be '--num-workers' -> Option C

4. Quick Check:

   Correct flag spelling fixes error = C [OK]

Hint: Check plural form of flags like '--num-workers' [OK]

Common Mistakes:

• Using singular '--num-worker' instead of plural
• Assuming region is invalid without checking
• Thinking machine type is mandatory

5. You want to run a Spark job on a Dataproc cluster that automatically scales worker nodes between 2 and 5 based on workload. Which feature should you enable when creating the cluster?

hard

A. Enable autoscaling policy and attach it to the cluster

B. Set '--num-workers=5' and manually delete nodes when idle

C. Use '--max-workers=5' without autoscaling policy

D. Create multiple clusters with different sizes and switch between them

Solution

1. Step 1: Understand autoscaling in Dataproc

   Dataproc supports autoscaling policies to adjust worker nodes automatically within set limits.

2. Step 2: Evaluate options

   Enable autoscaling policy and attach it to the cluster correctly enables autoscaling with a policy. The other options do not provide automatic scaling: they either use fixed sizes, incomplete flags without policy, or require manual switching between clusters.

3. Final Answer:

   Enable autoscaling policy and attach it to the cluster -> Option A

4. Quick Check:

   Autoscaling policy enables dynamic scaling = A [OK]

Hint: Use autoscaling policies for automatic worker scaling [OK]

Common Mistakes:

• Setting fixed worker count without autoscaling
• Ignoring autoscaling policies
• Creating multiple clusters instead of scaling one