Dataproc for Spark/Hadoop in GCP - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Architecture

intermediate

Choosing the right cluster type for a batch Spark job

You need to run a batch Spark job on Google Cloud Dataproc. The job runs once a day and processes large data sets. Which cluster configuration is the best choice to optimize cost and performance?

A Create a permanent Dataproc cluster with autoscaling enabled and submit the job to it.

B Create a transient Dataproc cluster that is created before the job and deleted after the job finishes.

C Use a permanent Dataproc cluster without autoscaling and keep it running 24/7.

D Use a single-node Dataproc cluster with preemptible workers for cost savings.

💡 Hint

Think about cost efficiency for jobs that run once a day.

Transient clusters are created for the job and deleted after completion, saving costs for batch jobs that run periodically. Permanent clusters running 24/7 incur costs even when idle. Autoscaling helps but is more suited for variable workloads.

❓ Configuration

intermediate

Configuring autoscaling policy for Dataproc cluster

You want to configure an autoscaling policy for a Dataproc cluster to scale workers between 2 and 10 nodes based on YARN memory usage. Which configuration snippet correctly sets this policy?

A { "workerConfig": { "minInstances": 2, "maxInstances": 10 }, "basicAlgorithm": { "yarnConfig": { "scaleUpMemoryThresholdPercent": 50, "scaleDownMemoryThresholdPercent": 80 } } }

B { "workerConfig": { "minInstances": 10, "maxInstances": 2 }, "basicAlgorithm": { "yarnConfig": { "scaleUpMemoryThresholdPercent": 50, "scaleDownMemoryThresholdPercent": 80 } } }

C { "workerConfig": { "minInstances": 2, "maxInstances": 10 }, "basicAlgorithm": { "yarnConfig": { "scaleUpMemoryThresholdPercent": 80, "scaleDownMemoryThresholdPercent": 50 } } }

D { "workerConfig": { "minInstances": 2, "maxInstances": 10 }, "basicAlgorithm": { "yarnConfig": { "scaleUpMemoryThresholdPercent": 90, "scaleDownMemoryThresholdPercent": 90 } } }

💡 Hint

Scale up when usage is high, scale down when usage is low.

The minInstances and maxInstances define the scaling range. The scaleUpMemoryThresholdPercent should be higher than scaleDownMemoryThresholdPercent to avoid rapid scaling. Option C correctly sets min 2, max 10, scale up at 80%, scale down at 50%.

❓ security

advanced

Securing access to Dataproc clusters with least privilege

You want to ensure that only specific users can submit jobs to a Dataproc cluster, and no one else can access the cluster's VMs. Which approach best enforces this principle?

A Grant the users the 'roles/dataproc.jobUser' IAM role on the cluster and configure firewall rules to restrict SSH access to only admin IPs.

B Grant the users the 'roles/dataproc.editor' IAM role on the cluster and disable SSH access to the cluster VMs.

C Grant the users the 'roles/compute.instanceAdmin' role and allow SSH access to all cluster VMs.

D Grant the users the 'roles/dataproc.viewer' role and enable OS Login for all users.

💡 Hint

Least privilege means giving only the permissions needed to submit jobs, not full editing or VM control.

The 'roles/dataproc.jobUser' role allows users to submit jobs without full cluster editing rights. Restricting SSH access via firewall rules limits VM access to admins only, enforcing least privilege.

❓ service_behavior

advanced

Understanding job failure behavior on preemptible workers

You run a Spark job on a Dataproc cluster with preemptible worker nodes. During execution, some preemptible workers are terminated by Google Cloud. What is the expected behavior of the Spark job?

A The Spark job automatically retries tasks lost due to preemptible worker termination and completes successfully if enough workers remain.

B The Spark job immediately fails when any preemptible worker is terminated.

C The Spark job continues without retrying lost tasks, resulting in incomplete output.

D The Spark job pauses until the preemptible workers are replaced by new nodes.

💡 Hint

Think about how Spark handles worker node failures.

Spark is designed to handle worker failures by retrying lost tasks on remaining or new nodes. Preemptible workers can be terminated, but Spark retries tasks to complete the job if enough resources remain.

🧠 Conceptual

expert

Impact of network tags on Dataproc cluster firewall rules

You create a Dataproc cluster with network tags 'dataproc-cluster' and want to allow SSH access only from your office IP range. You create a firewall rule allowing TCP port 22 for source IPs in your office range and target tags 'dataproc-cluster'. However, you cannot SSH into the cluster VMs. What is the most likely cause?

A The firewall rule source IP range is incorrect and does not include your office IPs.

B The firewall rule priority is too low, so it is overridden by a deny-all rule.

C Dataproc clusters ignore network tags for firewall rules and require IP-based rules only.

D The cluster VMs do not have the 'dataproc-cluster' network tag assigned, so the firewall rule does not apply.

💡 Hint

Firewall rules with target tags only apply to VMs with matching network tags.

If the cluster VMs lack the specified network tag, the firewall rule targeting that tag does not apply, blocking SSH access. Assigning the correct network tag to the VMs fixes this.

Practice

1. What is the main purpose of Google Cloud Dataproc when working with Spark and Hadoop?

easy

A. To manage and simplify running Spark and Hadoop clusters

B. To replace Spark and Hadoop with a new technology

C. To store data permanently like a database

D. To create virtual machines for web hosting

Solution

1. Step 1: Understand Dataproc's role

   Dataproc is a managed service that helps run Spark and Hadoop without manual setup.

2. Step 2: Compare options

   The other options describe replacing Spark and Hadoop, storing data permanently like a database, or creating VMs for web hosting, none of which is Dataproc's main purpose.

3. Final Answer:

   To manage and simplify running Spark and Hadoop clusters -> Option A

4. Quick Check:

   Dataproc manages Spark/Hadoop clusters = A [OK]

Hint: Dataproc manages clusters, not replaces or stores data [OK]

Common Mistakes:

• Thinking Dataproc replaces Spark/Hadoop
• Confusing Dataproc with storage services
• Assuming Dataproc creates VMs for web hosting

2. Which of the following is the correct gcloud command to create a Dataproc cluster named my-cluster in the us-central1 region?

easy

A. gcloud dataproc create cluster my-cluster --region us-central1

B. gcloud dataproc cluster create my-cluster --zone us-central1

C. gcloud dataproc clusters new my-cluster region=us-central1

D. gcloud dataproc clusters create my-cluster --region=us-central1

Solution

1. Step 1: Recall correct gcloud syntax

   The correct command uses 'clusters create' followed by the cluster name and '--region=' flag.

2. Step 2: Check each option

   'gcloud dataproc clusters create my-cluster --region=us-central1' matches the correct syntax. The other options have incorrect order, flags, or keywords.

3. Final Answer:

   gcloud dataproc clusters create my-cluster --region=us-central1 -> Option D

4. Quick Check:

   Correct gcloud cluster create syntax = B [OK]

Hint: Use 'clusters create' and '--region=' flag for cluster creation [OK]

Common Mistakes:

• Using 'create cluster' instead of 'clusters create'
• Missing equal sign in '--region='
• Confusing zone and region flags

3. Given the following command, what will be the result?

gcloud dataproc clusters create test-cluster --region=us-east1 --num-workers=2 --worker-machine-type=n1-standard-2

medium

A. Creates a cluster but in the default region ignoring '--region' flag

B. Creates a cluster named 'test-cluster' with 2 worker nodes of type n1-standard-2 in us-east1

C. Fails because '--num-workers' is not a valid flag

D. Creates a cluster with default worker count and machine type ignoring specified flags

Solution

1. Step 1: Analyze command flags

   The command specifies cluster name, region, number of workers, and worker machine type correctly.

2. Step 2: Understand Dataproc behavior

   Dataproc creates the cluster with given parameters if flags are valid and syntax is correct.

3. Final Answer:

   Creates a cluster named 'test-cluster' with 2 worker nodes of type n1-standard-2 in us-east1 -> Option B

4. Quick Check:

   Flags set cluster size and type correctly = D [OK]

Hint: Flags set cluster size and type; valid flags create cluster as specified [OK]

Common Mistakes:

• Assuming '--num-workers' is invalid
• Ignoring region flag effects
• Thinking default values override flags

4. You run this command to create a Dataproc cluster but get an error:

gcloud dataproc clusters create mycluster --num-worker=3 --region=us-west1

What is the likely cause of the error?

medium

A. Cluster name 'mycluster' is invalid because it has no hyphen

B. The region 'us-west1' is not supported by Dataproc

C. The flag '--num-worker' is incorrect; it should be '--num-workers'

D. Missing machine type flag causes the error

Solution

1. Step 1: Check flag correctness

   The correct flag for worker count is '--num-workers' (plural), not '--num-worker'.

2. Step 2: Validate other inputs

   Region 'us-west1' is valid, cluster name format is acceptable, and machine type is optional with defaults.

3. Final Answer:

   The flag '--num-worker' is incorrect; it should be '--num-workers' -> Option C

4. Quick Check:

   Correct flag spelling fixes error = C [OK]

Hint: Check plural form of flags like '--num-workers' [OK]

Common Mistakes:

• Using singular '--num-worker' instead of plural
• Assuming region is invalid without checking
• Thinking machine type is mandatory

5. You want to run a Spark job on a Dataproc cluster that automatically scales worker nodes between 2 and 5 based on workload. Which feature should you enable when creating the cluster?

hard

A. Enable autoscaling policy and attach it to the cluster

B. Set '--num-workers=5' and manually delete nodes when idle

C. Use '--max-workers=5' without autoscaling policy

D. Create multiple clusters with different sizes and switch between them

Solution

1. Step 1: Understand autoscaling in Dataproc

   Dataproc supports autoscaling policies to adjust worker nodes automatically within set limits.

2. Step 2: Evaluate options

   Enable autoscaling policy and attach it to the cluster correctly enables autoscaling with a policy. The other options do not provide automatic scaling: they either use fixed sizes, incomplete flags without policy, or require manual switching between clusters.

3. Final Answer:

   Enable autoscaling policy and attach it to the cluster -> Option A

4. Quick Check:

   Autoscaling policy enables dynamic scaling = A [OK]

Hint: Use autoscaling policies for automatic worker scaling [OK]

Common Mistakes:

• Setting fixed worker count without autoscaling
• Ignoring autoscaling policies
• Creating multiple clusters instead of scaling one