Google Dataproc overview in Apache Spark - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

What is the primary purpose of Google Dataproc?

Google Dataproc is a managed service for running which type of workloads?

A Running big data processing jobs using Apache Spark and Hadoop clusters.

B Hosting web applications with automatic scaling.

C Storing large amounts of unstructured data in a NoSQL database.

D Providing real-time analytics dashboards for business intelligence.

💡 Hint

Think about what Apache Spark and Hadoop are used for.

Google Dataproc is designed to run big data processing workloads using Apache Spark and Hadoop clusters in a managed environment.

❓ data_output

intermediate

Output of a Spark job on Dataproc cluster

Given the following PySpark code running on a Dataproc cluster, what is the output?

data = ["apple", "banana", "apple", "orange", "banana", "apple"]
rdd = spark.sparkContext.parallelize(data)
counts = rdd.map(lambda x: (x, 1)).reduceByKey(lambda a, b: a + b).collect()
print(sorted(counts))

Apache Spark

data = ["apple", "banana", "apple", "orange", "banana", "apple"]
rdd = spark.sparkContext.parallelize(data)
counts = rdd.map(lambda x: (x, 1)).reduceByKey(lambda a, b: a + b).collect()
print(sorted(counts))

A [('apple', 3), ('banana', 2), ('orange', 1)]

B [('apple', 1), ('banana', 1), ('orange', 1)]

C [('apple', 3), ('banana', 3), ('orange', 1)]

D [('apple', 2), ('banana', 2), ('orange', 2)]

💡 Hint

Count how many times each fruit appears in the list.

The code counts occurrences of each fruit. 'apple' appears 3 times, 'banana' 2 times, and 'orange' once.

🔧 Debug

advanced

Identify the error in Dataproc Spark job submission

Which option describes the error when submitting a Spark job to Dataproc with this command?

gcloud dataproc jobs submit spark --cluster=my-cluster --region=us-central1 --class=org.apache.spark.examples.SparkPi --jars=file:///usr/lib/spark/examples/jars/spark-examples.jar -- 1000

A The argument '1000' should be passed before the --jars option.

B The --class argument is missing the main class name.

C The --jars argument uses a local file path that is not accessible to the cluster nodes.

D The --region flag is not supported in gcloud dataproc jobs submit spark.

💡 Hint

Consider where the cluster nodes can access files specified in --jars.

The --jars option requires a path accessible by cluster nodes, usually a GCS path, not a local file path.

🚀 Application

advanced

Choosing the right Dataproc cluster configuration

You need to process a large batch of data with Apache Spark on Dataproc. The job requires high memory and CPU resources but runs only for a short time. Which cluster configuration is best?

A Use a small cluster with preemptible worker nodes to save cost.

B Use a large cluster with high-memory machine types and disable autoscaling.

C Use a single-node cluster with high-CPU machine type.

D Use a large cluster with standard worker nodes and enable autoscaling.

💡 Hint

Think about balancing cost, performance, and job duration.

A large cluster with standard nodes and autoscaling provides enough resources and adjusts size to optimize cost for short jobs.

❓ visualization

expert

Interpreting Dataproc job monitoring dashboard

In the Dataproc job monitoring dashboard, you see a Spark job with a long GC (Garbage Collection) time and low CPU utilization. What does this indicate?

A The job is CPU-bound and needs more CPU resources.

B The job is spending too much time cleaning memory, indicating possible memory leaks or inefficient memory use.

C The job is I/O-bound and waiting on disk operations.

D The job is network-bound due to data shuffling between nodes.

💡 Hint

High GC time usually relates to memory management issues.

Long garbage collection times with low CPU usage suggest the job spends time managing memory inefficiently, possibly due to memory leaks or large object creation.

Practice

1. What is the main purpose of Google Dataproc in data processing?

easy

A. To run Spark and Hadoop jobs easily in the cloud

B. To store large amounts of data permanently

C. To create visual dashboards for data

D. To write SQL queries on local machines

Solution

1. Step 1: Understand Google Dataproc's role

   Google Dataproc is designed to simplify running big data jobs like Spark and Hadoop in the cloud.

2. Step 2: Compare options with this role

   Only To run Spark and Hadoop jobs easily in the cloud describes running Spark and Hadoop jobs easily in the cloud, which matches Dataproc's main purpose.

3. Final Answer:

   To run Spark and Hadoop jobs easily in the cloud -> Option A

4. Quick Check:

   Dataproc = Run Spark/Hadoop jobs [OK]

Hint: Remember Dataproc runs Spark/Hadoop jobs in cloud [OK]

Common Mistakes:

• Confusing Dataproc with storage services
• Thinking Dataproc creates dashboards
• Assuming Dataproc runs queries locally

2. Which of the following is the correct way to create a cluster in Google Dataproc using gcloud CLI?

easy

A. gcloud dataproc clusters create my-cluster --region=us-central1

B. gcloud dataproc create cluster my-cluster --zone=us-central1

C. gcloud create dataproc cluster my-cluster --region=us-central1

D. gcloud clusters dataproc create my-cluster --region=us-central1

Solution

1. Step 1: Recall correct gcloud command syntax

   The correct command to create a Dataproc cluster is 'gcloud dataproc clusters create' followed by the cluster name and region.

2. Step 2: Match options with correct syntax

   gcloud dataproc clusters create my-cluster --region=us-central1 matches the correct syntax exactly. Other options have wrong order or missing keywords.

3. Final Answer:

   gcloud dataproc clusters create my-cluster --region=us-central1 -> Option A

4. Quick Check:

   Correct gcloud create cluster syntax = gcloud dataproc clusters create my-cluster --region=us-central1 [OK]

Hint: Use 'gcloud dataproc clusters create' to make clusters [OK]

Common Mistakes:

• Mixing order of commands
• Using 'create cluster' instead of 'clusters create'
• Confusing region and zone flags

3. Given the following code snippet to submit a Spark job on Dataproc, what will be the output if the cluster name is incorrect?

gcloud dataproc jobs submit spark --cluster wrong-cluster --region us-central1 --class org.apache.spark.examples.SparkPi --jars file:///usr/lib/spark/examples/jars/spark-examples.jar -- 1000

medium

A. Job runs successfully and outputs Pi estimation

B. Error: Cluster 'wrong-cluster' not found

C. Job runs but with default cluster

D. Syntax error in command

Solution

1. Step 1: Understand job submission with cluster name

   The command submits a Spark job to a specific cluster named 'wrong-cluster'. If this cluster does not exist, the job cannot run.

2. Step 2: Predict behavior when cluster name is wrong

   Dataproc will return an error stating the cluster is not found, so the job will not run.

3. Final Answer:

   Error: Cluster 'wrong-cluster' not found -> Option B

4. Quick Check:

   Wrong cluster name = Cluster not found error [OK]

Hint: Check cluster name carefully before submitting jobs [OK]

Common Mistakes:

• Assuming job runs on default cluster
• Thinking command syntax is wrong
• Ignoring error messages about cluster

4. You tried to delete a Dataproc cluster using the command:

gcloud dataproc clusters delete my-cluster --region us-central1

But you get an error saying the cluster is busy. What is the likely cause and how to fix it?

medium

A. Region flag is incorrect; remove the region flag

B. You need to restart the cluster before deleting

C. Cluster name is misspelled; rename the cluster

D. Cluster is running jobs; wait for jobs to finish or cancel them

Solution

1. Step 1: Understand why cluster deletion can fail

   If a cluster is busy running jobs, Dataproc prevents deletion to avoid interrupting work.

2. Step 2: Identify correct fix

   Wait for running jobs to finish or cancel them before deleting the cluster. Other options do not address the busy state.

3. Final Answer:

   Cluster is running jobs; wait for jobs to finish or cancel them -> Option D

4. Quick Check:

   Busy cluster = wait or cancel jobs before delete [OK]

Hint: Stop running jobs before deleting cluster [OK]

Common Mistakes:

• Ignoring running jobs causing busy state
• Removing region flag unnecessarily
• Trying to rename cluster to fix deletion

5. You want to process a large dataset using Spark on Google Dataproc. To save costs, which approach is best?

hard

A. Create multiple clusters for each small job and keep them all running

B. Create a cluster and keep it running 24/7 for all future jobs

C. Create a cluster, run the job, then delete the cluster immediately after

D. Run jobs locally to avoid cloud costs

Solution

1. Step 1: Understand cost management in Dataproc

   Dataproc charges for cluster uptime. Keeping clusters running when not needed increases cost.

2. Step 2: Choose cost-saving approach

   Creating a cluster only when needed, running the job, then deleting it saves money by not paying for idle time.

3. Final Answer:

   Create a cluster, run the job, then delete the cluster immediately after -> Option C

4. Quick Check:

   Delete cluster after job to save cost [OK]

Hint: Delete clusters after jobs to avoid extra charges [OK]

Common Mistakes:

• Leaving clusters running idle
• Running jobs locally ignoring cloud benefits
• Creating many clusters and not deleting them