Google Dataproc overview in Apache Spark - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

What is the primary purpose of Google Dataproc?

Google Dataproc is a managed service for running which type of workloads?

A Running big data processing jobs using Apache Spark and Hadoop clusters.

B Hosting web applications with automatic scaling.

C Storing large amounts of unstructured data in a NoSQL database.

D Providing real-time analytics dashboards for business intelligence.

💡 Hint

Think about what Apache Spark and Hadoop are used for.

Google Dataproc is designed to run big data processing workloads using Apache Spark and Hadoop clusters in a managed environment.

❓ data_output

intermediate

Output of a Spark job on Dataproc cluster

Given the following PySpark code running on a Dataproc cluster, what is the output?

data = ["apple", "banana", "apple", "orange", "banana", "apple"]
rdd = spark.sparkContext.parallelize(data)
counts = rdd.map(lambda x: (x, 1)).reduceByKey(lambda a, b: a + b).collect()
print(sorted(counts))

Apache Spark

data = ["apple", "banana", "apple", "orange", "banana", "apple"]
rdd = spark.sparkContext.parallelize(data)
counts = rdd.map(lambda x: (x, 1)).reduceByKey(lambda a, b: a + b).collect()
print(sorted(counts))

A [('apple', 3), ('banana', 2), ('orange', 1)]

B [('apple', 1), ('banana', 1), ('orange', 1)]

C [('apple', 3), ('banana', 3), ('orange', 1)]

D [('apple', 2), ('banana', 2), ('orange', 2)]

💡 Hint

Count how many times each fruit appears in the list.

The code counts occurrences of each fruit. 'apple' appears 3 times, 'banana' 2 times, and 'orange' once.

🔧 Debug

advanced

Identify the error in Dataproc Spark job submission

Which option describes the error when submitting a Spark job to Dataproc with this command?

gcloud dataproc jobs submit spark --cluster=my-cluster --region=us-central1 --class=org.apache.spark.examples.SparkPi --jars=file:///usr/lib/spark/examples/jars/spark-examples.jar -- 1000

A The argument '1000' should be passed before the --jars option.

B The --class argument is missing the main class name.

C The --jars argument uses a local file path that is not accessible to the cluster nodes.

D The --region flag is not supported in gcloud dataproc jobs submit spark.

💡 Hint

Consider where the cluster nodes can access files specified in --jars.

The --jars option requires a path accessible by cluster nodes, usually a GCS path, not a local file path.

🚀 Application

advanced

Choosing the right Dataproc cluster configuration

You need to process a large batch of data with Apache Spark on Dataproc. The job requires high memory and CPU resources but runs only for a short time. Which cluster configuration is best?

A Use a small cluster with preemptible worker nodes to save cost.

B Use a large cluster with high-memory machine types and disable autoscaling.

C Use a single-node cluster with high-CPU machine type.

D Use a large cluster with standard worker nodes and enable autoscaling.

💡 Hint

Think about balancing cost, performance, and job duration.

A large cluster with standard nodes and autoscaling provides enough resources and adjusts size to optimize cost for short jobs.

❓ visualization

expert

Interpreting Dataproc job monitoring dashboard

In the Dataproc job monitoring dashboard, you see a Spark job with a long GC (Garbage Collection) time and low CPU utilization. What does this indicate?

A The job is CPU-bound and needs more CPU resources.

B The job is spending too much time cleaning memory, indicating possible memory leaks or inefficient memory use.

C The job is I/O-bound and waiting on disk operations.

D The job is network-bound due to data shuffling between nodes.

💡 Hint

High GC time usually relates to memory management issues.

Long garbage collection times with low CPU usage suggest the job spends time managing memory inefficiently, possibly due to memory leaks or large object creation.