Dataproc for Spark/Hadoop in GCP - Step-by-Step Execution

Process Flow - Dataproc for Spark/Hadoop

User submits job

↓

Dataproc cluster receives job

↓

Job scheduled on cluster nodes

↓

Spark/Hadoop processes data

↓

Results stored in output location

↓

User retrieves results

This flow shows how a user submits a Spark or Hadoop job to a Dataproc cluster, which processes the data and stores the results.

Execution Sample

GCP

gcloud dataproc jobs submit spark --cluster=my-cluster --class=org.apache.spark.examples.SparkPi --region=us-central1 --jars=gs://dataproc-examples-2.0/jars/spark-examples_2.12-3.3.1.jar 1000

This command submits a Spark job to a Dataproc cluster to calculate Pi using 1000 samples.

Process Table

Step	Action	Input/Condition	Result/Output
1	Submit job	User runs gcloud command	Job sent to Dataproc cluster
2	Cluster receives job	Job arrives at cluster	Job queued for execution
3	Schedule job	Cluster resources available	Job assigned to nodes
4	Run Spark job	SparkPi class runs with 1000 samples	Pi calculated approximately
5	Store results	Job completes successfully	Output saved to storage
6	Retrieve results	User checks output location	User gets Pi result
7	Exit	Job finished	No more actions

💡 Job finishes after results are stored and retrieved

Status Tracker

Variable	Start	After Step 2	After Step 4	Final
Job Status	Not submitted	Queued	Running	Completed
Pi Value	N/A	N/A	3.14 approx	3.14 approx
Output Location	Empty	Empty	Empty	Contains result file

Key Moments - 3 Insights

Why does the job status change from 'Queued' to 'Running'?

Because the cluster assigns the job to nodes when resources become available, as shown in execution_table step 3 and 4.

Where are the results stored after the job completes?

Results are stored in the output location, typically cloud storage, as shown in execution_table step 5.

What happens if the cluster has no available resources?

The job stays in the queue until resources free up, delaying step 3 scheduling.

Visual Quiz - 3 Questions

Look at the execution_table, what is the job status after step 4?

A Completed

B Queued

C Running

D Not submitted

💡 Hint

Check the 'Job Status' variable in variable_tracker after step 4

At which step are the results stored in the output location?

A Step 5

B Step 4

C Step 3

D Step 6

💡 Hint

Look at execution_table row describing 'Store results'

If the user changes the sample size from 1000 to 10000, how would the execution_table change?

A Step 2 would be skipped

B Step 4 would take longer to run

C Step 5 would not store results

D Job would not be submitted

💡 Hint

Increasing samples affects processing time in step 4, see 'Run Spark job' action

Concept Snapshot

Dataproc runs Spark/Hadoop jobs on managed clusters.
User submits job via gcloud or API.
Cluster schedules and runs job on nodes.
Results saved to cloud storage.
Simple, scalable big data processing in the cloud.

Full Transcript

Dataproc lets users run Spark or Hadoop jobs easily on Google Cloud. The user submits a job, which the Dataproc cluster receives and queues. When resources are free, the cluster schedules the job on nodes. The Spark or Hadoop job processes data, for example calculating Pi with SparkPi. After processing, results are saved to cloud storage. The user can then retrieve the results. The job status changes from not submitted, to queued, to running, and finally completed. If resources are busy, the job waits in queue. Changing job parameters like sample size affects processing time. This flow simplifies big data processing by managing infrastructure automatically.