Databricks platform overview in Apache Spark - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to create a Spark session in Databricks.

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName([1]).getOrCreate()

Drag options to blanks, or click blank then click option'

Aspark

BMyApp

C"MyApp"

DappName

Attempts:

3 left

💡 Hint

Common Mistakes

Forgetting to put the app name in quotes.

Using a variable name without quotes.

The appName method requires a string argument to name the Spark application.

2fill in blank

medium

Complete the code to read a CSV file into a DataFrame in Databricks.

Apache Spark

df = spark.read.format([1]).option("header", "true").load("/mnt/data/sample.csv")

Drag options to blanks, or click blank then click option'

A"json"

B"csv"

C"parquet"

D"text"

Attempts:

3 left

💡 Hint

Common Mistakes

Using the wrong format like "json" or "parquet".

Not putting the format name in quotes.

To read a CSV file, the format must be set to "csv".

3fill in blank

hard

Fix the error in the code to display the first 5 rows of the DataFrame.

Apache Spark

df.[1](5).show()

Drag options to blanks, or click blank then click option'

Alimit

Bhead

Cshow

Dtake

Attempts:

3 left

💡 Hint

Common Mistakes

Using head(5) which returns a list, not a DataFrame.

Using 'show', which returns None and causes an AttributeError when chaining .show().

The limit method limits the DataFrame to the first n rows and returns a DataFrame, so show() can be called after it.

4fill in blank

hard

Fill both blanks to create a DataFrame with only rows where the age is greater than 30.

Apache Spark

filtered_df = df.filter(df.[1] [2] 30)

Drag options to blanks, or click blank then click option'

Aage

B>

C<

Dsalary

Attempts:

3 left

💡 Hint

Common Mistakes

Using the wrong column name like 'salary'.

Using the less than operator '<' instead of '>'.

To filter rows where age is greater than 30, use df.age > 30 inside filter().

5fill in blank

hard

Fill all three blanks to create a dictionary comprehension that maps each word to its length if the length is greater than 3.

Apache Spark

lengths = { [1]: [2] for [3] in words if len([3]) > 3 }

Drag options to blanks, or click blank then click option'

Aword

Blen(word)

Ditem

Attempts:

3 left

💡 Hint

Common Mistakes

Using different variable names inconsistently.

Mapping key to the wrong value.

The comprehension uses 'word' as the variable, maps word to len(word), and iterates over words with 'word'.

Practice

1. What is the main purpose of the Databricks platform?

easy

A. To only store data without processing

B. To combine data storage, processing, and analysis in one platform

C. To replace Apache Spark with a new engine

D. To provide a simple text editor for coding

Solution

1. Step 1: Understand Databricks platform features

   Databricks integrates data storage, processing, and analysis in one place.

2. Step 2: Compare options with platform purpose

   Only To combine data storage, processing, and analysis in one platform correctly describes this combined purpose.

3. Final Answer:

   To combine data storage, processing, and analysis in one platform -> Option B

4. Quick Check:

   Databricks = Unified data platform [OK]

Hint: Remember Databricks is an all-in-one data platform [OK]

Common Mistakes:

• Thinking Databricks only stores data
• Confusing Databricks with just a code editor
• Believing it replaces Apache Spark

2. Which of the following is the correct way to start a cluster in Databricks?

easy

A. Click the 'Start Cluster' button in the Clusters UI

B. Run start_cluster() in a notebook cell

C. Use the command line to type cluster start

D. Clusters start automatically without any action

Solution

1. Step 1: Recall Databricks cluster management

   Clusters are started via the UI by clicking the 'Start Cluster' button.

2. Step 2: Evaluate other options

   Running start_cluster() or command line commands is not standard; clusters do not start automatically.

3. Final Answer:

   Click the 'Start Cluster' button in the Clusters UI -> Option A

4. Quick Check:

   Start clusters via UI button [OK]

Hint: Use the UI button to start clusters, not code commands [OK]

Common Mistakes:

• Trying to start clusters with code commands
• Assuming clusters start automatically
• Confusing cluster start with notebook run

3. Given the following code in a Databricks notebook:

df = spark.range(5)
df.show()

What will be the output?

medium

A. A table showing numbers from 0 to 4

B. An error because spark.range is not valid

C. An empty table with no rows

D. A table showing numbers from 1 to 5

Solution

1. Step 1: Understand spark.range(5)

   This creates a DataFrame with numbers starting at 0 up to 4 (5 numbers total).

2. Step 2: Analyze df.show() output

   It displays the DataFrame rows, showing numbers 0,1,2,3,4.

3. Final Answer:

   A table showing numbers from 0 to 4 -> Option A

4. Quick Check:

   spark.range(5) = 0 to 4 [OK]

Hint: spark.range(n) creates numbers from 0 to n-1 [OK]

Common Mistakes:

• Thinking range starts at 1
• Expecting an error from spark.range
• Assuming empty output

4. You run this code in a Databricks notebook:

df = spark.range(3)
df.show(5)

But it only shows 3 rows. Why?

medium

A. The cluster is too small to show 5 rows

B. show(5) is invalid syntax and ignored

C. The DataFrame only has 3 rows, so show(5) shows all available rows

D. You must call collect() before show()

Solution

1. Step 1: Check DataFrame size

   spark.range(3) creates 3 rows: 0,1,2.

2. Step 2: Understand show(5) behavior

   show(5) tries to show 5 rows but only 3 exist, so it shows all 3.

3. Final Answer:

   The DataFrame only has 3 rows, so show(5) shows all available rows -> Option C

4. Quick Check:

   show(n) shows up to n rows [OK]

Hint: show(n) displays up to n rows, not more than exist [OK]

Common Mistakes:

• Thinking show(5) causes error
• Believing cluster size limits show output
• Forgetting DataFrame row count

5. You want to analyze a large dataset using Databricks. Which combination of features helps you write code and run it on powerful machines easily?

hard

A. Manual data download and local scripts

B. Only notebooks without clusters

C. Clusters without notebooks

D. Notebooks and clusters

Solution

1. Step 1: Identify tools for coding and execution

   Notebooks let you write code interactively; clusters provide powerful machines to run code.

2. Step 2: Evaluate options

   Only Notebooks and clusters combines both notebooks and clusters, enabling easy coding and execution on powerful resources.

3. Final Answer:

   Notebooks and clusters -> Option D

4. Quick Check:

   Notebooks + clusters = easy code + power [OK]

Hint: Use notebooks for code and clusters for power [OK]

Common Mistakes:

• Ignoring clusters and using only notebooks
• Trying to run code without clusters
• Downloading data manually instead of using platform