Databricks platform overview in Apache Spark - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

What is the primary purpose of Databricks?

Databricks is a platform built on Apache Spark. What is its main goal?

A To serve only as a cloud storage service for big data files.

B To replace all SQL databases with a new query language.

C To act as a visualization tool for business dashboards only.

D To provide a unified environment for data engineering, data science, and machine learning.

💡 Hint

Think about what tasks Databricks helps teams do together on big data.

Databricks combines data engineering, data science, and machine learning in one platform, making it easier to collaborate and process big data using Apache Spark.

❓ Predict Output

intermediate

What is the output of this Databricks notebook cell?

Consider this PySpark code run in a Databricks notebook:

data = [(1, 'apple'), (2, 'banana'), (3, 'cherry')]
df = spark.createDataFrame(data, ['id', 'fruit'])
df.filter(df.id > 1).count()

What number will this output?

Apache Spark

data = [(1, 'apple'), (2, 'banana'), (3, 'cherry')]
df = spark.createDataFrame(data, ['id', 'fruit'])
df.filter(df.id > 1).count()

A 2

B 3

C 1

D 0

💡 Hint

Count rows where id is greater than 1.

Rows with id 2 and 3 satisfy the condition, so count is 2.

❓ data_output

advanced

What does this Spark SQL query return in Databricks?

Given a table sales with columns region and amount, what is the output of this query?

SELECT region, SUM(amount) AS total_sales
FROM sales
GROUP BY region
ORDER BY total_sales DESC
LIMIT 2

A The top 2 regions with the highest total sales amounts.

B All regions sorted alphabetically with their total sales.

C The total sales amount for all regions combined.

D An error because LIMIT cannot be used with GROUP BY.

💡 Hint

Look at the GROUP BY, ORDER BY, and LIMIT clauses.

The query groups sales by region, sums amounts, orders by total sales descending, and returns only the top 2 regions.

🔧 Debug

advanced

Why does this Databricks PySpark code raise an error?

Look at this code snippet:

df = spark.createDataFrame([(1, 'a'), (2, 'b')], ['num', 'char'])
df.select('num' + 1).show()

What error will this code produce?

A SyntaxError: invalid syntax in select statement

B TypeError: can only concatenate str (not "int") to str

C AnalysisException: cannot resolve 'num + 1' due to data type mismatch

D AttributeError: 'DataFrame' object has no attribute 'num'

💡 Hint

Consider how to add a number to a column in PySpark.

In PySpark, you cannot use Python operators directly on columns. You must use column expressions like df.num + 1 or col('num') + 1. The code tries to add 1 to a string literal 'num', causing a TypeError.

🚀 Application

expert

How to optimize a large join operation in Databricks?

You have two large DataFrames in Databricks and want to join them efficiently. Which approach will best improve performance?

A Write both DataFrames to CSV files and join using file system commands.

B Convert both DataFrames to Pandas and join locally.

C Use broadcast join by broadcasting the smaller DataFrame to all nodes.

D Use a nested loop join by collecting both DataFrames to the driver.

💡 Hint

Think about how Spark handles joins with data size differences.

Broadcast join sends the smaller DataFrame to all worker nodes, avoiding expensive shuffles and speeding up the join.

Practice

1. What is the main purpose of the Databricks platform?

easy

A. To only store data without processing

B. To combine data storage, processing, and analysis in one platform

C. To replace Apache Spark with a new engine

D. To provide a simple text editor for coding

Solution

1. Step 1: Understand Databricks platform features

   Databricks integrates data storage, processing, and analysis in one place.

2. Step 2: Compare options with platform purpose

   Only To combine data storage, processing, and analysis in one platform correctly describes this combined purpose.

3. Final Answer:

   To combine data storage, processing, and analysis in one platform -> Option B

4. Quick Check:

   Databricks = Unified data platform [OK]

Hint: Remember Databricks is an all-in-one data platform [OK]

Common Mistakes:

• Thinking Databricks only stores data
• Confusing Databricks with just a code editor
• Believing it replaces Apache Spark

2. Which of the following is the correct way to start a cluster in Databricks?

easy

A. Click the 'Start Cluster' button in the Clusters UI

B. Run start_cluster() in a notebook cell

C. Use the command line to type cluster start

D. Clusters start automatically without any action

Solution

1. Step 1: Recall Databricks cluster management

   Clusters are started via the UI by clicking the 'Start Cluster' button.

2. Step 2: Evaluate other options

   Running start_cluster() or command line commands is not standard; clusters do not start automatically.

3. Final Answer:

   Click the 'Start Cluster' button in the Clusters UI -> Option A

4. Quick Check:

   Start clusters via UI button [OK]

Hint: Use the UI button to start clusters, not code commands [OK]

Common Mistakes:

• Trying to start clusters with code commands
• Assuming clusters start automatically
• Confusing cluster start with notebook run

3. Given the following code in a Databricks notebook:

df = spark.range(5)
df.show()

What will be the output?

medium

A. A table showing numbers from 0 to 4

B. An error because spark.range is not valid

C. An empty table with no rows

D. A table showing numbers from 1 to 5

Solution

1. Step 1: Understand spark.range(5)

   This creates a DataFrame with numbers starting at 0 up to 4 (5 numbers total).

2. Step 2: Analyze df.show() output

   It displays the DataFrame rows, showing numbers 0,1,2,3,4.

3. Final Answer:

   A table showing numbers from 0 to 4 -> Option A

4. Quick Check:

   spark.range(5) = 0 to 4 [OK]

Hint: spark.range(n) creates numbers from 0 to n-1 [OK]

Common Mistakes:

• Thinking range starts at 1
• Expecting an error from spark.range
• Assuming empty output

4. You run this code in a Databricks notebook:

df = spark.range(3)
df.show(5)

But it only shows 3 rows. Why?

medium

A. The cluster is too small to show 5 rows

B. show(5) is invalid syntax and ignored

C. The DataFrame only has 3 rows, so show(5) shows all available rows

D. You must call collect() before show()

Solution

1. Step 1: Check DataFrame size

   spark.range(3) creates 3 rows: 0,1,2.

2. Step 2: Understand show(5) behavior

   show(5) tries to show 5 rows but only 3 exist, so it shows all 3.

3. Final Answer:

   The DataFrame only has 3 rows, so show(5) shows all available rows -> Option C

4. Quick Check:

   show(n) shows up to n rows [OK]

Hint: show(n) displays up to n rows, not more than exist [OK]

Common Mistakes:

• Thinking show(5) causes error
• Believing cluster size limits show output
• Forgetting DataFrame row count

5. You want to analyze a large dataset using Databricks. Which combination of features helps you write code and run it on powerful machines easily?

hard

A. Manual data download and local scripts

B. Only notebooks without clusters

C. Clusters without notebooks

D. Notebooks and clusters

Solution

1. Step 1: Identify tools for coding and execution

   Notebooks let you write code interactively; clusters provide powerful machines to run code.

2. Step 2: Evaluate options

   Only Notebooks and clusters combines both notebooks and clusters, enabling easy coding and execution on powerful resources.

3. Final Answer:

   Notebooks and clusters -> Option D

4. Quick Check:

   Notebooks + clusters = easy code + power [OK]

Hint: Use notebooks for code and clusters for power [OK]

Common Mistakes:

• Ignoring clusters and using only notebooks
• Trying to run code without clusters
• Downloading data manually instead of using platform