Databricks platform overview in Apache Spark - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

What is the primary purpose of Databricks?

Databricks is a platform built on Apache Spark. What is its main goal?

A To serve only as a cloud storage service for big data files.

B To replace all SQL databases with a new query language.

C To act as a visualization tool for business dashboards only.

D To provide a unified environment for data engineering, data science, and machine learning.

💡 Hint

Think about what tasks Databricks helps teams do together on big data.

Databricks combines data engineering, data science, and machine learning in one platform, making it easier to collaborate and process big data using Apache Spark.

❓ Predict Output

intermediate

What is the output of this Databricks notebook cell?

Consider this PySpark code run in a Databricks notebook:

data = [(1, 'apple'), (2, 'banana'), (3, 'cherry')]
df = spark.createDataFrame(data, ['id', 'fruit'])
df.filter(df.id > 1).count()

What number will this output?

Apache Spark

data = [(1, 'apple'), (2, 'banana'), (3, 'cherry')]
df = spark.createDataFrame(data, ['id', 'fruit'])
df.filter(df.id > 1).count()

A 2

B 3

C 1

D 0

💡 Hint

Count rows where id is greater than 1.

Rows with id 2 and 3 satisfy the condition, so count is 2.

❓ data_output

advanced

What does this Spark SQL query return in Databricks?

Given a table sales with columns region and amount, what is the output of this query?

SELECT region, SUM(amount) AS total_sales
FROM sales
GROUP BY region
ORDER BY total_sales DESC
LIMIT 2

A The top 2 regions with the highest total sales amounts.

B All regions sorted alphabetically with their total sales.

C The total sales amount for all regions combined.

D An error because LIMIT cannot be used with GROUP BY.

💡 Hint

Look at the GROUP BY, ORDER BY, and LIMIT clauses.

The query groups sales by region, sums amounts, orders by total sales descending, and returns only the top 2 regions.

🔧 Debug

advanced

Why does this Databricks PySpark code raise an error?

Look at this code snippet:

df = spark.createDataFrame([(1, 'a'), (2, 'b')], ['num', 'char'])
df.select('num' + 1).show()

What error will this code produce?

A SyntaxError: invalid syntax in select statement

B TypeError: can only concatenate str (not "int") to str

C AnalysisException: cannot resolve 'num + 1' due to data type mismatch

D AttributeError: 'DataFrame' object has no attribute 'num'

💡 Hint

Consider how to add a number to a column in PySpark.

In PySpark, you cannot use Python operators directly on columns. You must use column expressions like df.num + 1 or col('num') + 1. The code tries to add 1 to a string literal 'num', causing a TypeError.

🚀 Application

expert

How to optimize a large join operation in Databricks?

You have two large DataFrames in Databricks and want to join them efficiently. Which approach will best improve performance?

A Write both DataFrames to CSV files and join using file system commands.

B Convert both DataFrames to Pandas and join locally.

C Use broadcast join by broadcasting the smaller DataFrame to all nodes.

D Use a nested loop join by collecting both DataFrames to the driver.

💡 Hint

Think about how Spark handles joins with data size differences.

Broadcast join sends the smaller DataFrame to all worker nodes, avoiding expensive shuffles and speeding up the join.