SQL queries on DataFrames in Apache Spark - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is the purpose of using SQL queries on DataFrames in Apache Spark?

SQL queries on DataFrames allow you to use familiar SQL syntax to analyze and manipulate data stored in Spark DataFrames, making it easier to perform complex data operations without writing complex code.

beginner

How do you register a DataFrame as a temporary view in Spark to run SQL queries on it?

You use the createOrReplaceTempView method on the DataFrame with a name, for example: df.createOrReplaceTempView("my_table"). This lets you run SQL queries on 'my_table'.

beginner

What Spark method do you use to run an SQL query on a registered DataFrame view?

You use spark.sql('your SQL query'). This returns a new DataFrame with the query results.

beginner

Can you explain how to select specific columns from a DataFrame using SQL queries in Spark?

After registering the DataFrame as a temporary view, you write a SQL query like SELECT column1, column2 FROM view_name. This returns only those columns in the result DataFrame.

intermediate

What is the difference between createOrReplaceTempView and createGlobalTempView in Spark?

createOrReplaceTempView creates a temporary view visible only in the current Spark session. createGlobalTempView creates a global temporary view accessible across different sessions but requires prefixing the view name with global_temp. when querying.

Which method registers a DataFrame as a temporary view in Spark?

A createOrReplaceTempView

B registerDataFrame

C createTempDataFrame

D registerTempSQL

The correct method to register a DataFrame as a temporary view is createOrReplaceTempView.

How do you run an SQL query on a registered DataFrame view in Spark?

A df.sql('SQL query')

B spark.sql('SQL query')

C spark.query('SQL query')

D df.querySQL('SQL query')

You use spark.sql('SQL query') to run SQL queries on registered views.

What does the SQL query SELECT * FROM my_table do on a DataFrame view?

A Creates a new table named my_table

B Deletes all data from the view

C Updates all rows in the view

D Selects all columns and rows from the view

SELECT * returns all columns and rows from the specified view.

Which prefix is required to query a global temporary view in Spark?

A global.

B temp_global.

C global_temp.

D temp.

Global temporary views require the prefix global_temp. when querying.

What type of object does spark.sql() return after running a query?

A DataFrame

B RDD

C List

D Dictionary

spark.sql() returns a DataFrame containing the query results.

Explain how to run an SQL query on a Spark DataFrame step-by-step.

Think about how SQL queries need a table name and how Spark lets you create that from a DataFrame.

Describe the difference between temporary and global temporary views in Spark.

Consider visibility and scope of views in Spark sessions.