Date and timestamp functions in Apache Spark - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of date_add function in Spark

What is the output of the following Spark code snippet?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.functions import date_add, to_date

spark = SparkSession.builder.getOrCreate()
data = [("2024-06-15",)]
df = spark.createDataFrame(data, ["date_str"])
df = df.withColumn("date", to_date("date_str"))
df = df.withColumn("new_date", date_add("date", 10))
df.select("new_date").show()

A

+----------+
|  new_date|
+----------+
|2024-06-15|
+----------+

B

+----------+
|  new_date|
+----------+
|2024-06-25|
+----------+

C

+----------+
|  new_date|
+----------+
|2024-06-05|
+----------+

D

+----------+
|  new_date|
+----------+
|2024-07-15|
+----------+

💡 Hint

date_add adds days to a date column.

The date_add function adds the specified number of days (10) to the original date (2024-06-15), resulting in 2024-06-25.

❓ data_output

intermediate

Result of truncating timestamp to month

Given the following Spark DataFrame, what is the output after truncating the timestamp to the month?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.functions import trunc, to_timestamp

spark = SparkSession.builder.getOrCreate()
data = [("2024-06-15 13:45:30",)]
df = spark.createDataFrame(data, ["ts_str"])
df = df.withColumn("ts", to_timestamp("ts_str"))
df = df.withColumn("month_start", trunc("ts", "MM"))
df.select("month_start").show()

A

+-----------+
|month_start|
+-----------+
| 2024-06-01|
+-----------+

B

+-----------+
|month_start|
+-----------+
| 2024-06-15|
+-----------+

C

+-----------+
|month_start|
+-----------+
| 2024-01-01|
+-----------+

D

+-----------+
|month_start|
+-----------+
| 2024-07-01|
+-----------+

💡 Hint

trunc with 'MM' returns the first day of the month.

The trunc function with 'MM' truncates the timestamp to the first day of its month, so 2024-06-15 13:45:30 becomes 2024-06-01.

🔧 Debug

advanced

Identify the error in timestamp conversion

What error will the following Spark code raise when trying to convert a string to timestamp?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.functions import to_timestamp

spark = SparkSession.builder.getOrCreate()
data = [("2024-13-01 10:00:00",)]
df = spark.createDataFrame(data, ["ts_str"])
df = df.withColumn("ts", to_timestamp("ts_str"))
df.show()

A SyntaxError: invalid syntax in to_timestamp function call.

B ValueError: month must be in 1..12.

C RuntimeError: Timestamp format not supported.

D The ts column will contain null values due to invalid month.

💡 Hint

Invalid dates in Spark timestamp conversion result in nulls, not exceptions.

Spark's to_timestamp returns null for invalid date strings instead of raising an error.

🧠 Conceptual

advanced

Understanding unix_timestamp function output

What does the unix_timestamp function return when applied to a timestamp column in Spark?

A The number of seconds since January 1, 1970 UTC as a long integer.

B The timestamp formatted as a string in 'yyyy-MM-dd HH:mm:ss'.

C The timestamp converted to the local timezone string.

D The number of milliseconds since January 1, 1970 UTC as a long integer.

💡 Hint

unix_timestamp returns seconds, not milliseconds.

unix_timestamp returns the number of seconds since the Unix epoch (1970-01-01 00:00:00 UTC) as a long integer.

🚀 Application

expert

Calculate age in years from birthdate column

Given a Spark DataFrame with a birthdate column of type date, which code snippet correctly calculates the age in years as an integer?

A

from pyspark.sql.functions import col, year, current_date

df = df.withColumn('age', year(current_date()) - year(col('birthdate')))

B

from pyspark.sql.functions import col, datediff

df = df.withColumn('age', datediff(current_date(), col('birthdate')) // 365)

C

from pyspark.sql.functions import col, floor, months_between

df = df.withColumn('age', floor(months_between(current_date(), col('birthdate')) / 12))

D

from pyspark.sql.functions import col, to_date

df = df.withColumn('age', to_date(current_date()) - to_date(col('birthdate')))

💡 Hint

Use months_between and floor to get accurate age in years.

Option C uses months_between to get months difference, divides by 12, and floors the result to get full years accurately.