Date and timestamp functions in Apache Spark - Time & Space Complexity

Time Complexity: Date and timestamp functions

O(n)

Understanding Time Complexity

We want to understand how the time it takes to run date and timestamp functions changes as the data size grows.

How does the number of rows affect the work done by these functions?

Scenario Under Consideration

Analyze the time complexity of the following code snippet.

from pyspark.sql.functions import current_date, current_timestamp, date_add, date_sub

# Assume df is a DataFrame with many rows
result = df.select(
    current_date().alias('today'),
    current_timestamp().alias('now'),
    date_add(df['date_column'], 5).alias('date_plus_5'),
    date_sub(df['date_column'], 3).alias('date_minus_3')
)

This code applies several date and timestamp functions to each row of a DataFrame.

Identify Repeating Operations

• Primary operation: Applying date functions to each row in the DataFrame.
• How many times: Once per row, so as many times as there are rows.

How Execution Grows With Input

Each row requires the same fixed amount of work to compute the date functions.

Input Size (n)	Approx. Operations
10	10 operations
100	100 operations
1000	1000 operations

Pattern observation: The work grows directly with the number of rows. Double the rows, double the work.

Final Time Complexity

Time Complexity: O(n)

This means the time to run these date functions grows linearly with the number of rows in the data.

Common Mistake

[X] Wrong: "Date functions run once and don't depend on data size."

[OK] Correct: Each row needs its own date calculation, so the total work increases with more rows.

Interview Connect

Understanding how functions scale with data size helps you write efficient Spark code and explain performance clearly.

Self-Check

"What if we applied a date function only once to a single value instead of every row? How would the time complexity change?"