Bird
Raised Fist0
Apache Airflowdevops~10 mins

Execution date vs logical date in Apache Airflow - Visual Side-by-Side Comparison

Choose your learning style10 modes available
LearnWhyDeepVisualTryChallengeProjectRecallTime
Process Flow - Execution date vs logical date
DAG scheduled at time T
↓
Execution Date = T - schedule interval
↓
Logical Date = Execution Date
↓
Task runs using Logical Date
↓
Next DAG run scheduled at T + schedule interval
Shows how Airflow schedules a DAG run at time T but assigns an execution date that is the previous schedule interval, called the logical date, which tasks use for data partitioning.
Execution Sample
Apache Airflow
from datetime import datetime, timedelta

schedule_interval = '@daily'
run_time = datetime(2024, 6, 2, 0, 0)
execution_date = run_time - timedelta(days=1)
logical_date = execution_date
print(f"Run at {run_time.strftime('%Y-%m-%d %H:%M')}, execution_date: {execution_date.strftime('%Y-%m-%d %H:%M')}, logical_date: {logical_date.strftime('%Y-%m-%d %H:%M')}")
This code simulates a daily DAG run at midnight June 2, 2024, showing execution and logical dates as June 1, 2024.
Process Table
StepRun Time (Actual)Execution DateLogical DateExplanation
12024-06-01 00:002024-05-31 00:002024-05-31 00:00First DAG run scheduled at June 1 midnight, execution date is previous day midnight
22024-06-02 00:002024-06-01 00:002024-06-01 00:00Second DAG run scheduled at June 2 midnight, execution date is June 1 midnight
32024-06-03 00:002024-06-02 00:002024-06-02 00:00Third DAG run scheduled at June 3 midnight, execution date is June 2 midnight
42024-06-04 00:002024-06-03 00:002024-06-03 00:00Fourth DAG run scheduled at June 4 midnight, execution date is June 3 midnight
52024-06-05 00:002024-06-04 00:002024-06-04 00:00Fifth DAG run scheduled at June 5 midnight, execution date is June 4 midnight
6N/AN/AN/ANo more runs in this example
💡 DAG runs at midnight each day, execution date is always previous day midnight, which is the logical date tasks use.
Status Tracker
VariableStartAfter 1After 2After 3After 4After 5
run_time2024-06-01 00:002024-06-02 00:002024-06-03 00:002024-06-04 00:002024-06-05 00:00N/A
execution_date2024-05-31 00:002024-06-01 00:002024-06-02 00:002024-06-03 00:002024-06-04 00:00N/A
logical_date2024-05-31 00:002024-06-01 00:002024-06-02 00:002024-06-03 00:002024-06-04 00:00N/A
Key Moments - 3 Insights
Why is the execution date one day before the actual run time?
Because Airflow schedules DAG runs at the end of the period, the execution date represents the start of the data interval. See execution_table rows 1-5 where execution_date is always one day before run_time.
Are execution date and logical date different?
No, in Airflow they are the same. Logical date is just another name for execution date used in task context. This is shown in the execution_table where both columns match exactly.
What happens if the schedule interval changes?
The gap between run_time and execution_date changes accordingly. For example, with '@hourly' schedule, execution_date is one hour before run_time. This example uses '@daily' so the gap is one day.
Visual Quiz - 3 Questions
Test your understanding
Look at the execution_table at step 3. What is the execution date?
A2024-06-03 00:00
B2024-06-02 00:00
C2024-06-01 00:00
D2024-06-04 00:00
💡 Hint
Check the 'Execution Date' column in row 3 of the execution_table.
At which run time does the execution date equal 2024-06-01 00:00?
A2024-06-01 00:00
B2024-06-03 00:00
C2024-06-02 00:00
D2024-06-04 00:00
💡 Hint
Look for execution_date = 2024-06-01 00:00 in the execution_table and find corresponding run_time.
If the schedule interval changed to '@hourly', how would execution_date relate to run_time?
Aexecution_date would be one hour before run_time
Bexecution_date would be the same as run_time
Cexecution_date would be one day after run_time
Dexecution_date would be two hours before run_time
💡 Hint
Recall the explanation in key_moments about how execution_date is run_time minus schedule interval.
Concept Snapshot
Airflow schedules DAG runs at fixed intervals.
Execution date is the start of the data interval, usually run_time minus schedule interval.
Logical date is another name for execution date.
Tasks use logical date to process data for that period.
This helps organize data pipelines by data intervals, not run times.
Full Transcript
In Airflow, the execution date is the date assigned to a DAG run that represents the start of the data interval it processes. Although the DAG runs at a scheduled time, called run_time, the execution date is usually the run_time minus the schedule interval. This execution date is also called the logical date. For example, a daily DAG scheduled to run at midnight on June 2, 2024, has an execution date of June 1, 2024. This means the DAG run processes data for June 1. The logical date is used inside tasks to reference the data partition. This concept helps keep data pipelines organized by the data they process rather than the time they run. The execution_table shows this step by step for five daily runs. The variable_tracker shows how run_time, execution_date, and logical_date change over runs. Key moments clarify common confusions about why execution date is before run time and the equivalence of execution and logical dates. The visual quiz tests understanding of these dates in the table and how schedule interval affects them.

Practice

(1/5)
1. What does the logical date in Airflow represent?
easy
A. The time period the task is processing data for
B. The exact time the task started running
C. The current system time when the task runs
D. The time when the DAG was created

Solution

  1. Step 1: Understand the definition of logical date

    The logical date is the timestamp representing the data interval the task is working on, not when it runs.

  2. Step 2: Differentiate from execution date

    Execution date is when the task actually runs, but logical date points to the data period it processes.

  3. Final Answer:

    The time period the task is processing data for -> Option A

  4. Quick Check:

    Logical date = data period [OK]
Hint: Logical date = data period, execution date = run time [OK]
Common Mistakes:
  • Confusing logical date with actual run time
  • Thinking logical date is current system time
  • Mixing execution date with DAG creation time
2. Which of the following is the correct way to access the logical date inside an Airflow task?
easy
A. Using context['execution_date']
B. Using context['logical_date']
C. Using context['run_time']
D. Using context['start_date']

Solution

  1. Step 1: Recall Airflow context variables

    Airflow provides logical_date in the context to represent the logical date of the task instance.

  2. Step 2: Identify the correct key for logical date

    The key 'logical_date' is used to access the logical date, while 'execution_date' is deprecated in newer versions.

  3. Final Answer:

    Using context['logical_date'] -> Option B

  4. Quick Check:

    Access logical date with context['logical_date'] [OK]
Hint: Use context['logical_date'] for logical date in Airflow 2.2+ [OK]
Common Mistakes:
  • Using deprecated 'execution_date' key
  • Trying to access logical date with 'run_time'
  • Confusing start_date with logical date
3. Given a DAG scheduled daily at midnight, if a task runs on 2024-06-10 00:00 UTC, what is its logical date?
medium
A. 2024-06-10 00:00 UTC
B. 2024-06-11 00:00 UTC
C. 2024-06-09 00:00 UTC
D. The current system time when the task runs

Solution

  1. Step 1: Understand daily scheduling in Airflow

    A DAG scheduled daily at midnight runs for the data of the previous day, so the logical date is the start of that previous day.

  2. Step 2: Calculate logical date for run on 2024-06-10 00:00 UTC

    The run at midnight on June 10 processes data for June 9, so logical date is 2024-06-09 00:00 UTC.

  3. Final Answer:

    2024-06-09 00:00 UTC -> Option C

  4. Quick Check:

    Run time = June 10, logical date = June 9 [OK]
Hint: Logical date is previous day for daily midnight runs [OK]
Common Mistakes:
  • Assuming logical date equals execution date
  • Using current system time as logical date
  • Confusing next day as logical date
4. You see a task's logs showing execution_date as 2024-06-10 but the data processed is from 2024-06-11. What is the likely issue?
medium
A. The task is running twice for the same data
B. The DAG schedule interval is set to hourly instead of daily
C. The system clock is incorrect
D. The task is using execution_date instead of logical_date for data filtering

Solution

  1. Step 1: Analyze mismatch between execution_date and data date

    If data processed is from a later date than execution_date, the task likely uses execution_date (run time) instead of logical_date (data period) for filtering.

  2. Step 2: Understand correct usage of logical date

    Logical date should be used to select data for the intended period; using execution_date causes off-by-one errors.

  3. Final Answer:

    The task is using execution_date instead of logical_date for data filtering -> Option D

  4. Quick Check:

    Use logical_date for data period, not execution_date [OK]
Hint: Use logical_date for data filtering, not execution_date [OK]
Common Mistakes:
  • Ignoring difference between execution and logical dates
  • Assuming schedule interval causes date mismatch
  • Blaming system clock without checking code
5. You want to process hourly data with a DAG scheduled every hour. The task should process data for the previous hour. Which date should you use inside the task to filter data correctly?
hard
A. Use logical_date which represents the data interval start
B. Use DAG start_date
C. Use current system time when the task runs
D. Use execution_date which is the task run time

Solution

  1. Step 1: Understand hourly scheduling and data processing

    For hourly DAGs, the logical date marks the start of the hour for which data is processed, typically the previous hour.

  2. Step 2: Choose the correct date for filtering data

    Using logical_date ensures the task filters data for the correct hour interval, not the actual run time.

  3. Final Answer:

    Use logical_date which represents the data interval start -> Option A

  4. Quick Check:

    Hourly data uses logical_date for correct filtering [OK]
Hint: For hourly data, always use logical_date for filtering [OK]
Common Mistakes:
  • Using execution_date causes off-by-one hour errors
  • Using current time ignores schedule intervals
  • Confusing DAG start_date with data interval
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard