Execution date vs logical date in Apache Airflow - Hands-On Comparison

Understanding Execution Date vs Logical Date in Airflow

📖 Scenario: You are working with Apache Airflow to schedule data pipelines. Airflow uses two important dates: execution date and logical date. Understanding these helps you know when your tasks run and what data they process.

🎯 Goal: Build a simple Airflow DAG that prints both the execution date and logical date for each run. This will help you see the difference between these two dates in practice.

📋 What You'll Learn

Create a DAG with a fixed schedule interval

Use Airflow's execution_date and logical_date in a PythonOperator

Print both dates in the task's Python function

Run the DAG and observe the output

💡 Why This Matters

🌍 Real World

In real data pipelines, knowing the execution and logical dates helps you process the correct data for each scheduled run, especially when backfilling or handling late data.

💼 Career

Understanding these dates is essential for data engineers and DevOps professionals managing workflows in Airflow to ensure data accuracy and pipeline reliability.

1

Create a basic Airflow DAG with a schedule

Create a DAG called date_example_dag with a schedule_interval of @daily and a start_date of 2024-01-01. Import necessary Airflow modules.

Apache Airflow

# Your code here

Hint

Use from airflow import DAG and from datetime import datetime. Define the DAG with with DAG(...):.

2

Add a Python function to print execution and logical dates

Define a Python function called print_dates that takes **kwargs and prints execution_date and logical_date from kwargs['logical_date'] and kwargs['execution_date']. Add a PythonOperator named print_dates_task that calls this function.

Apache Airflow

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

def print_dates(**kwargs):
    # Your code here

with DAG(
    dag_id='date_example_dag',
    schedule_interval='@daily',
    start_date=datetime(2024, 1, 1),
    catchup=False
) as dag:
    # Your code here

Hint

Use kwargs['execution_date'] and kwargs['logical_date'] inside the function. Pass the function to PythonOperator.

3

Set task to receive Airflow context for dates

Modify the print_dates_task to set provide_context=True so the function receives Airflow context including execution_date and logical_date.

Apache Airflow

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

def print_dates(**kwargs):
    execution_date = kwargs['execution_date']
    logical_date = kwargs['logical_date']
    print(f"Execution date: {execution_date}")
    print(f"Logical date: {logical_date}")

with DAG(
    dag_id='date_example_dag',
    schedule_interval='@daily',
    start_date=datetime(2024, 1, 1),
    catchup=False
) as dag:
    print_dates_task = PythonOperator(
        task_id='print_dates_task',
        python_callable=print_dates,
        # Your code here
    )

Hint

Add provide_context=True to the PythonOperator so the function gets the context.

4

Run the DAG and print the dates

Add a print statement to run the DAG's print_dates_task manually by calling print_dates() with a sample context dictionary containing 'execution_date' and 'logical_date' set to datetime(2024, 1, 2). Print the output of the function.

Apache Airflow

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

def print_dates(**kwargs):
    execution_date = kwargs['execution_date']
    logical_date = kwargs['logical_date']
    print(f"Execution date: {execution_date}")
    print(f"Logical date: {logical_date}")

with DAG(
    dag_id='date_example_dag',
    schedule_interval='@daily',
    start_date=datetime(2024, 1, 1),
    catchup=False
) as dag:
    print_dates_task = PythonOperator(
        task_id='print_dates_task',
        python_callable=print_dates,
        provide_context=True
    )

# Your code here

Hint

Call print_dates() with execution_date=datetime(2024, 1, 2) and logical_date=datetime(2024, 1, 2) to see the printed dates.

Practice

1. What does the logical date in Airflow represent?

easy

A. The time period the task is processing data for

B. The exact time the task started running

C. The current system time when the task runs

D. The time when the DAG was created

Solution

1. Step 1: Understand the definition of logical date

   The logical date is the timestamp representing the data interval the task is working on, not when it runs.

2. Step 2: Differentiate from execution date

   Execution date is when the task actually runs, but logical date points to the data period it processes.

3. Final Answer:

   The time period the task is processing data for -> Option A

4. Quick Check:

   Logical date = data period [OK]

Hint: Logical date = data period, execution date = run time [OK]

Common Mistakes:

• Confusing logical date with actual run time
• Thinking logical date is current system time
• Mixing execution date with DAG creation time

2. Which of the following is the correct way to access the logical date inside an Airflow task?

easy

A. Using context['execution_date']

B. Using context['logical_date']

C. Using context['run_time']

D. Using context['start_date']

Solution

1. Step 1: Recall Airflow context variables

   Airflow provides logical_date in the context to represent the logical date of the task instance.

2. Step 2: Identify the correct key for logical date

   The key 'logical_date' is used to access the logical date, while 'execution_date' is deprecated in newer versions.

3. Final Answer:

   Using context['logical_date'] -> Option B

4. Quick Check:

   Access logical date with context['logical_date'] [OK]

Hint: Use context['logical_date'] for logical date in Airflow 2.2+ [OK]

Common Mistakes:

• Using deprecated 'execution_date' key
• Trying to access logical date with 'run_time'
• Confusing start_date with logical date

3. Given a DAG scheduled daily at midnight, if a task runs on 2024-06-10 00:00 UTC, what is its logical date?

medium

A. 2024-06-10 00:00 UTC

B. 2024-06-11 00:00 UTC

C. 2024-06-09 00:00 UTC

D. The current system time when the task runs

Solution

1. Step 1: Understand daily scheduling in Airflow

   A DAG scheduled daily at midnight runs for the data of the previous day, so the logical date is the start of that previous day.

2. Step 2: Calculate logical date for run on 2024-06-10 00:00 UTC

   The run at midnight on June 10 processes data for June 9, so logical date is 2024-06-09 00:00 UTC.

3. Final Answer:

   2024-06-09 00:00 UTC -> Option C

4. Quick Check:

   Run time = June 10, logical date = June 9 [OK]

Hint: Logical date is previous day for daily midnight runs [OK]

Common Mistakes:

• Assuming logical date equals execution date
• Using current system time as logical date
• Confusing next day as logical date

4. You see a task's logs showing execution_date as 2024-06-10 but the data processed is from 2024-06-11. What is the likely issue?

medium

A. The task is running twice for the same data

B. The DAG schedule interval is set to hourly instead of daily

C. The system clock is incorrect

D. The task is using execution_date instead of logical_date for data filtering

Solution

1. Step 1: Analyze mismatch between execution_date and data date

   If data processed is from a later date than execution_date, the task likely uses execution_date (run time) instead of logical_date (data period) for filtering.

2. Step 2: Understand correct usage of logical date

   Logical date should be used to select data for the intended period; using execution_date causes off-by-one errors.

3. Final Answer:

   The task is using execution_date instead of logical_date for data filtering -> Option D

4. Quick Check:

   Use logical_date for data period, not execution_date [OK]

Hint: Use logical_date for data filtering, not execution_date [OK]

Common Mistakes:

• Ignoring difference between execution and logical dates
• Assuming schedule interval causes date mismatch
• Blaming system clock without checking code

5. You want to process hourly data with a DAG scheduled every hour. The task should process data for the previous hour. Which date should you use inside the task to filter data correctly?

hard

A. Use logical_date which represents the data interval start

B. Use DAG start_date

C. Use current system time when the task runs

D. Use execution_date which is the task run time

Solution

1. Step 1: Understand hourly scheduling and data processing

   For hourly DAGs, the logical date marks the start of the hour for which data is processed, typically the previous hour.

2. Step 2: Choose the correct date for filtering data

   Using logical_date ensures the task filters data for the correct hour interval, not the actual run time.

3. Final Answer:

   Use logical_date which represents the data interval start -> Option A

4. Quick Check:

   Hourly data uses logical_date for correct filtering [OK]

Hint: For hourly data, always use logical_date for filtering [OK]

Common Mistakes:

• Using execution_date causes off-by-one hour errors
• Using current time ignores schedule intervals
• Confusing DAG start_date with data interval