Execution date vs logical date in Apache Airflow - Hands-On Comparison

Understanding Execution Date vs Logical Date in Airflow

📖 Scenario: You are working with Apache Airflow to schedule data pipelines. Airflow uses two important dates: execution date and logical date. Understanding these helps you know when your tasks run and what data they process.

🎯 Goal: Build a simple Airflow DAG that prints both the execution date and logical date for each run. This will help you see the difference between these two dates in practice.

📋 What You'll Learn

Create a DAG with a fixed schedule interval

Use Airflow's execution_date and logical_date in a PythonOperator

Print both dates in the task's Python function

Run the DAG and observe the output

💡 Why This Matters

🌍 Real World

In real data pipelines, knowing the execution and logical dates helps you process the correct data for each scheduled run, especially when backfilling or handling late data.

💼 Career

Understanding these dates is essential for data engineers and DevOps professionals managing workflows in Airflow to ensure data accuracy and pipeline reliability.

1

Create a basic Airflow DAG with a schedule

Create a DAG called date_example_dag with a schedule_interval of @daily and a start_date of 2024-01-01. Import necessary Airflow modules.

Apache Airflow

# Your code here

Need a hint?

Use from airflow import DAG and from datetime import datetime. Define the DAG with with DAG(...):.

2

Add a Python function to print execution and logical dates

Define a Python function called print_dates that takes **kwargs and prints execution_date and logical_date from kwargs['logical_date'] and kwargs['execution_date']. Add a PythonOperator named print_dates_task that calls this function.

Apache Airflow

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

def print_dates(**kwargs):
    # Your code here

with DAG(
    dag_id='date_example_dag',
    schedule_interval='@daily',
    start_date=datetime(2024, 1, 1),
    catchup=False
) as dag:
    # Your code here

Need a hint?

Use kwargs['execution_date'] and kwargs['logical_date'] inside the function. Pass the function to PythonOperator.

3

Set task to receive Airflow context for dates

Modify the print_dates_task to set provide_context=True so the function receives Airflow context including execution_date and logical_date.

Apache Airflow

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

def print_dates(**kwargs):
    execution_date = kwargs['execution_date']
    logical_date = kwargs['logical_date']
    print(f"Execution date: {execution_date}")
    print(f"Logical date: {logical_date}")

with DAG(
    dag_id='date_example_dag',
    schedule_interval='@daily',
    start_date=datetime(2024, 1, 1),
    catchup=False
) as dag:
    print_dates_task = PythonOperator(
        task_id='print_dates_task',
        python_callable=print_dates,
        # Your code here
    )

Need a hint?

Add provide_context=True to the PythonOperator so the function gets the context.

4

Run the DAG and print the dates

Add a print statement to run the DAG's print_dates_task manually by calling print_dates() with a sample context dictionary containing 'execution_date' and 'logical_date' set to datetime(2024, 1, 2). Print the output of the function.

Apache Airflow

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

def print_dates(**kwargs):
    execution_date = kwargs['execution_date']
    logical_date = kwargs['logical_date']
    print(f"Execution date: {execution_date}")
    print(f"Logical date: {logical_date}")

with DAG(
    dag_id='date_example_dag',
    schedule_interval='@daily',
    start_date=datetime(2024, 1, 1),
    catchup=False
) as dag:
    print_dates_task = PythonOperator(
        task_id='print_dates_task',
        python_callable=print_dates,
        provide_context=True
    )

# Your code here

Need a hint?

Call print_dates() with execution_date=datetime(2024, 1, 2) and logical_date=datetime(2024, 1, 2) to see the printed dates.