PythonOperator for custom logic in Apache Airflow - Mini Project: Build & Apply

PythonOperator for custom logic

📖 Scenario: You are working with Apache Airflow to automate tasks. You want to create a simple workflow that runs a custom Python function using the PythonOperator.This project will guide you step-by-step to create a DAG with a PythonOperator that runs a function printing a greeting message.

🎯 Goal: Build an Airflow DAG that uses PythonOperator to run a custom Python function which prints a greeting message.

📋 What You'll Learn

Create a Python function called greet that prints 'Hello from Airflow!'

Create a DAG named greeting_dag with start_date set to January 1, 2024

Add a PythonOperator task named greet_task that calls the greet function

Set the task greet_task as the only task in the DAG

Print the task instance context in the greet function

💡 Why This Matters

🌍 Real World

Automating workflows in data pipelines, ETL jobs, or any scheduled tasks using Apache Airflow.

💼 Career

Understanding how to use PythonOperator is essential for DevOps engineers and data engineers working with Airflow to automate and manage workflows.

1

Create the greet function

Create a Python function called greet that takes a single argument **kwargs and prints the message 'Hello from Airflow!'.

Apache Airflow

# Your code here

Need a hint?

Define a function named greet with **kwargs to accept context.

2

Create the Airflow DAG

Import DAG and datetime. Create a DAG named greeting_dag with start_date set to datetime(2024, 1, 1) and schedule_interval set to @daily.

Apache Airflow

def greet(**kwargs):
    print('Hello from Airflow!')

# Your code here

Need a hint?

Use DAG and datetime to create the DAG with the given name and schedule.

3

Add the PythonOperator task

Import PythonOperator from airflow.operators.python. Create a task named greet_task using PythonOperator that calls the greet function and belongs to the greeting_dag DAG.

Apache Airflow

from airflow import DAG
from datetime import datetime

def greet(**kwargs):
    print('Hello from Airflow!')

from airflow.operators.python import PythonOperator

# Your code here

Need a hint?

Create a PythonOperator with the specified task_id, python_callable, and dag.

4

Print the task instance context in greet and run the task

Modify the greet function to print the task_instance from kwargs using kwargs['ti']. Then add a line to print 'Task greet_task is set up in DAG greeting_dag'.

Apache Airflow

from airflow import DAG
from datetime import datetime
from airflow.operators.python import PythonOperator

def greet(**kwargs):
    print('Hello from Airflow!')

# Your code here

Need a hint?

Use print(f"Task instance: {kwargs['ti']}") inside greet and print the confirmation message after task creation.