BashOperator for shell commands in Apache Airflow - Mini Project: Build & Apply

Using BashOperator to Run Shell Commands in Airflow

📖 Scenario: You are setting up an Airflow workflow to automate simple shell commands. This helps you run tasks like listing files or printing messages automatically.

🎯 Goal: Build an Airflow DAG that uses BashOperator to run shell commands step-by-step.

📋 What You'll Learn

Create a DAG with a specific dag_id

Use BashOperator to run shell commands

Set a schedule interval for the DAG

Print the output of the shell command

💡 Why This Matters

🌍 Real World

Automating shell commands with Airflow helps run routine tasks like backups, file management, or notifications automatically without manual intervention.

💼 Career

Knowing how to use BashOperator is essential for DevOps engineers and data engineers who build automated workflows and pipelines.

1

Create the Airflow DAG skeleton

Create a DAG called bash_command_dag with start_date set to 2024, 1, 1 and schedule_interval set to @daily. Import DAG from airflow and datetime from datetime.

Apache Airflow

# Your code here to import and create the DAG

Need a hint?

Use DAG class with the given parameters to create the DAG.

2

Import BashOperator and define a task

Import BashOperator from airflow.operators.bash. Create a task called list_files using BashOperator that runs the shell command ls -l and assign it to the DAG bash_command_dag.

Apache Airflow

from airflow import DAG
from datetime import datetime

# Import BashOperator
# Your code here

# Create list_files task
# Your code here

Need a hint?

Use BashOperator with task_id, bash_command, and dag parameters.

3

Add a second BashOperator task

Create another task called print_message using BashOperator that runs the shell command echo "Hello from Airflow" and assign it to the DAG bash_command_dag.

Apache Airflow

from airflow import DAG
from datetime import datetime
from airflow.operators.bash import BashOperator

# Create print_message task
# Your code here

Need a hint?

Use BashOperator with the given task_id and bash_command.

4

Set task order and print confirmation

Set the task order so that list_files runs before print_message using the bitshift operator list_files >> print_message. Then print the string "DAG tasks set successfully".

Apache Airflow

from airflow import DAG
from datetime import datetime
from airflow.operators.bash import BashOperator

list_files >> print_message

# Print confirmation message
# Your code here

Need a hint?

Use list_files >> print_message to set order and print() to show the message.