BashOperator for shell commands in Apache Airflow - Step-by-Step Execution

Process Flow - BashOperator for shell commands

Define BashOperator

↓

Set shell command

↓

Schedule DAG run

↓

Airflow executes BashOperator

↓

Shell command runs in OS

↓

Capture output and status

↓

Task success or failure

The BashOperator runs a shell command inside an Airflow task, capturing its output and status to determine success.

Execution Sample

Apache Airflow

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

dag = DAG('example_bash', start_date=datetime(2024, 1, 1))

bash_task = BashOperator(
    task_id='print_date',
    bash_command='date',
    dag=dag
)

This code defines a DAG with a BashOperator task that runs the 'date' shell command.

Process Table

Step	Action	Command Executed	Output	Task Status
1	DAG starts	N/A	N/A	Waiting
2	BashOperator triggers	date	N/A	Running
3	Shell runs 'date'	date	Fri Apr 26 12:00:00 UTC 2024	Running
4	Command completes	date	Fri Apr 26 12:00:00 UTC 2024	Success

💡 Command 'date' completed successfully, task marked as Success.

Status Tracker

Variable	Start	After Step 2	After Step 3	Final
bash_command	None	'date'	'date'	'date'
task_status	None	Running	Running	Success
command_output	None	None	'Fri Apr 26 12:00:00 UTC 2024'	'Fri Apr 26 12:00:00 UTC 2024'

Key Moments - 2 Insights

Why does the task status change from Running to Success only after the command finishes?

Because Airflow waits for the shell command to complete before marking the task as Success, as shown in steps 3 and 4 of the execution table.

What happens if the shell command fails?

If the command fails, Airflow marks the task as Failed instead of Success, stopping further downstream tasks.

Visual Quiz - 3 Questions

Look at the execution table, what is the task status at Step 3?

A Waiting

B Running

C Success

D Failed

💡 Hint

Check the 'Task Status' column at Step 3 in the execution table.

At which step does the shell command output become available?

A Step 1

B Step 2

C Step 3

D Step 4

💡 Hint

Look at the 'Output' column in the execution table to see when the command output appears.

If the bash_command was changed to 'echo Hello', how would the output in Step 4 change?

A 'Hello'

B 'Fri Apr 26 12:00:00 UTC 2024'

C No output

D Error message

💡 Hint

Refer to the 'command_output' variable in variable_tracker and think about what 'echo Hello' prints.

Concept Snapshot

BashOperator runs shell commands in Airflow tasks.
Define with bash_command parameter.
Runs command on task execution.
Captures output and exit status.
Success if command exits 0, else fails.

Full Transcript

The BashOperator in Airflow lets you run shell commands as tasks inside a workflow. You define it by specifying the bash_command you want to run. When the DAG runs, Airflow triggers the BashOperator, which runs the shell command on the operating system. The command's output and exit status are captured. If the command finishes successfully (exit code 0), the task is marked as Success. If it fails, the task fails. This process allows you to integrate shell commands easily into your Airflow workflows.