BashOperator for shell commands in Apache Airflow - Step-by-Step Execution

Process Flow - BashOperator for shell commands

Define BashOperator

↓

Set shell command

↓

Schedule DAG run

↓

Airflow executes BashOperator

↓

Shell command runs in OS

↓

Capture output and status

↓

Task success or failure

The BashOperator runs a shell command inside an Airflow task, capturing its output and status to determine success.

Execution Sample

Apache Airflow

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

dag = DAG('example_bash', start_date=datetime(2024, 1, 1))

bash_task = BashOperator(
    task_id='print_date',
    bash_command='date',
    dag=dag
)

This code defines a DAG with a BashOperator task that runs the 'date' shell command.

Process Table

Step	Action	Command Executed	Output	Task Status
1	DAG starts	N/A	N/A	Waiting
2	BashOperator triggers	date	N/A	Running
3	Shell runs 'date'	date	Fri Apr 26 12:00:00 UTC 2024	Running
4	Command completes	date	Fri Apr 26 12:00:00 UTC 2024	Success

💡 Command 'date' completed successfully, task marked as Success.

Status Tracker

Variable	Start	After Step 2	After Step 3	Final
bash_command	None	'date'	'date'	'date'
task_status	None	Running	Running	Success
command_output	None	None	'Fri Apr 26 12:00:00 UTC 2024'	'Fri Apr 26 12:00:00 UTC 2024'

Key Moments - 2 Insights

Why does the task status change from Running to Success only after the command finishes?

Because Airflow waits for the shell command to complete before marking the task as Success, as shown in steps 3 and 4 of the execution table.

What happens if the shell command fails?

If the command fails, Airflow marks the task as Failed instead of Success, stopping further downstream tasks.

Visual Quiz - 3 Questions

Look at the execution table, what is the task status at Step 3?

A Waiting

B Running

C Success

D Failed

💡 Hint

Check the 'Task Status' column at Step 3 in the execution table.

At which step does the shell command output become available?

A Step 1

B Step 2

C Step 3

D Step 4

💡 Hint

Look at the 'Output' column in the execution table to see when the command output appears.

If the bash_command was changed to 'echo Hello', how would the output in Step 4 change?

A 'Hello'

B 'Fri Apr 26 12:00:00 UTC 2024'

C No output

D Error message

💡 Hint

Refer to the 'command_output' variable in variable_tracker and think about what 'echo Hello' prints.

Concept Snapshot

BashOperator runs shell commands in Airflow tasks.
Define with bash_command parameter.
Runs command on task execution.
Captures output and exit status.
Success if command exits 0, else fails.

Full Transcript

The BashOperator in Airflow lets you run shell commands as tasks inside a workflow. You define it by specifying the bash_command you want to run. When the DAG runs, Airflow triggers the BashOperator, which runs the shell command on the operating system. The command's output and exit status are captured. If the command finishes successfully (exit code 0), the task is marked as Success. If it fails, the task fails. This process allows you to integrate shell commands easily into your Airflow workflows.

Practice

1. What is the main purpose of the BashOperator in Airflow?

easy

A. To run shell commands as part of a workflow task

B. To create Python functions for tasks

C. To schedule workflows at specific times

D. To monitor system resources during task execution

Solution

1. Step 1: Understand BashOperator's role

   BashOperator is designed to execute shell commands within Airflow tasks.

2. Step 2: Compare with other options

   Options B, C, and D describe other Airflow features but not BashOperator's main purpose.

3. Final Answer:

   To run shell commands as part of a workflow task -> Option A

4. Quick Check:

   BashOperator runs shell commands = A [OK]

Hint: BashOperator runs shell commands, not Python code [OK]

Common Mistakes:

• Confusing BashOperator with PythonOperator
• Thinking it schedules workflows
• Assuming it monitors system resources

2. Which of the following is the correct way to define a BashOperator task in Airflow?

easy

A. task = BashOperator(name='run_ls', command='ls -l')

B. task = BashOperator(id='run_ls', command='ls -l')

C. task = BashOperator(task_id='run_ls', bash_command='ls -l')

D. task = BashOperator(task='run_ls', shell_command='ls -l')

Solution

1. Step 1: Recall BashOperator parameters

   The correct parameters are task_id and bash_command.

2. Step 2: Match parameters with options

   Only task = BashOperator(task_id='run_ls', bash_command='ls -l') uses task_id and bash_command correctly.

3. Final Answer:

   task = BashOperator(task_id='run_ls', bash_command='ls -l') -> Option C

4. Quick Check:

   Correct parameters = D [OK]

Hint: Use task_id and bash_command keys exactly [OK]

Common Mistakes:

• Using 'id' instead of 'task_id'
• Using 'command' instead of 'bash_command'
• Mixing parameter names

3. Given this BashOperator task:

task = BashOperator(task_id='print_date', bash_command='date')

What will be the output when this task runs?

medium

A. An error because 'date' is not a valid command

B. The current date and time printed in the logs

C. No output because bash_command is empty

D. The string 'date' printed literally

Solution

1. Step 1: Understand the bash_command

   The command 'date' prints the current date and time in the shell.

2. Step 2: Predict task output

   When the task runs, it executes 'date' and logs the current date/time output.

3. Final Answer:

   The current date and time printed in the logs -> Option B

4. Quick Check:

   bash_command='date' outputs date/time = A [OK]

Hint: Commands run as in shell, so 'date' prints current date [OK]

Common Mistakes:

• Thinking 'date' is invalid command
• Expecting no output if command is given
• Confusing command string with literal output

4. What is wrong with this BashOperator task definition?

task = BashOperator(task_id='list_files', bash_command=ls -l)

medium

A. bash_command should be a string, not bare command

B. task_id cannot contain underscores

C. BashOperator requires a Python callable, not a string

D. Missing import statement for BashOperator

Solution

1. Step 1: Check bash_command syntax

   bash_command must be a string, so it should be quoted like 'ls -l'.

2. Step 2: Verify other options

   task_id can have underscores; BashOperator accepts strings; import is assumed outside task definition.

3. Final Answer:

   bash_command should be a string, not bare command -> Option A

4. Quick Check:

   bash_command needs quotes = B [OK]

Hint: Always quote bash_command string [OK]

Common Mistakes:

• Not quoting bash_command string
• Thinking task_id can't have underscores
• Confusing BashOperator with PythonOperator

5. You want to create a BashOperator task that lists all files in /tmp and saves the output to /tmp/file_list.txt. Which is the correct bash_command to use?

hard

A. bash_command='ls /tmp > file_list.txt'

B. bash_command='ls /tmp | /tmp/file_list.txt'

C. bash_command='ls /tmp >> /tmp/file_list.txt'

D. bash_command='ls /tmp > /tmp/file_list.txt'

Solution

1. Step 1: Understand output redirection

   The '>' operator redirects output to a file, overwriting it.

2. Step 2: Check file paths and command correctness

   bash_command='ls /tmp > /tmp/file_list.txt' correctly redirects 'ls /tmp' output to '/tmp/file_list.txt'. bash_command='ls /tmp >> /tmp/file_list.txt' appends instead of overwriting, which may be unwanted. bash_command='ls /tmp | /tmp/file_list.txt' uses pipe incorrectly. bash_command='ls /tmp > file_list.txt' writes to relative path, not /tmp.

3. Final Answer:

   bash_command='ls /tmp > /tmp/file_list.txt' -> Option D

4. Quick Check:

   Use '>' to redirect output to full path file [OK]

Hint: Use '>' for overwrite redirection with full file path [OK]

Common Mistakes:

• Using pipe '|' instead of redirection '>'
• Forgetting to specify full file path
• Using '>>' when overwrite is needed