BashOperator for shell commands in Apache Airflow - Commands & Configuration

Introduction

Sometimes you need to run simple shell commands or scripts as part of your workflow. BashOperator lets you do this easily inside Airflow, so you can automate tasks like file operations or running scripts without extra setup.

When you want to run a shell script to move or copy files as part of a data pipeline.

When you need to execute a command-line tool that does not have a dedicated Airflow operator.

When you want to run quick Linux commands like listing files or checking disk space during a workflow.

When you want to chain shell commands in a sequence inside your Airflow DAG.

When you want to automate running a backup script on a schedule.

Config File - bash_operator_dag.py

bash_operator_dag.py

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

default_args = {
    'start_date': datetime(2024, 1, 1),
}

dag = DAG(
    'example_bash_operator',
    default_args=default_args,
    schedule_interval='@daily',
    catchup=False
)

run_ls = BashOperator(
    task_id='list_files',
    bash_command='ls -l /tmp',
    dag=dag
)

run_echo = BashOperator(
    task_id='print_message',
    bash_command='echo "Hello from Airflow BashOperator"',
    dag=dag
)

run_ls >> run_echo

This file defines an Airflow DAG named example_bash_operator that runs daily starting from January 1, 2024.

It has two tasks using BashOperator:

• list_files: runs ls -l /tmp to list files in the /tmp directory.
• print_message: prints a simple message to the logs.

The tasks run in order: first list files, then print the message.

Commands

This command lists all available DAGs in Airflow to confirm your DAG is recognized.

Terminal

airflow dags list

Expected Output

example_bash_operator

This command manually triggers the DAG to run the tasks immediately.

Terminal

airflow dags trigger example_bash_operator

Expected Output

Created <DagRun example_bash_operator @ 2024-06-01T12:00:00+00:00: manual__2024-06-01T12:00:00+00:00, externally triggered: True>

This command lists all tasks in the DAG to verify the tasks are defined correctly.

Terminal

airflow tasks list example_bash_operator

Expected Output

list_files

print_message

This command runs the 'list_files' task for the given date without scheduling, useful for testing the command output.

Terminal

airflow tasks test example_bash_operator list_files 2024-06-01

Expected Output

[2024-06-01 12:00:00,000] {bash.py:123} INFO - Running command: ls -l /tmp

[2024-06-01 12:00:00,100] {bash.py:130} INFO - total 0

-rw-r--r-- 1 user user 0 Jun  1 12:00 example.txt

[2024-06-01 12:00:00,200] {bash.py:140} INFO - Command exited with return code 0

Key Concept

If you remember nothing else from this pattern, remember: BashOperator lets you run any shell command inside Airflow tasks simply and directly.

Common Mistakes

Using BashOperator with complex multi-line scripts without proper quoting or escaping.

The shell command may fail or behave unexpectedly because of incorrect syntax or line breaks.

Use triple quotes or write the script in a separate file and call it from BashOperator.

Not setting the correct start_date or schedule_interval in the DAG, causing the DAG not to run.

Airflow requires a valid start date and schedule to trigger DAG runs automatically.

Always set start_date in the past and a valid schedule_interval when defining the DAG.

Assuming BashOperator runs commands on your local machine instead of the Airflow worker environment.

Commands run where Airflow workers execute, which may differ from your local setup.

Ensure the commands and paths exist on the Airflow worker nodes.

Summary

Define a DAG with BashOperator tasks to run shell commands inside Airflow.

Use airflow CLI commands to list, trigger, and test DAG tasks.

BashOperator runs commands on Airflow workers, so commands must be valid there.

Practice

1. What is the main purpose of the BashOperator in Airflow?

easy

A. To run shell commands as part of a workflow task

B. To create Python functions for tasks

C. To schedule workflows at specific times

D. To monitor system resources during task execution

Solution

1. Step 1: Understand BashOperator's role

   BashOperator is designed to execute shell commands within Airflow tasks.

2. Step 2: Compare with other options

   Options B, C, and D describe other Airflow features but not BashOperator's main purpose.

3. Final Answer:

   To run shell commands as part of a workflow task -> Option A

4. Quick Check:

   BashOperator runs shell commands = A [OK]

Hint: BashOperator runs shell commands, not Python code [OK]

Common Mistakes:

• Confusing BashOperator with PythonOperator
• Thinking it schedules workflows
• Assuming it monitors system resources

2. Which of the following is the correct way to define a BashOperator task in Airflow?

easy

A. task = BashOperator(name='run_ls', command='ls -l')

B. task = BashOperator(id='run_ls', command='ls -l')

C. task = BashOperator(task_id='run_ls', bash_command='ls -l')

D. task = BashOperator(task='run_ls', shell_command='ls -l')

Solution

1. Step 1: Recall BashOperator parameters

   The correct parameters are task_id and bash_command.

2. Step 2: Match parameters with options

   Only task = BashOperator(task_id='run_ls', bash_command='ls -l') uses task_id and bash_command correctly.

3. Final Answer:

   task = BashOperator(task_id='run_ls', bash_command='ls -l') -> Option C

4. Quick Check:

   Correct parameters = D [OK]

Hint: Use task_id and bash_command keys exactly [OK]

Common Mistakes:

• Using 'id' instead of 'task_id'
• Using 'command' instead of 'bash_command'
• Mixing parameter names

3. Given this BashOperator task:

task = BashOperator(task_id='print_date', bash_command='date')

What will be the output when this task runs?

medium

A. An error because 'date' is not a valid command

B. The current date and time printed in the logs

C. No output because bash_command is empty

D. The string 'date' printed literally

Solution

1. Step 1: Understand the bash_command

   The command 'date' prints the current date and time in the shell.

2. Step 2: Predict task output

   When the task runs, it executes 'date' and logs the current date/time output.

3. Final Answer:

   The current date and time printed in the logs -> Option B

4. Quick Check:

   bash_command='date' outputs date/time = A [OK]

Hint: Commands run as in shell, so 'date' prints current date [OK]

Common Mistakes:

• Thinking 'date' is invalid command
• Expecting no output if command is given
• Confusing command string with literal output

4. What is wrong with this BashOperator task definition?

task = BashOperator(task_id='list_files', bash_command=ls -l)

medium

A. bash_command should be a string, not bare command

B. task_id cannot contain underscores

C. BashOperator requires a Python callable, not a string

D. Missing import statement for BashOperator

Solution

1. Step 1: Check bash_command syntax

   bash_command must be a string, so it should be quoted like 'ls -l'.

2. Step 2: Verify other options

   task_id can have underscores; BashOperator accepts strings; import is assumed outside task definition.

3. Final Answer:

   bash_command should be a string, not bare command -> Option A

4. Quick Check:

   bash_command needs quotes = B [OK]

Hint: Always quote bash_command string [OK]

Common Mistakes:

• Not quoting bash_command string
• Thinking task_id can't have underscores
• Confusing BashOperator with PythonOperator

5. You want to create a BashOperator task that lists all files in /tmp and saves the output to /tmp/file_list.txt. Which is the correct bash_command to use?

hard

A. bash_command='ls /tmp > file_list.txt'

B. bash_command='ls /tmp | /tmp/file_list.txt'

C. bash_command='ls /tmp >> /tmp/file_list.txt'

D. bash_command='ls /tmp > /tmp/file_list.txt'

Solution

1. Step 1: Understand output redirection

   The '>' operator redirects output to a file, overwriting it.

2. Step 2: Check file paths and command correctness

   bash_command='ls /tmp > /tmp/file_list.txt' correctly redirects 'ls /tmp' output to '/tmp/file_list.txt'. bash_command='ls /tmp >> /tmp/file_list.txt' appends instead of overwriting, which may be unwanted. bash_command='ls /tmp | /tmp/file_list.txt' uses pipe incorrectly. bash_command='ls /tmp > file_list.txt' writes to relative path, not /tmp.

3. Final Answer:

   bash_command='ls /tmp > /tmp/file_list.txt' -> Option D

4. Quick Check:

   Use '>' to redirect output to full path file [OK]

Hint: Use '>' for overwrite redirection with full file path [OK]

Common Mistakes:

• Using pipe '|' instead of redirection '>'
• Forgetting to specify full file path
• Using '>>' when overwrite is needed