BashOperator for shell commands in Apache Airflow - Mini Project: Build & Apply

Using BashOperator to Run Shell Commands in Airflow

📖 Scenario: You are setting up an Airflow workflow to automate simple shell commands. This helps you run tasks like listing files or printing messages automatically.

🎯 Goal: Build an Airflow DAG that uses BashOperator to run shell commands step-by-step.

📋 What You'll Learn

Create a DAG with a specific dag_id

Use BashOperator to run shell commands

Set a schedule interval for the DAG

Print the output of the shell command

💡 Why This Matters

🌍 Real World

Automating shell commands with Airflow helps run routine tasks like backups, file management, or notifications automatically without manual intervention.

💼 Career

Knowing how to use BashOperator is essential for DevOps engineers and data engineers who build automated workflows and pipelines.

1

Create the Airflow DAG skeleton

Create a DAG called bash_command_dag with start_date set to 2024, 1, 1 and schedule_interval set to @daily. Import DAG from airflow and datetime from datetime.

Apache Airflow

# Your code here to import and create the DAG

Hint

Use DAG class with the given parameters to create the DAG.

2

Import BashOperator and define a task

Import BashOperator from airflow.operators.bash. Create a task called list_files using BashOperator that runs the shell command ls -l and assign it to the DAG bash_command_dag.

Apache Airflow

from airflow import DAG
from datetime import datetime

# Import BashOperator
# Your code here

# Create list_files task
# Your code here

Hint

Use BashOperator with task_id, bash_command, and dag parameters.

3

Add a second BashOperator task

Create another task called print_message using BashOperator that runs the shell command echo "Hello from Airflow" and assign it to the DAG bash_command_dag.

Apache Airflow

from airflow import DAG
from datetime import datetime
from airflow.operators.bash import BashOperator

# Create print_message task
# Your code here

Hint

Use BashOperator with the given task_id and bash_command.

4

Set task order and print confirmation

Set the task order so that list_files runs before print_message using the bitshift operator list_files >> print_message. Then print the string "DAG tasks set successfully".

Apache Airflow

from airflow import DAG
from datetime import datetime
from airflow.operators.bash import BashOperator

list_files >> print_message

# Print confirmation message
# Your code here

Hint

Use list_files >> print_message to set order and print() to show the message.

Practice

1. What is the main purpose of the BashOperator in Airflow?

easy

A. To run shell commands as part of a workflow task

B. To create Python functions for tasks

C. To schedule workflows at specific times

D. To monitor system resources during task execution

Solution

1. Step 1: Understand BashOperator's role

   BashOperator is designed to execute shell commands within Airflow tasks.

2. Step 2: Compare with other options

   Options B, C, and D describe other Airflow features but not BashOperator's main purpose.

3. Final Answer:

   To run shell commands as part of a workflow task -> Option A

4. Quick Check:

   BashOperator runs shell commands = A [OK]

Hint: BashOperator runs shell commands, not Python code [OK]

Common Mistakes:

• Confusing BashOperator with PythonOperator
• Thinking it schedules workflows
• Assuming it monitors system resources

2. Which of the following is the correct way to define a BashOperator task in Airflow?

easy

A. task = BashOperator(name='run_ls', command='ls -l')

B. task = BashOperator(id='run_ls', command='ls -l')

C. task = BashOperator(task_id='run_ls', bash_command='ls -l')

D. task = BashOperator(task='run_ls', shell_command='ls -l')

Solution

1. Step 1: Recall BashOperator parameters

   The correct parameters are task_id and bash_command.

2. Step 2: Match parameters with options

   Only task = BashOperator(task_id='run_ls', bash_command='ls -l') uses task_id and bash_command correctly.

3. Final Answer:

   task = BashOperator(task_id='run_ls', bash_command='ls -l') -> Option C

4. Quick Check:

   Correct parameters = D [OK]

Hint: Use task_id and bash_command keys exactly [OK]

Common Mistakes:

• Using 'id' instead of 'task_id'
• Using 'command' instead of 'bash_command'
• Mixing parameter names

3. Given this BashOperator task:

task = BashOperator(task_id='print_date', bash_command='date')

What will be the output when this task runs?

medium

A. An error because 'date' is not a valid command

B. The current date and time printed in the logs

C. No output because bash_command is empty

D. The string 'date' printed literally

Solution

1. Step 1: Understand the bash_command

   The command 'date' prints the current date and time in the shell.

2. Step 2: Predict task output

   When the task runs, it executes 'date' and logs the current date/time output.

3. Final Answer:

   The current date and time printed in the logs -> Option B

4. Quick Check:

   bash_command='date' outputs date/time = A [OK]

Hint: Commands run as in shell, so 'date' prints current date [OK]

Common Mistakes:

• Thinking 'date' is invalid command
• Expecting no output if command is given
• Confusing command string with literal output

4. What is wrong with this BashOperator task definition?

task = BashOperator(task_id='list_files', bash_command=ls -l)

medium

A. bash_command should be a string, not bare command

B. task_id cannot contain underscores

C. BashOperator requires a Python callable, not a string

D. Missing import statement for BashOperator

Solution

1. Step 1: Check bash_command syntax

   bash_command must be a string, so it should be quoted like 'ls -l'.

2. Step 2: Verify other options

   task_id can have underscores; BashOperator accepts strings; import is assumed outside task definition.

3. Final Answer:

   bash_command should be a string, not bare command -> Option A

4. Quick Check:

   bash_command needs quotes = B [OK]

Hint: Always quote bash_command string [OK]

Common Mistakes:

• Not quoting bash_command string
• Thinking task_id can't have underscores
• Confusing BashOperator with PythonOperator

5. You want to create a BashOperator task that lists all files in /tmp and saves the output to /tmp/file_list.txt. Which is the correct bash_command to use?

hard

A. bash_command='ls /tmp > file_list.txt'

B. bash_command='ls /tmp | /tmp/file_list.txt'

C. bash_command='ls /tmp >> /tmp/file_list.txt'

D. bash_command='ls /tmp > /tmp/file_list.txt'

Solution

1. Step 1: Understand output redirection

   The '>' operator redirects output to a file, overwriting it.

2. Step 2: Check file paths and command correctness

   bash_command='ls /tmp > /tmp/file_list.txt' correctly redirects 'ls /tmp' output to '/tmp/file_list.txt'. bash_command='ls /tmp >> /tmp/file_list.txt' appends instead of overwriting, which may be unwanted. bash_command='ls /tmp | /tmp/file_list.txt' uses pipe incorrectly. bash_command='ls /tmp > file_list.txt' writes to relative path, not /tmp.

3. Final Answer:

   bash_command='ls /tmp > /tmp/file_list.txt' -> Option D

4. Quick Check:

   Use '>' to redirect output to full path file [OK]

Hint: Use '>' for overwrite redirection with full file path [OK]

Common Mistakes:

• Using pipe '|' instead of redirection '>'
• Forgetting to specify full file path
• Using '>>' when overwrite is needed