BashOperator for shell commands in Apache Airflow - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is the purpose of the BashOperator in Airflow?

The BashOperator allows you to run shell commands or scripts as tasks within an Airflow workflow (DAG). It helps automate command-line operations.

beginner

How do you specify the shell command to run in a BashOperator?

You use the 'bash_command' parameter to provide the shell command or script you want to execute.

intermediate

What happens if the shell command run by BashOperator fails?

If the shell command exits with a non-zero status, the BashOperator marks the task as failed, which can trigger retries or alerts.

intermediate

Can BashOperator run multi-line shell scripts?

Yes, you can pass multi-line shell scripts using triple quotes or by referencing a script file in the 'bash_command' parameter.

intermediate

How do you pass parameters or variables to a BashOperator command?

You can use Jinja templating in the 'bash_command' to insert variables dynamically, like '{{ ds }}' for the execution date.

Which parameter do you use to define the shell command in BashOperator?

A command

B script_path

C shell_script

D bash_command

The 'bash_command' parameter is used to specify the shell command or script to run.

What does BashOperator do if the shell command exits with a non-zero status?

A Marks the task as failed

B Ignores the error and continues

C Automatically retries without limit

D Restarts the entire DAG

A non-zero exit status causes the task to be marked as failed.

Can you use Jinja templating inside the bash_command parameter?

A Only in Airflow 1.x versions

B Yes, to insert dynamic variables

C Only for PythonOperators

D No, only static commands allowed

Jinja templating is supported to insert dynamic values like execution date.

How can you run a multi-line shell script with BashOperator?

A Use triple quotes or reference a script file

B Only single-line commands are allowed

C Use PythonOperator instead

D Multi-line scripts are not supported

Multi-line scripts can be passed using triple quotes or by calling an external script.

What is a common use case for BashOperator?

A Managing database connections

B Creating Python functions

C Running shell commands or scripts as tasks

D Scheduling DAGs

BashOperator is designed to run shell commands or scripts within Airflow tasks.

Explain how to use BashOperator to run a shell command in an Airflow DAG.

Think about how you tell Airflow what shell command to run and what happens if it fails.

Describe how you can pass dynamic values like execution date into a BashOperator command.

Remember Airflow supports templates inside bash_command.

Practice

1. What is the main purpose of the BashOperator in Airflow?

easy

A. To run shell commands as part of a workflow task

B. To create Python functions for tasks

C. To schedule workflows at specific times

D. To monitor system resources during task execution

Solution

1. Step 1: Understand BashOperator's role

   BashOperator is designed to execute shell commands within Airflow tasks.

2. Step 2: Compare with other options

   Options B, C, and D describe other Airflow features but not BashOperator's main purpose.

3. Final Answer:

   To run shell commands as part of a workflow task -> Option A

4. Quick Check:

   BashOperator runs shell commands = A [OK]

Hint: BashOperator runs shell commands, not Python code [OK]

Common Mistakes:

• Confusing BashOperator with PythonOperator
• Thinking it schedules workflows
• Assuming it monitors system resources

2. Which of the following is the correct way to define a BashOperator task in Airflow?

easy

A. task = BashOperator(name='run_ls', command='ls -l')

B. task = BashOperator(id='run_ls', command='ls -l')

C. task = BashOperator(task_id='run_ls', bash_command='ls -l')

D. task = BashOperator(task='run_ls', shell_command='ls -l')

Solution

1. Step 1: Recall BashOperator parameters

   The correct parameters are task_id and bash_command.

2. Step 2: Match parameters with options

   Only task = BashOperator(task_id='run_ls', bash_command='ls -l') uses task_id and bash_command correctly.

3. Final Answer:

   task = BashOperator(task_id='run_ls', bash_command='ls -l') -> Option C

4. Quick Check:

   Correct parameters = D [OK]

Hint: Use task_id and bash_command keys exactly [OK]

Common Mistakes:

• Using 'id' instead of 'task_id'
• Using 'command' instead of 'bash_command'
• Mixing parameter names

3. Given this BashOperator task:

task = BashOperator(task_id='print_date', bash_command='date')

What will be the output when this task runs?

medium

A. An error because 'date' is not a valid command

B. The current date and time printed in the logs

C. No output because bash_command is empty

D. The string 'date' printed literally

Solution

1. Step 1: Understand the bash_command

   The command 'date' prints the current date and time in the shell.

2. Step 2: Predict task output

   When the task runs, it executes 'date' and logs the current date/time output.

3. Final Answer:

   The current date and time printed in the logs -> Option B

4. Quick Check:

   bash_command='date' outputs date/time = A [OK]

Hint: Commands run as in shell, so 'date' prints current date [OK]

Common Mistakes:

• Thinking 'date' is invalid command
• Expecting no output if command is given
• Confusing command string with literal output

4. What is wrong with this BashOperator task definition?

task = BashOperator(task_id='list_files', bash_command=ls -l)

medium

A. bash_command should be a string, not bare command

B. task_id cannot contain underscores

C. BashOperator requires a Python callable, not a string

D. Missing import statement for BashOperator

Solution

1. Step 1: Check bash_command syntax

   bash_command must be a string, so it should be quoted like 'ls -l'.

2. Step 2: Verify other options

   task_id can have underscores; BashOperator accepts strings; import is assumed outside task definition.

3. Final Answer:

   bash_command should be a string, not bare command -> Option A

4. Quick Check:

   bash_command needs quotes = B [OK]

Hint: Always quote bash_command string [OK]

Common Mistakes:

• Not quoting bash_command string
• Thinking task_id can't have underscores
• Confusing BashOperator with PythonOperator

5. You want to create a BashOperator task that lists all files in /tmp and saves the output to /tmp/file_list.txt. Which is the correct bash_command to use?

hard

A. bash_command='ls /tmp > file_list.txt'

B. bash_command='ls /tmp | /tmp/file_list.txt'

C. bash_command='ls /tmp >> /tmp/file_list.txt'

D. bash_command='ls /tmp > /tmp/file_list.txt'

Solution

1. Step 1: Understand output redirection

   The '>' operator redirects output to a file, overwriting it.

2. Step 2: Check file paths and command correctness

   bash_command='ls /tmp > /tmp/file_list.txt' correctly redirects 'ls /tmp' output to '/tmp/file_list.txt'. bash_command='ls /tmp >> /tmp/file_list.txt' appends instead of overwriting, which may be unwanted. bash_command='ls /tmp | /tmp/file_list.txt' uses pipe incorrectly. bash_command='ls /tmp > file_list.txt' writes to relative path, not /tmp.

3. Final Answer:

   bash_command='ls /tmp > /tmp/file_list.txt' -> Option D

4. Quick Check:

   Use '>' to redirect output to full path file [OK]

Hint: Use '>' for overwrite redirection with full file path [OK]

Common Mistakes:

• Using pipe '|' instead of redirection '>'
• Forgetting to specify full file path
• Using '>>' when overwrite is needed