BashOperator for shell commands in Apache Airflow - Practice Problems & Coding Challenges

Challenge - 5 Problems

💻 Command Output

intermediate

Output of BashOperator with simple echo command

What will be the output logged by the BashOperator when running this task?

Apache Airflow

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

dag = DAG('test_dag', start_date=datetime(2024, 1, 1))

task = BashOperator(
    task_id='print_hello',
    bash_command='echo Hello Airflow',
    dag=dag
)

A Hello Airflow

B echo Hello Airflow

C bash_command executed

D Task failed with error

💡 Hint

Think about what the echo command does in a shell.

The BashOperator runs the bash_command string in a shell. The command 'echo Hello Airflow' prints 'Hello Airflow' to the output.

❓ Configuration

intermediate

Correct way to pass multiple commands in BashOperator

Which option correctly runs two shell commands sequentially in a single BashOperator?

A bash_command='echo Start && echo End'

B bash_command=['echo Start', 'echo End']

C bash_command='echo Start | echo End'

D bash_command='echo Start; echo End'

💡 Hint

In shell, commands separated by semicolon run sequentially.

Using a semicolon ';' runs commands one after another. '&&' runs second only if first succeeds. The pipe '|' sends output of first to input of second, which is not correct here. Passing a list is invalid for bash_command.

❓ Troubleshoot

advanced

Error caused by incorrect bash_command syntax

What error will occur when running this BashOperator? bash_command='echo Hello &&& echo World'

A BashOperator task fails with shell syntax error

B SyntaxError in Python

C Task runs successfully printing 'Hello &&& echo World'

D Airflow DAG fails to parse

💡 Hint

Consider how the shell interprets '&&&'.

The shell does not recognize '&&&' as a valid operator, so the BashOperator task will fail with a shell syntax error during execution.

🔀 Workflow

advanced

BashOperator task dependencies in Airflow DAG

Given two BashOperator tasks, task1 and task2, how do you set task2 to run only after task1 completes successfully?

Apache Airflow

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

dag = DAG('dependency_dag', start_date=datetime(2024, 1, 1))

task1 = BashOperator(task_id='task1', bash_command='echo Task 1', dag=dag)
task2 = BashOperator(task_id='task2', bash_command='echo Task 2', dag=dag)

A task2 >> task1

B Both C and D

C task1.set_downstream(task2)

D task1 >> task2

💡 Hint

Airflow uses bitshift operators or methods to set dependencies.

Both 'task1 >> task2' and 'task1.set_downstream(task2)' set task2 to run after task1. 'task2 >> task1' reverses the order.

✅ Best Practice

expert

Best practice for passing dynamic parameters to BashOperator

Which method is best to pass a dynamic date string to a BashOperator command in Airflow?

A Hardcode the date string in bash_command

B Concatenate date string in Python before passing to bash_command

C Use Jinja templating with '{{ ds }}' in bash_command

D Use environment variables set outside Airflow

💡 Hint

Airflow supports templating in operators for dynamic values.

Using Jinja templating like '{{ ds }}' allows Airflow to inject the execution date dynamically at runtime, which is best practice for dynamic parameters.

Practice

1. What is the main purpose of the BashOperator in Airflow?

easy

A. To run shell commands as part of a workflow task

B. To create Python functions for tasks

C. To schedule workflows at specific times

D. To monitor system resources during task execution

Solution

1. Step 1: Understand BashOperator's role

   BashOperator is designed to execute shell commands within Airflow tasks.

2. Step 2: Compare with other options

   Options B, C, and D describe other Airflow features but not BashOperator's main purpose.

3. Final Answer:

   To run shell commands as part of a workflow task -> Option A

4. Quick Check:

   BashOperator runs shell commands = A [OK]

Hint: BashOperator runs shell commands, not Python code [OK]

Common Mistakes:

• Confusing BashOperator with PythonOperator
• Thinking it schedules workflows
• Assuming it monitors system resources

2. Which of the following is the correct way to define a BashOperator task in Airflow?

easy

A. task = BashOperator(name='run_ls', command='ls -l')

B. task = BashOperator(id='run_ls', command='ls -l')

C. task = BashOperator(task_id='run_ls', bash_command='ls -l')

D. task = BashOperator(task='run_ls', shell_command='ls -l')

Solution

1. Step 1: Recall BashOperator parameters

   The correct parameters are task_id and bash_command.

2. Step 2: Match parameters with options

   Only task = BashOperator(task_id='run_ls', bash_command='ls -l') uses task_id and bash_command correctly.

3. Final Answer:

   task = BashOperator(task_id='run_ls', bash_command='ls -l') -> Option C

4. Quick Check:

   Correct parameters = D [OK]

Hint: Use task_id and bash_command keys exactly [OK]

Common Mistakes:

• Using 'id' instead of 'task_id'
• Using 'command' instead of 'bash_command'
• Mixing parameter names

3. Given this BashOperator task:

task = BashOperator(task_id='print_date', bash_command='date')

What will be the output when this task runs?

medium

A. An error because 'date' is not a valid command

B. The current date and time printed in the logs

C. No output because bash_command is empty

D. The string 'date' printed literally

Solution

1. Step 1: Understand the bash_command

   The command 'date' prints the current date and time in the shell.

2. Step 2: Predict task output

   When the task runs, it executes 'date' and logs the current date/time output.

3. Final Answer:

   The current date and time printed in the logs -> Option B

4. Quick Check:

   bash_command='date' outputs date/time = A [OK]

Hint: Commands run as in shell, so 'date' prints current date [OK]

Common Mistakes:

• Thinking 'date' is invalid command
• Expecting no output if command is given
• Confusing command string with literal output

4. What is wrong with this BashOperator task definition?

task = BashOperator(task_id='list_files', bash_command=ls -l)

medium

A. bash_command should be a string, not bare command

B. task_id cannot contain underscores

C. BashOperator requires a Python callable, not a string

D. Missing import statement for BashOperator

Solution

1. Step 1: Check bash_command syntax

   bash_command must be a string, so it should be quoted like 'ls -l'.

2. Step 2: Verify other options

   task_id can have underscores; BashOperator accepts strings; import is assumed outside task definition.

3. Final Answer:

   bash_command should be a string, not bare command -> Option A

4. Quick Check:

   bash_command needs quotes = B [OK]

Hint: Always quote bash_command string [OK]

Common Mistakes:

• Not quoting bash_command string
• Thinking task_id can't have underscores
• Confusing BashOperator with PythonOperator

5. You want to create a BashOperator task that lists all files in /tmp and saves the output to /tmp/file_list.txt. Which is the correct bash_command to use?

hard

A. bash_command='ls /tmp > file_list.txt'

B. bash_command='ls /tmp | /tmp/file_list.txt'

C. bash_command='ls /tmp >> /tmp/file_list.txt'

D. bash_command='ls /tmp > /tmp/file_list.txt'

Solution

1. Step 1: Understand output redirection

   The '>' operator redirects output to a file, overwriting it.

2. Step 2: Check file paths and command correctness

   bash_command='ls /tmp > /tmp/file_list.txt' correctly redirects 'ls /tmp' output to '/tmp/file_list.txt'. bash_command='ls /tmp >> /tmp/file_list.txt' appends instead of overwriting, which may be unwanted. bash_command='ls /tmp | /tmp/file_list.txt' uses pipe incorrectly. bash_command='ls /tmp > file_list.txt' writes to relative path, not /tmp.

3. Final Answer:

   bash_command='ls /tmp > /tmp/file_list.txt' -> Option D

4. Quick Check:

   Use '>' to redirect output to full path file [OK]

Hint: Use '>' for overwrite redirection with full file path [OK]

Common Mistakes:

• Using pipe '|' instead of redirection '>'
• Forgetting to specify full file path
• Using '>>' when overwrite is needed