Manual triggers and parameters in Apache Airflow - Practice Problems & Coding Challenges

Challenge - 5 Problems

💻 Command Output

intermediate

Output of manual trigger with parameters in Airflow CLI

You manually trigger an Airflow DAG named data_pipeline with parameters using the CLI command:

airflow dags trigger data_pipeline --conf '{"run_date": "2024-06-01"}'

What will be the output message shown in the terminal?

A DAG data_pipeline triggered without parameters

B Error: Invalid JSON format in --conf parameter

C Created <DagRun data_pipeline @ 2024-06-01T00:00:00+00:00: manual__2024-06-01T00:00:00+00:00, externally triggered: True>

D No such DAG: data_pipeline

💡 Hint

Check the CLI output format when triggering DAGs with parameters.

When you trigger a DAG manually with the --conf parameter, Airflow creates a DagRun with the given configuration and shows a confirmation message including the DAG name and run time.

🧠 Conceptual

intermediate

Purpose of parameters in manual DAG triggers

Why do we use parameters when manually triggering an Airflow DAG?

A To change the DAG code without redeploying

B To automatically schedule the DAG for future runs

C To disable the DAG temporarily

D To customize the DAG run with specific input values for that execution

💡 Hint

Think about what changes between manual runs.

Parameters let you pass custom data to the DAG run, so tasks can behave differently based on those inputs.

🔀 Workflow

advanced

Correct order to manually trigger an Airflow DAG with parameters via UI

Arrange the steps in the correct order to manually trigger an Airflow DAG named etl_job with parameters using the Airflow web UI.

A 1,2,3,4

B 1,3,2,4

C 2,1,3,4

D 1,2,4,3

💡 Hint

Think about the natural navigation flow in the UI.

You first open the UI, select the DAG, then trigger it and provide parameters in the popup.

❓ Troubleshoot

advanced

Error when triggering DAG with invalid JSON parameters

You try to trigger a DAG manually with this CLI command:

airflow dags trigger my_dag --conf '{run_date: 2024-06-01}'

What error will you most likely see?

A KeyError: run_date not found

B JSONDecodeError: Expecting property name enclosed in double quotes

C TypeError: conf parameter must be a string

D No error, DAG triggered successfully

💡 Hint

Check the JSON format rules for keys and strings.

The JSON keys must be enclosed in double quotes. Missing quotes cause JSONDecodeError.

✅ Best Practice

expert

Best practice for passing sensitive parameters in manual DAG triggers

When manually triggering an Airflow DAG that requires sensitive parameters (like passwords), what is the best practice?

A Use Airflow Variables or Connections to store sensitive data and reference them in the DAG code

B Pass sensitive data directly in the --conf JSON parameter when triggering manually

C Write sensitive data in plain text inside the DAG Python file

D Send sensitive data via email to the Airflow admin before triggering

💡 Hint

Think about security and avoiding exposure of secrets.

Storing secrets in Airflow Variables or Connections keeps them secure and avoids exposing them in logs or CLI commands.

Practice

1. What is the main purpose of manually triggering a DAG in Airflow?

easy

A. To stop a running DAG

B. To schedule a DAG to run automatically at a set time

C. To delete a DAG from Airflow

D. To start a workflow immediately with optional custom parameters

Solution

1. Step 1: Understand manual trigger concept

   Manual triggers allow users to start a workflow whenever they want, not waiting for a schedule.

2. Step 2: Recognize parameter use

   Manual triggers can include parameters to customize the run, making the workflow flexible.

3. Final Answer:

   To start a workflow immediately with optional custom parameters -> Option D

4. Quick Check:

   Manual trigger = immediate start + parameters [OK]

Hint: Manual trigger means start now, not later [OK]

Common Mistakes:

• Confusing manual trigger with scheduled runs
• Thinking manual trigger stops a DAG
• Assuming manual trigger deletes DAG

2. Which of the following is the correct way to access a parameter named 'user_id' passed during a manual trigger inside a DAG task?

easy

A. user_id = dag_run.conf.get('user_id')

B. user_id = dag_run.conf['user_id']

C. user_id = dag_run.params['user_id']

D. user_id = dag_run.get('user_id')

Solution

1. Step 1: Recall parameter access method

   Parameters passed during manual trigger are accessed via dag_run.conf dictionary.

2. Step 2: Use safe access method

   Using dag_run.conf.get('user_id') safely returns the value or None if missing, avoiding errors.

3. Final Answer:

   user_id = dag_run.conf.get('user_id') -> Option A

4. Quick Check:

   Use dag_run.conf.get() to read parameters [OK]

Hint: Use .conf.get() to safely get parameters [OK]

Common Mistakes:

• Using square brackets without .get() causing KeyError
• Using non-existent dag_run.params
• Calling dag_run.get() which is invalid

3. Given this DAG task code snippet inside a PythonOperator:

def greet(**context):
    name = context['dag_run'].conf.get('name', 'Guest')
    print(f"Hello, {name}!")

What will be printed if the DAG is manually triggered with parameters {'name': 'Alice'}?

medium

A. Hello, Guest!

B. Hello, Alice!

C. Hello, None!

D. Error: 'name' not found

Solution

1. Step 1: Understand parameter retrieval

   The code uses dag_run.conf.get('name', 'Guest') which returns 'Alice' if passed, else 'Guest'.

2. Step 2: Apply given parameters

   Since 'name' is provided as 'Alice', the print statement outputs "Hello, Alice!".

3. Final Answer:

   Hello, Alice! -> Option B

4. Quick Check:

   Parameter 'name' passed = 'Alice' printed [OK]

Hint: Default used only if parameter missing [OK]

Common Mistakes:

• Ignoring default value and expecting error
• Assuming parameter is missing
• Confusing print output with variable name

4. You wrote this code inside a DAG task:

def process(**kwargs):
    param = kwargs['dag_run'].conf['param']
    print(param)

When manually triggering the DAG without passing 'param', what happens?

medium

A. Raises KeyError exception

B. Prints empty string

C. Prints None

D. Prints 'param'

Solution

1. Step 1: Analyze parameter access method

   Accessing dag_run.conf['param'] without .get() causes KeyError if 'param' is missing.

2. Step 2: Understand missing parameter effect

   Since 'param' is not passed, the code raises KeyError and does not print anything.

3. Final Answer:

   Raises KeyError exception -> Option A

4. Quick Check:

   Missing key with [] causes KeyError [OK]

Hint: Use .get() to avoid KeyError on missing params [OK]

Common Mistakes:

• Expecting None or empty string instead of error
• Assuming print outputs 'param' string
• Not realizing [] access throws error

5. You want to manually trigger a DAG that processes orders. The DAG expects a parameter 'order_id'. You want to ensure the DAG uses 'unknown' if 'order_id' is missing. Which code snippet inside the DAG task correctly implements this?

hard

A. order_id = dag_run.conf['order_id'] or 'unknown'

B. order_id = dag_run.conf['order_id'] if dag_run.conf else 'unknown'

C. order_id = dag_run.conf.get('order_id', 'unknown')

D. order_id = dag_run.conf.get('order_id') or 'unknown'

Solution

1. Step 1: Understand default parameter handling

   Using dag_run.conf.get('order_id', 'unknown') returns 'order_id' if present, else 'unknown'.

2. Step 2: Compare options for correctness

   A fails with KeyError if missing. B checks if conf exists (empty {} is falsy), but if other parameters present conf is truthy causing KeyError. C works for missing but overrides falsy values like empty string. D is clean, safe, and handles all cases correctly.

3. Final Answer:

   order_id = dag_run.conf.get('order_id', 'unknown') -> Option C

4. Quick Check:

   Use .get(key, default) for safe defaults [OK]

Hint: Use .get(key, default) to set fallback values [OK]

Common Mistakes:

• Using [] without checking key existence
• Using or without default in .get()
• Overcomplicating with if-else instead of .get()