Manual triggers and parameters in Apache Airflow - Deep Dive

Overview - Manual triggers and parameters

What is it?

Manual triggers in Airflow allow you to start a workflow on demand instead of waiting for a scheduled time. Parameters are extra pieces of information you can pass when triggering a workflow manually to customize its behavior. This helps you run workflows flexibly with different inputs without changing the code. It is like pressing a button to start a process and telling it exactly what to do each time.

Why it matters

Without manual triggers and parameters, workflows would only run on fixed schedules, limiting flexibility. You couldn't easily test workflows with different inputs or respond quickly to urgent needs. Manual triggers let you control when and how workflows run, making your automation smarter and more responsive. This saves time and reduces errors in real-world operations.

Where it fits

Before learning manual triggers, you should understand basic Airflow concepts like DAGs, tasks, and scheduling. After mastering manual triggers and parameters, you can explore advanced topics like dynamic workflows, sensors, and event-driven automation.

Mental Model

Core Idea

Manual triggers let you start workflows anytime with custom inputs, giving you flexible control over automation.

Think of it like...

It's like ordering food at a restaurant: instead of waiting for a fixed menu schedule, you call the waiter when you want and specify exactly what you want to eat.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│ User decides  │──────▶│ Manual trigger│──────▶│ Workflow runs │
│ to start now │       │ with params   │       │ with inputs   │
└───────────────┘       └───────────────┘       └───────────────┘

Build-Up - 6 Steps

1

Foundation

What is a manual trigger in Airflow

Concept: Manual trigger means starting a workflow by hand instead of waiting for a schedule.

Airflow usually runs workflows (DAGs) on a schedule, like every hour or day. But sometimes you want to start a workflow immediately. Manual trigger lets you do this from the Airflow UI or CLI by clicking a button or running a command.

Result

You can start a workflow run instantly, ignoring the schedule.

Understanding manual triggers helps you break free from fixed schedules and run workflows exactly when needed.

2

Foundation

How to trigger a DAG manually

Concept: You can trigger a DAG manually using Airflow's UI or CLI commands.

In the Airflow UI, find your DAG and click the 'Trigger DAG' button. In the CLI, run: airflow dags trigger . This starts a new run immediately.

Result

A new DAG run appears in the UI and starts executing tasks.

Knowing both UI and CLI methods gives you flexible ways to start workflows manually.

3

Intermediate

Passing parameters with manual triggers

🤔Before reading on: do you think you can pass custom data to a DAG run when triggering manually? Commit to yes or no.

Concept: You can send extra data (parameters) when triggering a DAG manually to customize its behavior.

When triggering a DAG manually, you can pass a JSON dictionary as 'conf' parameters. For example, in CLI: airflow dags trigger --conf '{"key":"value"}'. Inside your DAG, you access these parameters via context in tasks.

Result

The workflow receives and uses the parameters you passed to change its behavior.

Passing parameters lets you reuse the same workflow for different inputs without changing code.

4

Intermediate

Accessing parameters inside tasks

🤔Before reading on: do you think parameters passed at trigger time are available inside all tasks automatically? Commit to yes or no.

Concept: Tasks can read the parameters passed during manual trigger from the execution context.

In PythonOperator or other operators, you can access parameters via the 'dag_run.conf' dictionary. Example: def task_func(**context): params = context['dag_run'].conf; print(params.get('key')). This lets tasks adapt based on input.

Result

Tasks behave differently depending on the parameters passed at trigger time.

Knowing how to access parameters inside tasks unlocks dynamic workflows that respond to user input.

5

Advanced

Validating and defaulting parameters

🤔Before reading on: do you think Airflow automatically validates parameters passed during manual triggers? Commit to yes or no.

Concept: You should validate parameters inside your DAG code and provide defaults to avoid errors.

Airflow does not check parameter correctness automatically. You can write code in your tasks to check if required keys exist and set defaults if missing. This prevents crashes and unexpected behavior.

Result

Your workflow runs safely even if parameters are missing or wrong.

Validating parameters prevents runtime failures and improves workflow robustness.

6

Expert

Triggering DAGs programmatically with parameters

🤔Before reading on: do you think you can trigger DAGs with parameters from other DAGs or external scripts? Commit to yes or no.

Concept: You can trigger DAGs with parameters programmatically using Airflow's REST API or Python client.

Airflow provides a REST API and Python client to trigger DAGs with parameters. For example, using Python: from airflow.api.client.local_client import Client; client = Client(); client.trigger_dag(dag_id='my_dag', conf={'key':'value'}). This enables automation and chaining workflows.

Result

DAGs start automatically with custom parameters from code or other workflows.

Programmatic triggers enable complex automation pipelines and integration with external systems.

Under the Hood

When you trigger a DAG manually, Airflow creates a new DAG run record in its metadata database with a timestamp and optional parameters stored as JSON. The scheduler detects this new run and queues tasks for execution. Tasks access parameters via the DAG run context passed during execution. The parameters are stored as a JSON string in the database and deserialized when accessed.

Why designed this way?

Airflow separates scheduling from execution to allow flexible triggers. Storing parameters as JSON in the metadata database keeps the system simple and extensible. This design supports both scheduled and manual runs uniformly. Alternatives like embedding parameters in task code would reduce flexibility and increase complexity.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│ Manual trigger│──────▶│ DAG run entry │──────▶│ Scheduler     │
│ with params   │       │ created in DB │       │ queues tasks  │
└───────────────┘       └───────────────┘       └───────────────┘
                                │
                                ▼
                      ┌───────────────────┐
                      │ Tasks access params│
                      │ from DAG run conf  │
                      └───────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: do you think manual triggers always run immediately without delay? Commit to yes or no.

Common Belief: Manual triggers start the workflow instantly as soon as you click or run the command.

Reality: Manual triggers create a DAG run, but the scheduler decides when tasks actually start based on resource availability and concurrency limits.

Why it matters: Expecting immediate execution can cause confusion when tasks wait in queue, leading to wrong assumptions about system health.

Quick: do you think parameters passed during manual triggers are mandatory for all DAG runs? Commit to yes or no.

Common Belief: Every manual trigger must include parameters, or the DAG will fail.

Reality: Parameters are optional; DAGs can run without them if designed with defaults or checks.

Why it matters: Believing parameters are mandatory can cause unnecessary errors or overcomplicated DAG designs.

Quick: do you think parameters passed at manual trigger time are shared across all tasks automatically? Commit to yes or no.

Common Belief: All tasks automatically receive parameters passed during manual trigger without extra code.

Reality: Tasks must explicitly access parameters from the DAG run context; parameters are not injected automatically into task code.

Why it matters: Assuming automatic availability leads to bugs where tasks don't behave as expected.

Quick: do you think manual triggers can only be done via the Airflow UI? Commit to yes or no.

Common Belief: You can only trigger DAGs manually using the Airflow web interface.

Reality: Manual triggers can also be done via CLI commands, REST API, or programmatically from other DAGs.

Why it matters: Limiting manual triggers to UI reduces automation possibilities and integration with other systems.

Expert Zone

1

Parameters passed during manual triggers are stored as JSON strings in the metadata database, so complex objects must be serialized carefully.

2

Manual triggers bypass schedule intervals but still respect DAG concurrency and pool limits, which can delay execution.

3

Using programmatic triggers with parameters enables dynamic DAG chaining and event-driven workflows beyond simple manual starts.

When NOT to use

Manual triggers are not suitable for fully automated, repeatable workflows that must run on strict schedules. For those, rely on scheduled DAG runs or event sensors. Also, avoid manual triggers for high-frequency runs as they require human intervention or external automation.

Production Patterns

In production, manual triggers with parameters are used for ad-hoc data processing, testing new DAG versions, or running workflows with special inputs. Programmatic triggers enable complex pipelines where one DAG triggers another with context, supporting modular and maintainable automation.

Connections

Event-driven architecture

Manual triggers with parameters build on event-driven principles by starting workflows in response to external inputs.

Understanding manual triggers helps grasp how systems react to events and inputs dynamically, a core idea in modern software design.

Command pattern (software design)

Manual triggers act like commands that encapsulate a request to run a workflow with parameters.

Recognizing this pattern clarifies how Airflow decouples workflow invocation from execution, improving flexibility.

Restaurant ordering system

Both involve a user specifying what they want and when, triggering a process with custom inputs.

This cross-domain link shows how user-driven requests shape workflows, making automation more responsive and personalized.

Common Pitfalls

#1 Triggering a DAG without passing required parameters causes task failures.

Wrong approach:

airflow dags trigger my_dag

Correct approach:

airflow dags trigger my_dag --conf '{"param1":"value1"}'

Root cause:

Not understanding that the DAG expects parameters to run correctly leads to missing inputs and errors.

#2 Assuming tasks automatically receive parameters without accessing dag_run.conf.

Wrong approach:

def task_func(): print(param1) # param1 undefined

Correct approach:

def task_func(**context): param1 = context['dag_run'].conf.get('param1'); print(param1)

Root cause:

Misunderstanding how Airflow passes parameters to tasks causes undefined variable errors.

#3 Expecting manual trigger to start tasks immediately ignoring concurrency limits.

Wrong approach:

Trigger DAG and assume tasks run instantly regardless of system load.

Correct approach:

Understand scheduler queues tasks respecting concurrency and pool limits; monitor task states.

Root cause:

Ignoring Airflow's scheduling and resource management leads to wrong expectations about execution timing.

Key Takeaways

Manual triggers let you start Airflow workflows anytime, breaking free from fixed schedules.

You can pass parameters during manual triggers to customize workflow behavior dynamically.

Tasks access these parameters through the DAG run context, enabling flexible and reusable workflows.

Validating parameters inside your DAG prevents runtime errors and improves reliability.

Programmatic manual triggers enable advanced automation and integration beyond the UI.