Orchestrating dbt with Airflow - Step-by-Step Execution

Concept Flow - Orchestrating dbt with Airflow

Start Airflow DAG

↓

Trigger dbt run task

↓

dbt runs models

↓

Check dbt run success?

No→Handle failure

Yes↓

Trigger dbt test task

↓

dbt runs tests

↓

Check dbt test success?

No→Handle failure

Yes↓

DAG completes successfully

Airflow starts a DAG that runs dbt models, checks success, then runs dbt tests, handling failures if any.

Execution Sample

dbt

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

default_args = {'start_date': datetime(2024, 1, 1)}

dag = DAG('dbt_airflow', default_args=default_args, schedule_interval='@daily')

dbt_run = BashOperator(task_id='dbt_run', bash_command='dbt run', dag=dag)
dbt_test = BashOperator(task_id='dbt_test', bash_command='dbt test', dag=dag)
dbt_run >> dbt_test

This Airflow DAG runs dbt models first, then runs dbt tests after the models finish.

Execution Table

Step	Task	Action	Result	Next Step
1	Start DAG	Trigger DAG run	DAG started	Run dbt_run task
2	dbt_run	Execute 'dbt run'	Models built successfully	Run dbt_test task
3	dbt_test	Execute 'dbt test'	Tests passed	DAG completes successfully
4	DAG Completion	All tasks done	Success	End

💡 DAG ends after dbt tests pass successfully

Variable Tracker

Variable	Start	After dbt_run	After dbt_test	Final
DAG Status	Not started	dbt_run success	dbt_test success	Success
dbt_run Status	Not started	Success	N/A	Success
dbt_test Status	Not started	N/A	Success	Success

Key Moments - 2 Insights

Why does dbt_test run only after dbt_run finishes?

Because in the execution_table, step 2 shows dbt_run completes before step 3 starts dbt_test, ensuring models exist before testing.

What happens if dbt_run fails?

The DAG stops and handles failure as shown by the 'Check dbt run success?' decision in the concept_flow, so dbt_test does not run.

Visual Quiz - 3 Questions

Look at the execution_table, what is the result of the dbt_run task at step 2?

A DAG started

B Tests passed

C Models built successfully

D Failure

💡 Hint

Check the 'Result' column for step 2 in the execution_table.

At which step does the DAG complete successfully?

A Step 2

B Step 4

C Step 3

D Step 1

💡 Hint

Look at the 'Task' and 'Result' columns in the execution_table for the final step.

If dbt_run fails, what happens to dbt_test?

A dbt_test is skipped

B DAG restarts from beginning

C dbt_test runs anyway

D dbt_test runs before dbt_run

💡 Hint

Refer to the concept_flow where failure after dbt_run leads to handling failure and no dbt_test run.

Concept Snapshot

Airflow DAG runs dbt tasks in order:
1. dbt run builds models
2. dbt test runs tests
Tasks run sequentially
Failures stop the DAG
Use BashOperator to run dbt commands

Full Transcript

This visual execution shows how Airflow orchestrates dbt workflows. The DAG starts and triggers the dbt run task to build models. After successful completion, it triggers the dbt test task to run tests on those models. If any task fails, the DAG stops and handles the failure. Variables track the status of each task and the overall DAG. This ensures dbt models are built before tests run, maintaining data pipeline integrity.