dbt in CI/CD pipelines - Step-by-Step Execution

Concept Flow - dbt in CI/CD pipelines

Code Commit to Repo

↓

CI Pipeline Triggered

↓

Install dbt & Dependencies

↓

Run dbt Commands

↓

dbt run

↓

Report Results

↓

Deploy if OK

↓

Done

This flow shows how a code commit triggers a CI pipeline that installs dbt, runs dbt commands, reports results, and deploys if tests pass.

Execution Sample

dbt

git push origin main
# triggers CI pipeline
pip install dbt
dbt run
dbt test
dbt docs generate

This code simulates pushing changes that trigger a CI pipeline which installs dbt and runs dbt commands.

Execution Table

Step	Action	Command/Check	Result	Next Step
1	Code pushed to repo	git push origin main	Push accepted, CI triggered	Install dbt
2	Install dbt	pip install dbt	dbt installed successfully	Run dbt run
3	Run models	dbt run	Models compiled and run	Run dbt test
4	Run tests	dbt test	All tests passed	Generate docs
5	Generate docs	dbt docs generate	Documentation created	Report results
6	Report results	CI logs output	Success reported	Deploy if tests pass
7	Deploy	Deploy to production	Deployment done	Done
8	End	Pipeline complete	CI/CD pipeline finished successfully	Exit

💡 Pipeline ends after deployment and reporting success.

Variable Tracker

Variable	Start	After Step 2	After Step 3	After Step 4	After Step 5	Final
dbt_installed	False	True	True	True	True	True
models_run	False	False	True	True	True	True
tests_passed	False	False	False	True	True	True
docs_generated	False	False	False	False	True	True
deployment_done	False	False	False	False	False	True

Key Moments - 3 Insights

Why do we run 'dbt test' after 'dbt run'?

Because 'dbt run' builds the models, and 'dbt test' checks if those models are correct. This order ensures tests run on the latest models, as shown in steps 3 and 4 in the execution_table.

What happens if tests fail during 'dbt test'?

The pipeline stops before deployment to avoid pushing bad changes. This is implied by the flow where deployment only happens after tests pass (step 4 to 7).

Why generate docs in the CI pipeline?

Generating docs (step 5) helps keep documentation updated automatically with the latest models, improving team communication and transparency.

Visual Quiz - 3 Questions

Look at the execution_table, what is the state of 'tests_passed' after step 4?

A False

B Unknown

C True

D Not applicable

💡 Hint

Check the variable_tracker row for 'tests_passed' after step 4.

At which step does the pipeline generate documentation?

A Step 5

B Step 4

C Step 3

D Step 6

💡 Hint

Look at the 'Action' column in the execution_table for 'Generate docs'.

If 'dbt test' fails, what happens next in the pipeline?

A Deployment proceeds anyway

B Pipeline stops before deployment

C Docs are generated anyway

D Tests are rerun automatically

💡 Hint

Refer to the key_moments explanation about test failure stopping deployment.

Concept Snapshot

dbt in CI/CD pipelines:
- Code push triggers CI pipeline
- Install dbt and dependencies
- Run 'dbt run' to build models
- Run 'dbt test' to validate models
- Generate docs with 'dbt docs generate'
- Deploy only if tests pass
- Automates safe, tested data model deployment

Full Transcript

When you push code to your repository, it triggers a CI pipeline. The pipeline first installs dbt and its dependencies. Then it runs 'dbt run' to build your data models. After that, it runs 'dbt test' to check if the models are correct. If all tests pass, it generates documentation using 'dbt docs generate'. Finally, if everything is successful, the pipeline deploys the changes to production. If tests fail, deployment stops to keep your data safe. This flow helps automate and secure your data transformations.