Slim CI with state comparison in dbt - Mini Project: Build & Apply

Slim CI with state comparison

📖 Scenario: You work in a data team that uses dbt to build and maintain data models. Your team wants to speed up the Continuous Integration (CI) process by running tests only on models that have changed since the last successful run.This technique is called Slim CI with state comparison. It helps save time and computing resources by focusing only on changed models.

🎯 Goal: Build a dbt project setup that uses state comparison to identify changed models and runs tests only on those models.You will create a simple dbt model, configure state comparison, and run tests selectively.

📋 What You'll Learn

Create a dbt model file with a simple SQL select statement

Add a variable to specify the path to the previous run's manifest file

Use dbt's --state flag to compare current and previous states

Run tests only on models that have changed since the last run

💡 Why This Matters

🌍 Real World

Slim CI with state comparison helps data teams run tests faster by focusing only on changed models, saving time and computing resources.

💼 Career

Understanding slim CI is valuable for data engineers and analysts working with dbt and modern data workflows to optimize testing and deployment.

1

Create a simple dbt model

Create a dbt model file named models/my_model.sql with the exact SQL query: select 1 as id, 'test' as name.

dbt

# Create the file models/my_model.sql with the SQL query
# Your code here

Need a hint?

This is a simple SQL select statement that returns one row with two columns.

2

Add a variable for previous state path

In your dbt_project.yml file, add a variable named previous_state_path and set it to target/previous_run/manifest.json.

dbt

select 1 as id, 'test' as name

# Add variable previous_state_path in dbt_project.yml
# Your code here

Need a hint?

The vars section in dbt_project.yml lets you define variables accessible in your dbt commands.

3

Use --state flag to compare current and previous states

Run the dbt test command with the --state flag set to target/previous_run/manifest.json to compare the current project state with the previous run. Use the command: dbt test --state target/previous_run/manifest.json.

dbt

select 1 as id, 'test' as name

vars:
  previous_state_path: target/previous_run/manifest.json

# Run dbt test with --state flag
# Your code here

Need a hint?

The --state flag tells dbt to compare the current project with a previous manifest file.

4

Print the models tested in slim CI

Print the message exactly: Running tests only on changed models using state comparison.

dbt

select 1 as id, 'test' as name

vars:
  previous_state_path: target/previous_run/manifest.json

# Command to run:
dbt test --state target/previous_run/manifest.json

# Print the message below
# Your code here

Need a hint?

Use a print statement to show the message.