Slim CI with state comparison in dbt

Introduction

Slim CI helps run only the parts of your data project that changed. It saves time and computing power by comparing the current state with the last run.

When you want faster testing of your data models after small changes.

When your data project is large and full runs take too long.

When you want to avoid running unchanged models to save resources.

When you want to catch errors quickly during development.

When you want to keep your continuous integration (CI) process efficient.

Syntax

dbt

dbt build --state path/to/previous/run --select state:modified

--state points to the folder with the previous run's manifest and run results.

--select state:modified runs only models that changed since the last run.

Examples

Runs only models that changed compared to the previous run stored in ./target/previous_run.

dbt

dbt build --state ./target/previous_run --select state:modified

Runs tests only on models that were modified since the last run.

dbt

dbt test --state ./target/previous_run --select state:modified

Runs modified models and their downstream dependencies.

dbt

dbt run --state ./target/previous_run --select state:modified+

Sample Program

This Python script runs a dbt build command that uses slim CI with state comparison. It runs only the models that changed since the last run saved in ./target/previous_run. The output shows which models were run.

dbt

import os
import subprocess

# Assume previous run artifacts are saved in './target/previous_run'
previous_state_path = './target/previous_run'

# Command to run only modified models compared to previous run
command = [
    'dbt', 'build',
    '--state', previous_state_path,
    '--select', 'state:modified'
]

# Run the command and capture output
result = subprocess.run(command, capture_output=True, text=True)

print(result.stdout)

Important Notes

Make sure to save the previous run's artifacts (manifest.json, run_results.json) in the folder you point to with --state.

Slim CI only works if you have a previous run to compare with.

You can combine state:modified with other selectors like + to include dependencies.

Summary

Slim CI runs only changed models by comparing current and previous states.

Use --state to point to previous run artifacts and --select state:modified to select changed models.

This saves time and resources during development and testing.