Slim CI with state comparison in dbt - Mini Project: Build & Apply

Slim CI with state comparison

📖 Scenario: You work in a data team that uses dbt to build and maintain data models. Your team wants to speed up the Continuous Integration (CI) process by running tests only on models that have changed since the last successful run.This technique is called Slim CI with state comparison. It helps save time and computing resources by focusing only on changed models.

🎯 Goal: Build a dbt project setup that uses state comparison to identify changed models and runs tests only on those models.You will create a simple dbt model, configure state comparison, and run tests selectively.

📋 What You'll Learn

Create a dbt model file with a simple SQL select statement

Add a variable to specify the path to the previous run's manifest file

Use dbt's --state flag to compare current and previous states

Run tests only on models that have changed since the last run

💡 Why This Matters

🌍 Real World

Slim CI with state comparison helps data teams run tests faster by focusing only on changed models, saving time and computing resources.

💼 Career

Understanding slim CI is valuable for data engineers and analysts working with dbt and modern data workflows to optimize testing and deployment.

1

Create a simple dbt model

Create a dbt model file named models/my_model.sql with the exact SQL query: select 1 as id, 'test' as name.

dbt

# Create the file models/my_model.sql with the SQL query
# Your code here

Hint

This is a simple SQL select statement that returns one row with two columns.

2

Add a variable for previous state path

In your dbt_project.yml file, add a variable named previous_state_path and set it to target/previous_run/manifest.json.

dbt

select 1 as id, 'test' as name

# Add variable previous_state_path in dbt_project.yml
# Your code here

Hint

The vars section in dbt_project.yml lets you define variables accessible in your dbt commands.

3

Use --state flag to compare current and previous states

Run the dbt test command with the --state flag set to target/previous_run/manifest.json to compare the current project state with the previous run. Use the command: dbt test --state target/previous_run/manifest.json.

dbt

select 1 as id, 'test' as name

vars:
  previous_state_path: target/previous_run/manifest.json

# Run dbt test with --state flag
# Your code here

Hint

The --state flag tells dbt to compare the current project with a previous manifest file.

4

Print the models tested in slim CI

Print the message exactly: Running tests only on changed models using state comparison.

dbt

select 1 as id, 'test' as name

vars:
  previous_state_path: target/previous_run/manifest.json

# Command to run:
dbt test --state target/previous_run/manifest.json

# Print the message below
# Your code here

Hint

Use a print statement to show the message.

Practice

1. What is the main purpose of using --state and --select state:modified in dbt's Slim CI?

easy

A. To delete all previous run artifacts

B. To run all models regardless of changes

C. To create a backup of the current state

D. To run only the models that have changed since the last run

Solution

1. Step 1: Understand the role of --state

   The --state flag points dbt to the previous run's artifacts, allowing comparison between runs.

2. Step 2: Understand --select state:modified

   This selects only the models that have changed compared to the previous state, skipping unchanged models.

3. Final Answer:

   To run only the models that have changed since the last run -> Option D

4. Quick Check:

   Slim CI runs changed models only [OK]

Hint: Slim CI runs changed models only, saving time [OK]

Common Mistakes:

• Thinking it runs all models every time
• Confusing state with backup or deletion
• Assuming it runs unchanged models too

2. Which of the following is the correct syntax to run dbt with Slim CI to select only modified models?

easy

A. dbt run --state path/to/previous --select all

B. dbt run --select state:modified --state path/to/previous

C. dbt run --select modified:state --state=previous_path

D. dbt run --state-modified --select path/to/previous

Solution

1. Step 1: Identify correct flag usage

   The correct flags are --state to specify previous run artifacts and --select state:modified to select changed models.

2. Step 2: Match syntax to flags

   dbt run --select state:modified --state path/to/previous correctly uses --select state:modified and --state path/to/previous in proper order and format.

3. Final Answer:

   dbt run --select state:modified --state path/to/previous -> Option B

4. Quick Check:

   Correct flags and order [OK]

Hint: Use --state then --select state:modified for Slim CI [OK]

Common Mistakes:

• Mixing up flag names or order
• Using incorrect flag syntax
• Selecting all models instead of modified

3. Given the following dbt command run twice with changes in model 'sales':

dbt run --state ./target/run1 --select state:modified
# After modifying 'sales' model
 dbt run --state ./target/run1 --select state:modified

What will the second command do?

medium

A. Run only the 'sales' model because it was modified

B. Run all models again ignoring state

C. Run no models because nothing changed

D. Throw an error due to incorrect state path

Solution

1. Step 1: Understand the first run

   The first command runs all models and saves state in ./target/run1.

2. Step 2: Analyze the second run with modification

   The second command compares current state to ./target/run1 and runs only models that changed, here 'sales'.

3. Final Answer:

   Run only the 'sales' model because it was modified -> Option A

4. Quick Check:

   Modified model runs only [OK]

Hint: Second run runs only changed models using saved state [OK]

Common Mistakes:

• Assuming all models run again
• Thinking no models run if only one changed
• Confusing state path usage

4. You run dbt run --state ./previous_run --select state:modified but get an error saying the state path is invalid. What is the most likely cause?

medium

A. The path './previous_run' does not contain valid dbt run artifacts

B. You forgot to add '--select all' flag

C. The '--state' flag must come after '--select'

D. The command requires '--full-refresh' to work

Solution

1. Step 1: Check the meaning of the error

   An invalid state path error means dbt cannot find or read previous run artifacts at the given path.

2. Step 2: Evaluate other options

   Flags order or missing '--select all' do not cause this error; '--full-refresh' is unrelated.

3. Final Answer:

   The path './previous_run' does not contain valid dbt run artifacts -> Option A

4. Quick Check:

   Invalid state path means missing artifacts [OK]

Hint: Ensure state path has valid previous run artifacts [OK]

Common Mistakes:

• Assuming flag order causes error
• Thinking '--full-refresh' fixes state errors
• Forgetting to generate previous run artifacts

5. You want to optimize your CI pipeline using Slim CI. Your previous run artifacts are stored in ./target/previous. You have models 'A', 'B', and 'C'. Only 'B' was modified. Which command will run only the modified model and also run its dependent models?

hard

A. dbt run --state ./target/previous --select state:modified

B. dbt run --state ./target/previous --select state:modified --full-refresh

C. dbt run --state ./target/previous --select state:modified+

D. dbt run --select state:modified --state ./target/previous --exclude B

Solution

1. Step 1: Understand the '+' operator in selection

   The '+' after state:modified selects modified models plus their dependents.

2. Step 2: Match command to requirement

   dbt run --state ./target/previous --select state:modified+ uses state:modified+ with correct --state path, running 'B' and its dependents.

3. Step 3: Evaluate other options

   dbt run --state ./target/previous --select state:modified runs only modified models without dependents; C adds unnecessary full-refresh; D excludes 'B' incorrectly.

4. Final Answer:

   dbt run --state ./target/previous --select state:modified+ -> Option C

5. Quick Check:

   Modified plus dependents run [OK]

Hint: Add '+' to include dependents of modified models [OK]

Common Mistakes:

• Not including dependents with '+'
• Using --full-refresh unnecessarily
• Excluding modified models by mistake