Slim CI with state comparison in dbt - Step-by-Step Execution

Concept Flow - Slim CI with state comparison

Start: Trigger CI

↓

Fetch current state

↓

Fetch previous state

↓

Compare states

↓

Run only [Skip run

↓

Report results

↓

End

The CI process fetches current and previous states, compares them, runs only changed models if any, otherwise skips, then reports results.

Execution Sample

dbt

dbt run --state ./target
# Compares current models with previous run
# Runs only changed models

This command runs dbt models selectively by comparing current state with previous state to speed up CI.

Execution Table

Step	Action	Current State	Previous State	Comparison Result	CI Action
1	Fetch current state	models: A, B, C	N/A	N/A	Prepare for comparison
2	Fetch previous state	models: A, B, C	models: A, B, C	N/A	Ready to compare
3	Compare states	models: A, B, C	models: A, B, C	No changes detected	Skip model runs
4	Report results	N/A	N/A	No models run	CI passes quickly

💡 No changes detected, so CI skips running models to save time.

Variable Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	Final
current_state	empty	models: A, B, C	models: A, B, C	models: A, B, C	models: A, B, C
previous_state	empty	empty	models: A, B, C	models: A, B, C	models: A, B, C
comparison_result	none	none	none	no changes	no changes
ci_action	none	none	none	skip runs	skip runs

Key Moments - 2 Insights

Why does the CI skip running models even though the command 'dbt run' was executed?

Because the comparison_result in the execution_table at step 3 shows 'No changes detected', so the CI action is to skip running models to save time.

What happens if the previous state is missing or empty?

If previous_state is empty, the comparison cannot detect changes, so CI will run all models by default to ensure correctness.

Visual Quiz - 3 Questions

Look at the execution_table at step 3, what is the comparison result?

A No changes detected

B Models A and B changed

C Previous state missing

D All models changed

💡 Hint

Check the 'Comparison Result' column in row 3 of the execution_table.

At which step does the CI decide to skip running models?

A Step 1

B Step 2

C Step 3

D Step 4

💡 Hint

Look at the 'CI Action' column in the execution_table to find when 'skip runs' happens.

If a model changed between states, how would the CI action change in the table?

A CI would skip running models

B CI would run only changed models

C CI would run no models

D CI would run all models regardless

💡 Hint

Refer to the concept_flow where changed models trigger running only those models.

Concept Snapshot

Slim CI with state comparison:
- Fetch current and previous states
- Compare models for changes
- Run only changed models if any
- Skip runs if no changes
- Saves CI time and resources

Full Transcript

Slim CI with state comparison is a process where the continuous integration system fetches the current state of dbt models and compares it with the previous state. If no changes are detected, the CI skips running models to save time. If changes exist, only those changed models run. This approach speeds up CI by avoiding unnecessary runs. The execution table shows steps from fetching states, comparing, deciding CI action, and reporting results. Variables track the states and decisions. Key moments clarify why skipping happens and what if previous state is missing. The quiz tests understanding of comparison results, decision steps, and CI actions based on changes.

Practice

1. What is the main purpose of using --state and --select state:modified in dbt's Slim CI?

easy

A. To delete all previous run artifacts

B. To run all models regardless of changes

C. To create a backup of the current state

D. To run only the models that have changed since the last run

Solution

1. Step 1: Understand the role of --state

   The --state flag points dbt to the previous run's artifacts, allowing comparison between runs.

2. Step 2: Understand --select state:modified

   This selects only the models that have changed compared to the previous state, skipping unchanged models.

3. Final Answer:

   To run only the models that have changed since the last run -> Option D

4. Quick Check:

   Slim CI runs changed models only [OK]

Hint: Slim CI runs changed models only, saving time [OK]

Common Mistakes:

• Thinking it runs all models every time
• Confusing state with backup or deletion
• Assuming it runs unchanged models too

2. Which of the following is the correct syntax to run dbt with Slim CI to select only modified models?

easy

A. dbt run --state path/to/previous --select all

B. dbt run --select state:modified --state path/to/previous

C. dbt run --select modified:state --state=previous_path

D. dbt run --state-modified --select path/to/previous

Solution

1. Step 1: Identify correct flag usage

   The correct flags are --state to specify previous run artifacts and --select state:modified to select changed models.

2. Step 2: Match syntax to flags

   dbt run --select state:modified --state path/to/previous correctly uses --select state:modified and --state path/to/previous in proper order and format.

3. Final Answer:

   dbt run --select state:modified --state path/to/previous -> Option B

4. Quick Check:

   Correct flags and order [OK]

Hint: Use --state then --select state:modified for Slim CI [OK]

Common Mistakes:

• Mixing up flag names or order
• Using incorrect flag syntax
• Selecting all models instead of modified

3. Given the following dbt command run twice with changes in model 'sales':

dbt run --state ./target/run1 --select state:modified
# After modifying 'sales' model
 dbt run --state ./target/run1 --select state:modified

What will the second command do?

medium

A. Run only the 'sales' model because it was modified

B. Run all models again ignoring state

C. Run no models because nothing changed

D. Throw an error due to incorrect state path

Solution

1. Step 1: Understand the first run

   The first command runs all models and saves state in ./target/run1.

2. Step 2: Analyze the second run with modification

   The second command compares current state to ./target/run1 and runs only models that changed, here 'sales'.

3. Final Answer:

   Run only the 'sales' model because it was modified -> Option A

4. Quick Check:

   Modified model runs only [OK]

Hint: Second run runs only changed models using saved state [OK]

Common Mistakes:

• Assuming all models run again
• Thinking no models run if only one changed
• Confusing state path usage

4. You run dbt run --state ./previous_run --select state:modified but get an error saying the state path is invalid. What is the most likely cause?

medium

A. The path './previous_run' does not contain valid dbt run artifacts

B. You forgot to add '--select all' flag

C. The '--state' flag must come after '--select'

D. The command requires '--full-refresh' to work

Solution

1. Step 1: Check the meaning of the error

   An invalid state path error means dbt cannot find or read previous run artifacts at the given path.

2. Step 2: Evaluate other options

   Flags order or missing '--select all' do not cause this error; '--full-refresh' is unrelated.

3. Final Answer:

   The path './previous_run' does not contain valid dbt run artifacts -> Option A

4. Quick Check:

   Invalid state path means missing artifacts [OK]

Hint: Ensure state path has valid previous run artifacts [OK]

Common Mistakes:

• Assuming flag order causes error
• Thinking '--full-refresh' fixes state errors
• Forgetting to generate previous run artifacts

5. You want to optimize your CI pipeline using Slim CI. Your previous run artifacts are stored in ./target/previous. You have models 'A', 'B', and 'C'. Only 'B' was modified. Which command will run only the modified model and also run its dependent models?

hard

A. dbt run --state ./target/previous --select state:modified

B. dbt run --state ./target/previous --select state:modified --full-refresh

C. dbt run --state ./target/previous --select state:modified+

D. dbt run --select state:modified --state ./target/previous --exclude B

Solution

1. Step 1: Understand the '+' operator in selection

   The '+' after state:modified selects modified models plus their dependents.

2. Step 2: Match command to requirement

   dbt run --state ./target/previous --select state:modified+ uses state:modified+ with correct --state path, running 'B' and its dependents.

3. Step 3: Evaluate other options

   dbt run --state ./target/previous --select state:modified runs only modified models without dependents; C adds unnecessary full-refresh; D excludes 'B' incorrectly.

4. Final Answer:

   dbt run --state ./target/previous --select state:modified+ -> Option C

5. Quick Check:

   Modified plus dependents run [OK]

Hint: Add '+' to include dependents of modified models [OK]

Common Mistakes:

• Not including dependents with '+'
• Using --full-refresh unnecessarily
• Excluding modified models by mistake