Full refresh vs incremental in dbt - Visual Side-by-Side Comparison

Concept Flow - Full refresh vs incremental

Start dbt run

↓

Check model config

↓

Run full refresh

↓

Drop and recreate table

↓

Load all data

↓

Finish

↓

Run incremental

↓

Check for new/updated data

↓

Append or update only changed data

↓

Finish

The flow checks if a full refresh is requested; if yes, it rebuilds the entire table. Otherwise, it processes only new or changed data incrementally.

Execution Sample

dbt

model_config:
  materialized: incremental
  full_refresh: false

-- SQL logic for incremental load

This config runs the model incrementally, loading only new or changed data unless full_refresh is set to true.

Execution Table

Step	full_refresh flag	Action	Data Processed	Resulting Table State
1	false	Check full_refresh flag	N/A	No full refresh, proceed incremental
2	false	Identify new/updated rows	Only new/changed rows	Prepare incremental load
3	false	Append/update incremental data	New/changed rows added	Table updated incrementally
4	false	Finish run	N/A	Incremental table ready
5	true	Check full_refresh flag	N/A	Full refresh triggered
6	true	Drop existing table	N/A	Table dropped
7	true	Load all data fresh	All rows	Table rebuilt fully
8	true	Finish run	N/A	Full refresh table ready

💡 Execution stops after finishing either full refresh or incremental load.

Variable Tracker

Variable	Start	After Step 1	After Step 5	Final
full_refresh	false	false	true	true
data_processed	none	new/changed rows	all rows	all rows
table_state	empty or old	incrementally updated	dropped	fully rebuilt

Key Moments - 3 Insights

Why does the full refresh drop the table before loading data?

Because full refresh means rebuilding the entire table from scratch, so the old data must be removed first (see execution_table rows 6 and 7).

What happens if full_refresh is false but the source data has no new rows?

The incremental load finds no new or changed rows, so no data is added or updated, and the table remains the same (see execution_table row 2 and 3).

Can incremental mode handle updates to existing rows?

Yes, incremental mode can update existing rows if configured properly, by identifying changed rows and applying updates (see execution_table row 3).

Visual Quiz - 3 Questions

Look at the execution table, what is the table state after step 3 when full_refresh is false?

A Table updated incrementally

B Table dropped

C Table fully rebuilt

D Table empty

💡 Hint

Refer to execution_table row 3 under 'Resulting Table State'

At which step does the full refresh drop the existing table?

A Step 4

B Step 6

C Step 2

D Step 8

💡 Hint

Check execution_table row 6 for the drop action

If full_refresh is set to true, which steps will be executed?

A Steps 1 to 4

B Steps 5 to 8

C Only step 1

D Steps 2 and 3 only

💡 Hint

Look at execution_table rows 5 to 8 for full refresh flow

Concept Snapshot

Full refresh rebuilds the entire table by dropping and reloading all data.
Incremental loads only new or changed data, appending or updating the table.
Use full_refresh: true to force full rebuild.
Use materialized: incremental for efficient updates.
Full refresh is slower but ensures clean state.
Incremental is faster for ongoing data loads.

Full Transcript

This visual execution shows how dbt decides between full refresh and incremental runs. When full_refresh is true, dbt drops the existing table and reloads all data, ensuring a fresh start. When false, it only processes new or changed rows, updating the table incrementally. Variables like full_refresh flag and data_processed change during execution. Key moments include understanding why full refresh drops the table and how incremental handles updates. The quiz tests knowledge of table states and step actions. This helps beginners see exactly what happens step-by-step during dbt runs.