Full refresh vs incremental in dbt - Hands-On Comparison

Full Refresh vs Incremental in dbt

📖 Scenario: You work as a data analyst managing sales data in a data warehouse. You want to build a dbt model that either fully refreshes the entire sales summary table or incrementally updates it with only new sales data.This helps save time and resources by not reprocessing all data every time.

🎯 Goal: Build a simple Python simulation to understand the difference between full refresh and incremental update approaches on sales data.You will create initial sales data, set a refresh mode, apply the update logic, and print the final sales summary.

📋 What You'll Learn

Create a dictionary called sales_data with sales IDs as keys and amounts as values

Create a variable called refresh_mode set to either 'full' or 'incremental'

Write logic to update a sales_summary dictionary based on refresh_mode

Print the final sales_summary dictionary

💡 Why This Matters

🌍 Real World

Data teams use full refresh to rebuild tables from scratch and incremental updates to save time by processing only new data.

💼 Career

Understanding these concepts helps data analysts and engineers optimize data pipelines and improve performance in tools like dbt.

1

Create initial sales data

Create a dictionary called sales_data with these exact entries: 101: 250, 102: 450, 103: 300

dbt

# Create the sales_data dictionary with given sales IDs and amounts
# Your code here

Need a hint?

Use curly braces to create a dictionary with keys 101, 102, 103 and their amounts.

2

Set the refresh mode

Create a variable called refresh_mode and set it to the string 'incremental'

dbt

sales_data = {101: 250, 102: 450, 103: 300}
# Set refresh_mode to 'incremental'
# Your code here

Need a hint?

Assign the string 'incremental' to the variable refresh_mode.

3

Apply update logic based on refresh mode

Create an empty dictionary called sales_summary. Then write an if statement that checks if refresh_mode is 'full'. If yes, copy all sales_data into sales_summary. Otherwise, add only new sales with IDs greater than 102 from sales_data into sales_summary.

dbt

sales_data = {101: 250, 102: 450, 103: 300}
refresh_mode = 'incremental'
# Create sales_summary and update it based on refresh_mode
# Your code here

Need a hint?

Use copy() to duplicate the dictionary for full refresh. Use a for loop and condition to add only new sales for incremental.

4

Print the final sales summary

Write a print statement to display the sales_summary dictionary.

dbt

sales_data = {101: 250, 102: 450, 103: 300}
refresh_mode = 'incremental'
sales_summary = {}
if refresh_mode == 'full':
    sales_summary = sales_data.copy()
else:
    for sale_id, amount in sales_data.items():
        if sale_id > 102:
            sales_summary[sale_id] = amount
# Print the sales_summary dictionary
# Your code here

Need a hint?

Use print(sales_summary) to show the final dictionary.