Why incremental models save time and cost in dbt - See It in Action

Why Incremental Models Save Time and Cost

📖 Scenario: You work as a data analyst in a company that processes daily sales data. The dataset grows every day, and running full data transformations every time takes a lot of time and computing resources.Your manager wants you to use incremental models in dbt to save time and reduce costs.

🎯 Goal: Build a simple incremental model in dbt that processes only new data each day instead of the entire dataset.

📋 What You'll Learn

Create a base table with daily sales data

Add a variable to track the last processed date

Write an incremental model that processes only new sales after the last processed date

Print the new_sales_data list to verify the logic

💡 Why This Matters

🌍 Real World

Companies with growing datasets use incremental models to update reports and dashboards quickly without reprocessing all data.

💼 Career

Data engineers and analysts use incremental models in dbt to optimize data pipelines, saving time and cloud computing costs.

1

Create the base sales data table

Create a variable called sales_data as a list of dictionaries with these exact entries: {'date': '2024-06-01', 'sales': 100}, {'date': '2024-06-02', 'sales': 150}, and {'date': '2024-06-03', 'sales': 200}.

dbt

# Create the sales_data list with the exact entries
# Your code here

Need a hint?

Use a list of dictionaries to represent each day's sales data.

2

Add a variable for the last processed date

Create a variable called last_processed_date and set it to the string '2024-06-01'.

dbt

sales_data = [
    {'date': '2024-06-01', 'sales': 100},
    {'date': '2024-06-02', 'sales': 150},
    {'date': '2024-06-03', 'sales': 200}
]
# Create last_processed_date variable
# Your code here

Need a hint?

This variable will help filter new data to process.

3

Write the incremental model logic

Create a list called new_sales_data that contains only the entries from sales_data where the 'date' is greater than last_processed_date.

dbt

sales_data = [
    {'date': '2024-06-01', 'sales': 100},
    {'date': '2024-06-02', 'sales': 150},
    {'date': '2024-06-03', 'sales': 200}
]
last_processed_date = '2024-06-01'
# Filter sales_data for dates after last_processed_date
# Your code here

Need a hint?

Use a list comprehension to select only new entries.

4

Print the new incremental data

Write a print statement to display the new_sales_data list.

dbt

sales_data = [
    {'date': '2024-06-01', 'sales': 100},
    {'date': '2024-06-02', 'sales': 150},
    {'date': '2024-06-03', 'sales': 200}
]
last_processed_date = '2024-06-01'
new_sales_data = [entry for entry in sales_data if entry['date'] > last_processed_date]
# Print the new_sales_data list
# Your code here

Need a hint?

The output should show only the sales data after 2024-06-01.