dbt in CI/CD pipelines - Mini Project: Build & Apply

dbt in CI/CD pipelines

📖 Scenario: You work in a data team that uses dbt (data build tool) to transform data in your warehouse. Your team wants to automate testing and deployment of dbt models using a CI/CD pipeline. This project will guide you through setting up a simple dbt project, configuring a test command, running the core dbt commands, and finally printing the test results.

🎯 Goal: Build a basic dbt project setup and simulate running dbt commands in a CI/CD pipeline script. You will create the initial dbt project structure, add a configuration for the test command, run the core dbt commands in sequence, and output the test results.

📋 What You'll Learn

Create a dbt project dictionary with key details

Add a configuration variable for the test command

Write a sequence of dbt commands to run in the pipeline

Print the final test results message

💡 Why This Matters

🌍 Real World

Data teams use dbt to transform and test data models. Automating dbt commands in CI/CD pipelines ensures data quality and faster deployments.

💼 Career

Understanding how to script and automate dbt commands is valuable for data engineers and analytics engineers working with modern data stacks.

1

Create the initial dbt project dictionary

Create a dictionary called dbt_project with these exact keys and values: 'name' set to 'my_dbt_project', 'version' set to '1.0', and 'models' set to a list containing 'customers' and 'orders'.

dbt

# Create the dbt_project dictionary with name, version, and models keys
# Your code here

Need a hint?

Use curly braces {} to create a dictionary and square brackets [] for the list of models.

2

Add a test command configuration

Add a variable called test_command and set it to the string 'dbt test' to represent the dbt test command in the pipeline.

dbt

dbt_project = {
    'name': 'my_dbt_project',
    'version': '1.0',
    'models': ['customers', 'orders']
}
# Add the test_command variable below
# Your code here

Need a hint?

Assign the string 'dbt test' to the variable test_command.

3

Write the core dbt pipeline commands

Create a list called pipeline_commands with these exact strings in order: 'dbt deps', 'dbt seed', 'dbt run', and the variable test_command as the last item.

dbt

dbt_project = {
    'name': 'my_dbt_project',
    'version': '1.0',
    'models': ['customers', 'orders']
}
test_command = 'dbt test'
# Create the pipeline_commands list below
# Your code here

Need a hint?

Use a list with the exact strings and include the variable test_command as the last element.

4

Print the test results message

Write a print statement that outputs exactly: 'All dbt tests passed successfully!'.

dbt

dbt_project = {
    'name': 'my_dbt_project',
    'version': '1.0',
    'models': ['customers', 'orders']
}
test_command = 'dbt test'
pipeline_commands = ['dbt deps', 'dbt seed', 'dbt run', test_command]
# Print the test results message below
# Your code here

Need a hint?

Use print() with the exact message inside quotes.