Environment management (dev, staging, prod) in dbt - Mini Project: Build & Apply

Environment management (dev, staging, prod)

📖 Scenario: You work as a data analyst using dbt to build data models. Your team uses three environments: development (dev), staging, and production (prod). Each environment has its own database schema to keep data separate and safe.Managing these environments correctly helps you test changes without affecting live data.

🎯 Goal: You will create a dbt project configuration that defines separate schemas for dev, staging, and prod environments. Then, you will write a model that uses the correct schema based on the environment. Finally, you will print the schema name used in the current environment.

📋 What You'll Learn

Create a dbt profiles.yml configuration with dev, staging, and prod schemas

Create dbt_project.yml linking to the profile

Write a dbt model SQL file that uses the schema based on target

Print the schema name used for the current environment

💡 Why This Matters

🌍 Real World

Managing multiple environments in dbt helps teams develop and test data models safely without affecting production data.

💼 Career

Data engineers and analysts often need to configure and use environment management in dbt to ensure reliable and safe data workflows.

1

Create dbt profiles.yml with dev, staging, and prod schemas

Create a profiles.yml file with a profile named my_dbt_project. Define three targets: dev with schema my_schema_dev, staging with schema my_schema_staging, and prod with schema my_schema_prod. Set dev as the default target.

dbt

# Write the profiles.yml content here
# Your code here

Hint

Use YAML syntax to define the profile and targets. Remember to indent properly.

2

Create the dbt_project.yml configuration

Create a dbt_project.yml file named my_dbt_project that links to the profile my_dbt_project. Use config-version: 2.

dbt

# Write the dbt_project.yml content here
# Your code here

Hint

Define the project name, version, config-version, and profile in YAML format.

3

Write a dbt model SQL file using the schema based on target

Create a dbt model SQL file named my_model.sql. Use the Jinja variable target to select the schema dynamically. Write a simple SELECT statement that selects all columns from a table named source_table in the schema {{ target.schema }}.

dbt

-- Write SQL for my_model.sql below
-- Your code here

Hint

Use Jinja templating syntax with double curly braces to insert the schema dynamically.

4

Print the schema name used in the current environment

Modify your my_model.sql to log the schema name used for the current target environment using dbt's {{ log() }} function.

dbt

-- Add the log statement here
-- Your code here

Hint

Use {{ log('Current schema: ' ~ target.schema) }} to print the schema name dynamically.

Practice

1. What is the main purpose of having separate environments like dev, staging, and prod in dbt?

easy

A. To run multiple dbt projects at the same time

B. To safely develop and test changes before applying them to production

C. To store different versions of dbt itself

D. To backup data automatically

Solution

1. Step 1: Understand environment roles

   Development (dev) is for building and testing changes. Staging is for final testing. Production (prod) is the live environment.

2. Step 2: Purpose of separation

   This separation prevents untested changes from affecting live data and users.

3. Final Answer:

   To safely develop and test changes before applying them to production -> Option B

4. Quick Check:

   Environment separation = safe testing before production [OK]

Hint: Think: Why not test directly in production? [OK]

Common Mistakes:

• Confusing environment roles
• Thinking environments store dbt versions
• Assuming environments backup data automatically

2. Which command correctly runs dbt models in the staging environment?

easy

A. dbt run --profile staging

B. dbt run --env staging

C. dbt run --target staging

D. dbt run --environment staging

Solution

1. Step 1: Identify the correct flag for environment selection

   dbt uses the --target flag to specify which environment to run against.

2. Step 2: Match the flag with the environment name

   Using --target staging runs models in the staging environment.

3. Final Answer:

   dbt run --target staging -> Option C

4. Quick Check:

   Use --target to switch environments [OK]

Hint: Remember: --target selects environment in dbt commands [OK]

Common Mistakes:

• Using --env or --environment instead of --target
• Confusing --profile with environment selection
• Omitting the flag and running default environment

3. Given the following profiles.yml snippet, what database will dbt connect to when running with --target prod?

my_project:
  outputs:
    dev:
      type: postgres
      host: dev-db.example.com
      dbname: dev_db
    prod:
      type: postgres
      host: prod-db.example.com
      dbname: prod_db
  target: dev

medium

A. prod_db on prod-db.example.com

B. dev_db on dev-db.example.com

C. dev_db on prod-db.example.com

D. prod_db on dev-db.example.com

Solution

1. Step 1: Understand the --target flag effect

   Running with --target prod tells dbt to use the prod output configuration.

2. Step 2: Identify prod output details

   The prod output connects to prod-db.example.com and database prod_db.

3. Final Answer:

   prod_db on prod-db.example.com -> Option A

4. Quick Check:

   --target prod uses prod output config [OK]

Hint: Match --target value to profiles.yml output key [OK]

Common Mistakes:

• Ignoring --target and using default target
• Mixing host and dbname from different environments
• Assuming target in profiles.yml overrides command line

4. You run dbt run --target staging but get a connection error. The profiles.yml has a typo in the staging host key as hst instead of host. What is the best fix?

medium

A. Run dbt clean before running again

B. Change the command to dbt run --target hst

C. Remove the staging environment from profiles.yml

D. Correct the typo from hst to host in profiles.yml

Solution

1. Step 1: Identify cause of connection error

   A typo in the key hst means dbt cannot find the host to connect to.

2. Step 2: Fix the typo in profiles.yml

   Changing hst to host corrects the connection details.

3. Final Answer:

   Correct the typo from hst to host in profiles.yml -> Option D

4. Quick Check:

   Fix config typos to resolve connection errors [OK]

Hint: Check spelling of keys in profiles.yml for connection issues [OK]

Common Mistakes:

• Changing command flags instead of fixing config
• Deleting environment instead of fixing it
• Running unrelated dbt commands

5. You want to test a new model in your dev environment without affecting staging or prod. Which approach is best to manage this safely?

hard

A. Create a separate dev target in profiles.yml and run with --target dev

B. Run the model directly in prod and rollback if needed

C. Delete the staging environment temporarily

D. Run all models in dev and prod simultaneously

Solution

1. Step 1: Understand environment isolation

   Using a separate dev target isolates testing from staging and production.

2. Step 2: Use --target dev to run safely

   Running with --target dev ensures changes only affect the dev environment.

3. Final Answer:

   Create a separate dev target in profiles.yml and run with --target dev -> Option A

4. Quick Check:

   Use dev target for safe isolated testing [OK]

Hint: Always test in dev environment using --target dev [OK]

Common Mistakes:

• Testing directly in production
• Removing staging environment
• Running multiple environments at once