Environment management (dev, staging, prod) in dbt

Introduction

Environment management helps keep your work organized and safe. It separates your development, testing, and live data so you don't mix them up.

When you want to try new changes without affecting live data.

When you need to test your data models before sharing results.

When you want to review changes with your team before going live.

When you want to keep your production data safe from errors.

When you want to deploy updates step-by-step.

Syntax

dbt

profiles.yml

my_profile:
  target: dev
  outputs:
    dev:
      type: postgres
      host: localhost
      user: user_dev
      password: pass_dev
      dbname: mydb_dev
      schema: dev_schema
    staging:
      type: postgres
      host: staging_host
      user: user_staging
      password: pass_staging
      dbname: mydb_staging
      schema: staging_schema
    prod:
      type: postgres
      host: prod_host
      user: user_prod
      password: pass_prod
      dbname: mydb_prod
      schema: prod_schema

The profiles.yml file defines connection details for each environment.

You switch environments by changing the target in your profile or using the --target flag when running dbt commands.

Examples

This runs your dbt models in the development environment.

dbt

dbt run --target dev

This runs your dbt models in the staging environment for testing.

dbt

dbt run --target staging

This runs your dbt models in the production environment, updating live data.

dbt

dbt run --target prod

Sample Program

This example shows how to run dbt commands in different environments using the --target flag. Each environment uses its own database and schema to keep data separate.

dbt

# profiles.yml example

# Define three environments: dev, staging, prod

# Run dbt in dev environment
!dbt run --target dev

# Run dbt in staging environment
!dbt run --target staging

# Run dbt in prod environment
!dbt run --target prod

Important Notes

Always test your changes in dev or staging before running in prod.

Keep your credentials safe and do not share production passwords.

You can automate environment switching in CI/CD pipelines for smooth deployments.

Summary

Environment management separates your work into dev, staging, and prod.

Use profiles.yml to set connection details for each environment.

Switch environments easily with the --target flag when running dbt commands.

Practice

1. What is the main purpose of having separate environments like dev, staging, and prod in dbt?

easy

A. To run multiple dbt projects at the same time

B. To safely develop and test changes before applying them to production

C. To store different versions of dbt itself

D. To backup data automatically

Solution

1. Step 1: Understand environment roles

   Development (dev) is for building and testing changes. Staging is for final testing. Production (prod) is the live environment.

2. Step 2: Purpose of separation

   This separation prevents untested changes from affecting live data and users.

3. Final Answer:

   To safely develop and test changes before applying them to production -> Option B

4. Quick Check:

   Environment separation = safe testing before production [OK]

Hint: Think: Why not test directly in production? [OK]

Common Mistakes:

• Confusing environment roles
• Thinking environments store dbt versions
• Assuming environments backup data automatically

2. Which command correctly runs dbt models in the staging environment?

easy

A. dbt run --profile staging

B. dbt run --env staging

C. dbt run --target staging

D. dbt run --environment staging

Solution

1. Step 1: Identify the correct flag for environment selection

   dbt uses the --target flag to specify which environment to run against.

2. Step 2: Match the flag with the environment name

   Using --target staging runs models in the staging environment.

3. Final Answer:

   dbt run --target staging -> Option C

4. Quick Check:

   Use --target to switch environments [OK]

Hint: Remember: --target selects environment in dbt commands [OK]

Common Mistakes:

• Using --env or --environment instead of --target
• Confusing --profile with environment selection
• Omitting the flag and running default environment

3. Given the following profiles.yml snippet, what database will dbt connect to when running with --target prod?

my_project:
  outputs:
    dev:
      type: postgres
      host: dev-db.example.com
      dbname: dev_db
    prod:
      type: postgres
      host: prod-db.example.com
      dbname: prod_db
  target: dev

medium

A. prod_db on prod-db.example.com

B. dev_db on dev-db.example.com

C. dev_db on prod-db.example.com

D. prod_db on dev-db.example.com

Solution

1. Step 1: Understand the --target flag effect

   Running with --target prod tells dbt to use the prod output configuration.

2. Step 2: Identify prod output details

   The prod output connects to prod-db.example.com and database prod_db.

3. Final Answer:

   prod_db on prod-db.example.com -> Option A

4. Quick Check:

   --target prod uses prod output config [OK]

Hint: Match --target value to profiles.yml output key [OK]

Common Mistakes:

• Ignoring --target and using default target
• Mixing host and dbname from different environments
• Assuming target in profiles.yml overrides command line

4. You run dbt run --target staging but get a connection error. The profiles.yml has a typo in the staging host key as hst instead of host. What is the best fix?

medium

A. Run dbt clean before running again

B. Change the command to dbt run --target hst

C. Remove the staging environment from profiles.yml

D. Correct the typo from hst to host in profiles.yml

Solution

1. Step 1: Identify cause of connection error

   A typo in the key hst means dbt cannot find the host to connect to.

2. Step 2: Fix the typo in profiles.yml

   Changing hst to host corrects the connection details.

3. Final Answer:

   Correct the typo from hst to host in profiles.yml -> Option D

4. Quick Check:

   Fix config typos to resolve connection errors [OK]

Hint: Check spelling of keys in profiles.yml for connection issues [OK]

Common Mistakes:

• Changing command flags instead of fixing config
• Deleting environment instead of fixing it
• Running unrelated dbt commands

5. You want to test a new model in your dev environment without affecting staging or prod. Which approach is best to manage this safely?

hard

A. Create a separate dev target in profiles.yml and run with --target dev

B. Run the model directly in prod and rollback if needed

C. Delete the staging environment temporarily

D. Run all models in dev and prod simultaneously

Solution

1. Step 1: Understand environment isolation

   Using a separate dev target isolates testing from staging and production.

2. Step 2: Use --target dev to run safely

   Running with --target dev ensures changes only affect the dev environment.

3. Final Answer:

   Create a separate dev target in profiles.yml and run with --target dev -> Option A

4. Quick Check:

   Use dev target for safe isolated testing [OK]

Hint: Always test in dev environment using --target dev [OK]

Common Mistakes:

• Testing directly in production
• Removing staging environment
• Running multiple environments at once