Environment variables and secrets in GCP - Deep Dive

Overview - Environment variables and secrets

What is it?

Environment variables are simple key-value pairs used to store configuration settings outside of application code. Secrets are sensitive pieces of information like passwords or API keys that need extra protection. In cloud platforms like Google Cloud Platform (GCP), environment variables and secrets help applications run securely and flexibly without hardcoding sensitive data. They allow easy changes to settings without changing the code itself.

Why it matters

Without environment variables and secrets, sensitive information would be stored directly in code, risking exposure and making updates difficult. This could lead to security breaches and operational problems. Using environment variables and secrets keeps sensitive data safe, supports different environments like testing and production, and makes applications easier to manage and update.

Where it fits

Before learning this, you should understand basic cloud concepts and how applications run in the cloud. After this, you can learn about secure secret management services, access control, and automated deployment pipelines that use environment variables and secrets.

Mental Model

Core Idea

Environment variables and secrets are like a secure, changeable settings board that applications read from to know how to behave without exposing sensitive details in their code.

Think of it like...

Imagine a restaurant kitchen where the recipe (code) is written on a card, but the secret sauce ingredients (secrets) and daily specials (environment variables) are kept on a separate board only the chef can see and change without rewriting the recipe.

┌─────────────────────────────┐
│       Application Code       │
│  (fixed instructions)        │
├─────────────┬───────────────┤
│ Environment │    Secrets    │
│ Variables   │ (protected)   │
│ (config)    │               │
└─────┬───────┴───────┬───────┘
      │               │
      ▼               ▼
  Application reads settings and sensitive data securely at runtime

Build-Up - 7 Steps

1

Foundation

What are environment variables

Concept: Introduce environment variables as external configuration for applications.

Environment variables are named values stored outside the application code. They tell the app things like which database to use or what mode to run in (test or production). For example, setting DATABASE_URL as an environment variable lets the app connect to the right database without changing code.

Result

Applications can change behavior by reading environment variables without code changes.

Understanding environment variables helps separate configuration from code, making apps flexible and easier to manage.

2

Foundation

What are secrets and why protect them

Concept: Explain secrets as sensitive data needing special handling.

Secrets include passwords, API keys, and tokens that must stay private. Unlike regular environment variables, secrets require secure storage and controlled access to prevent leaks. In GCP, secrets are stored in Secret Manager, which encrypts and controls who can see them.

Result

Sensitive data is kept safe and separate from code and regular config.

Knowing the difference between config and secrets prevents accidental exposure of sensitive information.

3

Intermediate

Using environment variables in GCP services

🤔Before reading on: do you think environment variables are set inside the application code or configured externally in GCP? Commit to your answer.

Concept: Show how to set environment variables in GCP services like Cloud Run or App Engine.

In GCP, you configure environment variables outside your code. For example, in Cloud Run, you specify key-value pairs in the service settings. The app reads these at runtime. This lets you change settings like API endpoints or feature flags without redeploying code.

Result

Applications running on GCP services can adapt to different environments by reading externally set variables.

Understanding external configuration in GCP services enables safer and more flexible deployments.

4

Intermediate

Accessing secrets securely with Secret Manager

🤔Before reading on: do you think secrets are automatically injected as environment variables or accessed via API calls? Commit to your answer.

Concept: Explain how to retrieve secrets from GCP Secret Manager at runtime.

Secrets are stored encrypted in Secret Manager. Applications request secrets using GCP APIs or client libraries with proper permissions. This avoids storing secrets in environment variables directly, reducing risk. For example, your app can call Secret Manager to get a database password when it starts.

Result

Secrets remain encrypted and access-controlled, improving security.

Knowing how to access secrets dynamically prevents hardcoding and accidental leaks.

5

Intermediate

Combining environment variables and secrets

🤔Before reading on: do you think all sensitive data should be environment variables or only non-sensitive config? Commit to your answer.

Concept: Teach best practice to use environment variables for non-sensitive config and secrets for sensitive data.

Use environment variables for general settings like feature toggles or URLs. Use Secret Manager for passwords and keys. Your app reads environment variables normally but calls Secret Manager API for secrets. This separation keeps sensitive data safer and config simpler.

Result

Clear separation improves security and maintainability.

Understanding this separation helps design secure and manageable cloud applications.

6

Advanced

Managing secret versions and rotation

🤔Before reading on: do you think secrets once stored never change or should be updated regularly? Commit to your answer.

Concept: Introduce secret versioning and rotation to keep secrets fresh and secure.

GCP Secret Manager supports multiple versions of a secret. You can create new versions when passwords change and mark old ones inactive. Rotation means regularly updating secrets to reduce risk if leaked. Your app can be designed to fetch the latest secret version automatically.

Result

Secrets stay up-to-date and reduce exposure risk.

Knowing secret rotation practices is key to long-term security in production.

7

Expert

Avoiding secret exposure in logs and builds

🤔Before reading on: do you think secrets can accidentally appear in logs or build files? Commit to your answer.

Concept: Explain how secrets can leak if not handled carefully in CI/CD and logging.

If secrets are printed in logs or included in build artifacts, they become exposed. Best practice is to never log secret values and use GCP IAM roles to restrict access. Use environment variables only for non-sensitive data in builds. Secrets should be fetched at runtime, not baked into images or code.

Result

Reduced risk of accidental secret leaks in production.

Understanding how secrets leak helps prevent costly security incidents.

Under the Hood

Environment variables are stored by the operating system or cloud runtime as key-value pairs accessible to running processes. Secrets in GCP are stored encrypted in Secret Manager, which uses strong encryption keys managed by Google. Access to secrets is controlled by Identity and Access Management (IAM) policies. When an application requests a secret, Secret Manager decrypts it on the fly and returns it securely over an authenticated API call.

Why designed this way?

Separating config from code and protecting secrets with encryption and access control was designed to reduce risk of accidental leaks and simplify management. Early systems stored secrets in code or config files, causing security breaches. Cloud providers introduced managed secret stores and environment variables to improve security and operational flexibility.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│ Application   │──────▶│ Environment   │       │ Secret Manager│
│ Runtime       │       │ Variables     │       │ (Encrypted)   │
│ (reads config)│       │ (OS or Cloud) │       │               │
└───────┬───────┘       └───────┬───────┘       └───────┬───────┘
        │                       │                       │
        │                       │                       │
        │                       │                       │
        │                       │                       │
        ▼                       ▼                       ▼
  Application uses config and secrets securely at runtime

Myth Busters - 4 Common Misconceptions

Quick: Do you think storing secrets as plain environment variables is safe in production? Commit to yes or no.

Common Belief: Storing secrets as environment variables is secure enough because they are not in the code.

Reality: Environment variables can be exposed in logs, error messages, or process listings, so secrets should be stored in dedicated secret management systems.

Why it matters: Relying on environment variables for secrets can lead to accidental leaks and security breaches.

Quick: Do you think secrets once set never need to be changed? Commit to yes or no.

Common Belief: Once a secret is created, it can be used forever without updates.

Reality: Secrets should be rotated regularly to reduce risk if compromised.

Why it matters: Not rotating secrets increases the window of opportunity for attackers if a secret is leaked.

Quick: Do you think applications automatically get updated secrets without redeployment? Commit to yes or no.

Common Belief: Applications always read the latest secret automatically without any changes.

Reality: Applications must be designed to fetch secrets dynamically; otherwise, they may use stale secrets baked into environment variables or config files.

Why it matters: Failing to fetch updated secrets can cause outages or security risks.

Quick: Do you think anyone with access to the cloud project can see all secrets? Commit to yes or no.

Common Belief: All team members with project access can view all secrets by default.

Reality: Access to secrets is controlled by fine-grained IAM permissions; not everyone can see all secrets.

Why it matters: Assuming open access can lead to improper permission granting and insider risks.

Expert Zone

1

Secrets should never be stored in source code repositories or container images to avoid accidental exposure.

2

Using workload identity and short-lived tokens to access secrets reduces risk compared to long-lived credentials.

3

Combining secret versioning with automated rotation pipelines improves security without downtime.

When NOT to use

Avoid using environment variables for highly sensitive secrets in production; instead, use managed secret stores like GCP Secret Manager. For local development, environment variables are fine but should be replaced with secrets in production. Also, do not store secrets in plaintext files or code repositories.

Production Patterns

In production, teams use GCP Secret Manager with IAM roles to control access, automate secret rotation, and fetch secrets at runtime via client libraries. Environment variables configure non-sensitive settings. CI/CD pipelines inject environment variables for build-time config but never secrets. Logging and monitoring exclude secret values.

Connections

Access Control and IAM

Builds-on

Understanding environment variables and secrets is incomplete without knowing how IAM controls who can access secrets securely.

Continuous Integration/Continuous Deployment (CI/CD)

Builds-on

CI/CD pipelines use environment variables and secrets to configure deployments securely and flexibly, linking secret management to automated workflows.

Human Memory and Cognitive Load

Analogy-based cross-domain

Just as humans separate sensitive information from everyday thoughts to avoid overload and mistakes, separating secrets from config reduces risk and complexity in systems.

Common Pitfalls

#1 Hardcoding secrets directly in application code.

Wrong approach:

const API_KEY = "my-secret-key"; // hardcoded secret

Correct approach:

const API_KEY = process.env.API_KEY; // read from environment variable or secret

Root cause:

Misunderstanding that code should never contain sensitive data to avoid leaks and ease updates.

#2 Storing secrets as plain environment variables in production.

Wrong approach:

export DB_PASSWORD="supersecret" # used directly as env var

Correct approach:

Use GCP Secret Manager to store DB_PASSWORD and fetch it securely at runtime.

Root cause:

Belief that environment variables are secure enough for secrets without additional protection.

#3 Logging secret values accidentally.

Wrong approach:

console.log("Database password is", process.env.DB_PASSWORD);

Correct approach:

Avoid logging secrets or mask them before logging.

Root cause:

Not realizing logs can be accessed by many and expose sensitive data.

Key Takeaways

Environment variables separate configuration from code, making applications flexible and easier to manage.

Secrets are sensitive data that require special protection beyond regular environment variables.

GCP Secret Manager securely stores and controls access to secrets with encryption and IAM policies.

Best practice is to use environment variables for non-sensitive config and Secret Manager for secrets, accessed dynamically at runtime.

Avoid hardcoding secrets, logging them, or storing them in code or images to prevent accidental exposure.