Creating Secrets in Kubernetes - Mechanics & Internals

Overview - Creating Secrets

What is it?

Creating Secrets in Kubernetes means securely storing sensitive information like passwords, tokens, or keys. Instead of putting this data directly in your app code or configuration files, Secrets keep it safe and separate. Kubernetes manages these Secrets and makes them available to your applications when needed. This helps protect your sensitive data from accidental exposure.

Why it matters

Without Secrets, sensitive data would be stored in plain text inside configuration files or container images, making it easy to leak or misuse. This can lead to security breaches, data loss, or unauthorized access. Secrets solve this by providing a secure, centralized way to manage sensitive information, reducing risks and making your applications safer.

Where it fits

Before learning about Creating Secrets, you should understand basic Kubernetes concepts like Pods, ConfigMaps, and YAML manifests. After mastering Secrets, you can explore advanced topics like Secret encryption, RBAC (Role-Based Access Control), and integrating Secrets with external vaults or cloud providers.

Mental Model

Core Idea

Kubernetes Secrets are secure containers that hold sensitive data separately from application code and configuration, providing controlled access to protect secrets from exposure.

Think of it like...

Imagine a locked safe in an office where only authorized people have the key. The safe holds important documents (secrets) separate from the desk (application code) so they don’t get lost or seen by everyone.

┌───────────────┐       ┌───────────────┐
│ Application   │       │ Kubernetes    │
│ Code & Config │──────▶│ Secret Store  │
└───────────────┘       └───────────────┘
         ▲                      │
         │                      ▼
   Access Secret           Secure Storage
   via Mount or Env

Build-Up - 7 Steps

1

Foundation

What Are Kubernetes Secrets

Concept: Introduce the basic idea of Secrets as a Kubernetes object to store sensitive data.

Kubernetes Secrets are special objects designed to hold sensitive information like passwords, API keys, or certificates. Unlike ConfigMaps, Secrets are intended to keep data confidential and are stored in base64 encoded form. You create a Secret object and then reference it in your Pods to provide secure access to this data.

Result

You understand that Secrets are Kubernetes objects for sensitive data storage, separate from normal configuration.

Understanding that Secrets are distinct Kubernetes objects helps you separate sensitive data from regular configuration, improving security.

2

Foundation

Creating a Basic Secret Manually

Concept: Learn how to create a Secret using kubectl with literal values.

Use the command: kubectl create secret generic my-secret --from-literal=username=admin --from-literal=password=secret123 This creates a Secret named 'my-secret' with two keys: username and password. The data is stored encoded but not encrypted by default.

Result

A Secret object named 'my-secret' is created in Kubernetes, holding the username and password securely.

Knowing how to create Secrets manually is the first step to managing sensitive data securely in Kubernetes.

3

Intermediate

Creating Secrets from Files

Concept: Learn to create Secrets by loading data from files instead of literals.

You can create a Secret from files containing sensitive data. For example: kubectl create secret generic tls-secret --from-file=tls.crt=path/to/cert.crt --from-file=tls.key=path/to/key.key This method is useful for certificates or large secrets stored in files.

Result

A Secret named 'tls-secret' is created containing the certificate and key data from files.

Loading Secrets from files allows you to manage complex or large sensitive data easily and keeps your commands clean.

4

Intermediate

Referencing Secrets in Pods

🤔Before reading on: do you think Secrets can be accessed only as environment variables or also as files inside containers? Commit to your answer.

Concept: Understand how to use Secrets inside Pods either as environment variables or mounted files.

You can reference Secrets in Pod specs in two ways: 1. Environment variables: env: - name: USERNAME valueFrom: secretKeyRef: name: my-secret key: username 2. Mounted as files: volumes: - name: secret-volume secret: secretName: my-secret containers: - name: app volumeMounts: - name: secret-volume mountPath: /etc/secret This flexibility lets your app read secrets as env vars or files.

Result

Pods can securely access Secrets either as environment variables or files, depending on app needs.

Knowing both access methods lets you choose the best way to inject secrets based on your application's design.

5

Intermediate

Base64 Encoding Explained

🤔Before reading on: do you think Kubernetes encrypts Secrets by default or just encodes them? Commit to your answer.

Concept: Explain that Kubernetes stores Secrets base64 encoded, not encrypted by default.

When you create a Secret, Kubernetes encodes the data using base64. This means the data is transformed into a text format but is not encrypted or hidden securely. Anyone with access to the Secret object can decode it easily. To protect Secrets, you need to enable encryption at rest or use external vaults.

Result

You understand that base64 encoding is not secure encryption and additional steps are needed for real protection.

Understanding base64 encoding prevents the false belief that Secrets are secure by default, highlighting the need for encryption.

6

Advanced

Encrypting Secrets at Rest

🤔Before reading on: do you think Kubernetes encrypts Secrets on disk by default? Commit to your answer.

Concept: Learn how to enable encryption of Secrets stored on the Kubernetes cluster's disk.

By default, Secrets are stored in etcd in base64 encoded form. To protect them, you can enable encryption at rest by configuring the Kubernetes API server with an encryption provider. This encrypts Secrets before saving to etcd, preventing unauthorized access if etcd is compromised. This requires editing the encryption configuration file and restarting the API server.

Result

Secrets are stored encrypted on disk, improving cluster security against data leaks.

Knowing how to enable encryption at rest is critical for production security and compliance.

7

Expert

Using External Secret Management Tools

🤔Before reading on: do you think Kubernetes Secrets alone are enough for all security needs? Commit to your answer.

Concept: Explore integrating Kubernetes with external secret managers like HashiCorp Vault or cloud provider vaults for advanced security.

Kubernetes Secrets have limitations like lack of automatic rotation and audit. External secret managers provide features like dynamic secrets, automatic rotation, detailed audit logs, and fine-grained access control. You can integrate these tools with Kubernetes using controllers or CSI drivers that sync secrets securely into your cluster, improving security and operational control.

Result

Your Kubernetes cluster uses external vaults to manage secrets dynamically and securely beyond native capabilities.

Understanding external secret management integration elevates your security posture and operational flexibility in real-world environments.

Under the Hood

Kubernetes Secrets are stored as API objects in etcd, the cluster's key-value store. When you create a Secret, Kubernetes encodes the data in base64 and saves it in etcd. The API server serves these Secrets to authorized Pods via environment variables or mounted volumes. By default, etcd stores Secrets unencrypted, so anyone with etcd access can decode them. Encryption at rest can be enabled to encrypt Secrets before saving. Access to Secrets is controlled by Kubernetes RBAC policies.

Why designed this way?

Secrets were designed to separate sensitive data from application code and configuration for security and manageability. Base64 encoding was chosen for simplicity and compatibility with YAML and JSON formats. Encryption at rest was left optional to allow flexibility in different environments. This design balances ease of use with security, allowing clusters to start simple and add protections as needed.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│ User creates  │──────▶│ Kubernetes    │──────▶│ etcd Store    │
│ Secret object │       │ API Server    │       │ (base64 data) │
└───────────────┘       └───────────────┘       └───────────────┘
         ▲                      │                      │
         │                      ▼                      ▼
   Pod requests           API Server             Optional Encryption
   Secret access          serves Secret          encrypts data before
                          to Pod                 saving to etcd

Myth Busters - 4 Common Misconceptions

Quick: Do you think Kubernetes Secrets are encrypted by default? Commit to yes or no.

Common Belief: Kubernetes Secrets are encrypted and fully secure by default.

Reality: Secrets are only base64 encoded by default, which is not encryption and can be easily decoded.

Why it matters: Believing Secrets are encrypted can lead to complacency and accidental exposure of sensitive data.

Quick: Can you store large binary files directly in Kubernetes Secrets? Commit to yes or no.

Common Belief: You can store any size of binary data directly in Secrets without issues.

Reality: Secrets have size limits (usually 1MB) and storing large binaries is not recommended; ConfigMaps or external storage are better.

Why it matters: Trying to store large files in Secrets can cause errors or performance problems in the cluster.

Quick: Do you think mounting Secrets as files is less secure than environment variables? Commit to yes or no.

Common Belief: Mounting Secrets as files is less secure than using environment variables.

Reality: Both methods expose Secrets in memory; security depends on access controls, not the method of injection.

Why it matters: Misunderstanding this can lead to poor design choices or false sense of security.

Quick: Do you think deleting a Secret immediately removes it from all running Pods? Commit to yes or no.

Common Belief: Deleting a Secret instantly removes it from all Pods using it.

Reality: Pods keep the Secret data in memory or mounted volumes until restarted; deletion does not revoke access immediately.

Why it matters: Assuming immediate removal can cause security gaps if Secrets are rotated or revoked.

Expert Zone

1

Kubernetes Secrets are stored in etcd and protected by RBAC, but cluster admins with etcd access can read all Secrets, so trust boundaries matter.

2

Secret data is mounted as tmpfs (in-memory filesystem) when used as volumes, reducing disk exposure but still accessible to processes in the container.

3

When multiple Secrets are mounted in the same volume, file name collisions can occur, requiring careful naming conventions.

When NOT to use

Avoid using Kubernetes Secrets for extremely sensitive data requiring hardware security modules or automatic rotation; use dedicated external vaults like HashiCorp Vault or cloud KMS instead.

Production Patterns

In production, teams often combine Kubernetes Secrets with encryption at rest enabled, RBAC policies restricting access, and external secret managers for dynamic secrets and audit logging. Automated pipelines inject Secrets securely during deployment, and rotation policies ensure credentials are regularly updated.

Connections

Role-Based Access Control (RBAC)

RBAC controls who can create, read, or modify Secrets in Kubernetes.

Understanding RBAC is essential to secure Secrets by limiting access only to authorized users and services.

Encryption at Rest

Encryption at rest protects Secrets stored in etcd from unauthorized disk access.

Knowing encryption at rest complements Secrets by securing data beyond just access controls.

Physical Safe Security

Like a physical safe protects valuables, Kubernetes Secrets protect sensitive data from unauthorized access.

This cross-domain connection highlights the importance of layered security: physical safes and digital Secrets both rely on controlled access and protection.

Common Pitfalls

#1 Storing sensitive data directly in Pod specs or ConfigMaps.

Wrong approach:

apiVersion: v1
kind: Pod
metadata:
name: mypod
spec:
containers:
- name: app
image: myapp
env:
- name: PASSWORD
value: "supersecret"

Correct approach:

apiVersion: v1
kind: Pod
metadata:
name: mypod
spec:
containers:
- name: app
image: myapp
env:
- name: PASSWORD
valueFrom:
secretKeyRef:
name: my-secret
key: password

Root cause:

Misunderstanding that environment variables in Pod specs are visible in plain text and not secure for sensitive data.

#2 Assuming base64 encoding encrypts Secrets.

Wrong approach:

kubectl create secret generic my-secret --from-literal=password=secret123
# Believing this is encrypted and secure

Correct approach:

Enable encryption at rest in Kubernetes API server configuration to truly encrypt Secrets stored in etcd.

Root cause:

Confusing base64 encoding (a text format) with encryption leads to false security assumptions.

#3 Not restricting access to Secrets with RBAC.

Wrong approach:

No RBAC rules set, allowing all users to get Secrets:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: open-access
subjects:
- kind: User
name: '*'
roleRef:
kind: ClusterRole
name: cluster-admin
apiGroup: rbac.authorization.k8s.io

Correct approach:

Define strict RBAC roles limiting Secret access only to necessary service accounts or users.

Root cause:

Lack of RBAC knowledge or oversight leads to over-permissioned access and security risks.

Key Takeaways

Kubernetes Secrets store sensitive data separately from application code to improve security.

Secrets are base64 encoded by default, not encrypted, so additional encryption at rest is needed for real protection.

You can create Secrets from literals or files and inject them into Pods as environment variables or mounted files.

Access to Secrets must be controlled with RBAC to prevent unauthorized reading or modification.

For advanced security, integrate Kubernetes with external secret managers that provide rotation, audit, and dynamic secrets.