Secrets encryption at rest in Kubernetes - Deep Dive

Overview - Secrets encryption at rest

What is it?

Secrets encryption at rest means protecting sensitive data stored inside Kubernetes by converting it into a secret code when saved on disk. This ensures that even if someone accesses the storage directly, they cannot read the secret information without the key. It is a security feature that keeps passwords, tokens, and keys safe inside the cluster.

Why it matters

Without encryption at rest, anyone who gains access to the storage backend could see all sensitive data in plain text, risking leaks and attacks. Encrypting secrets protects your applications and users by making stolen data useless without the decryption key. This is crucial for compliance, trust, and preventing costly breaches.

Where it fits

Before learning this, you should understand Kubernetes basics, especially how Secrets work and how data is stored in etcd. After this, you can explore advanced Kubernetes security topics like RBAC, network policies, and external secret management tools.

Mental Model

Core Idea

Secrets encryption at rest transforms sensitive data into unreadable code when stored, unlocking it only when needed inside Kubernetes.

Think of it like...

It's like locking your important documents in a safe before putting them in a filing cabinet; even if someone opens the cabinet, they can't read the documents without the safe's key.

┌─────────────────────────────┐
│ Kubernetes Secret (plaintext)│
└─────────────┬───────────────┘
              │ Encrypt with key
              ▼
┌─────────────────────────────┐
│ Encrypted Secret in etcd     │
└─────────────┬───────────────┘
              │ Decrypt when accessed
              ▼
┌─────────────────────────────┐
│ Kubernetes Pod uses Secret   │
└─────────────────────────────┘

Build-Up - 6 Steps

1

Foundation

What are Kubernetes Secrets

Concept: Kubernetes Secrets store sensitive data like passwords and tokens separately from application code.

Kubernetes Secrets are special objects designed to hold confidential information. They keep data like API keys or passwords safe from being exposed in plain text inside your application manifests. Secrets are stored in etcd, the Kubernetes database, by default in plain text.

Result

You can create and use Secrets to keep sensitive data out of your code and configuration files.

Understanding what Secrets are and how they store data is the first step to realizing why encrypting them matters.

2

Foundation

What is encryption at rest

Concept: Encryption at rest means converting data into a secret code when saved on disk to prevent unauthorized reading.

Encryption at rest protects stored data by transforming it into an unreadable format using a key. Only someone with the key can convert it back to readable form. This protects data even if the storage device is stolen or accessed without permission.

Result

Data saved on disk is unreadable without the encryption key.

Knowing the basic idea of encryption at rest helps you understand how Kubernetes protects Secrets stored in etcd.

3

Intermediate

How Kubernetes encrypts Secrets at rest

🤔Before reading on: do you think Kubernetes encrypts Secrets automatically or requires manual setup? Commit to your answer.

Concept: Kubernetes can encrypt Secrets stored in etcd using configured encryption providers, but it requires explicit setup.

By default, Kubernetes stores Secrets in plain text in etcd. To enable encryption at rest, you configure an encryption provider in the API server's configuration. Kubernetes then encrypts Secrets before saving them and decrypts them when accessed by authorized users or pods.

Result

Secrets in etcd become encrypted, protecting them from direct access.

Knowing that encryption is not automatic but configurable helps you control and secure your cluster properly.

4

Intermediate

Encryption providers and key management

🤔Before reading on: do you think Kubernetes uses one fixed key or supports multiple keys for encryption? Commit to your answer.

Concept: Kubernetes supports multiple encryption providers and key rotation to manage encryption keys securely.

You can choose different encryption providers like AES-CBC, AES-GCM, or KMS (Key Management Service). Kubernetes supports key rotation by allowing multiple keys in the configuration, using the newest key for encryption and older keys for decryption. This helps keep encryption strong over time.

Result

You can securely manage encryption keys and rotate them without downtime.

Understanding key management and rotation is crucial for maintaining long-term security and compliance.

5

Advanced

Enabling encryption at rest in Kubernetes

🤔Before reading on: do you think enabling encryption requires restarting the cluster or just the API server? Commit to your answer.

Concept: Enabling encryption at rest involves creating an encryption configuration file and restarting the API server.

To enable encryption, create a config file specifying encryption providers and keys. Then, update the API server to use this config and restart it. Existing Secrets remain unencrypted until updated or re-saved. You can verify encryption by checking etcd data directly.

Result

Secrets stored after enabling encryption are encrypted at rest.

Knowing the exact steps and impact of enabling encryption helps avoid downtime and data loss.

6

Expert

Challenges and surprises in encryption at rest

🤔Before reading on: do you think all Kubernetes data is encrypted after enabling encryption at rest? Commit to your answer.

Concept: Encryption at rest only protects Secrets in etcd, not all cluster data, and has operational nuances.

Only Secrets are encrypted by this feature; other data like ConfigMaps remain unencrypted unless separately configured. Also, enabling encryption does not retroactively encrypt existing Secrets until they are updated. Misconfigurations can lead to data loss or inaccessible Secrets. Using external KMS providers adds complexity but improves security.

Result

You gain strong protection for Secrets but must manage limitations and operational risks carefully.

Understanding these nuances prevents false security assumptions and operational mistakes in production.

Under the Hood

When encryption at rest is enabled, the Kubernetes API server intercepts Secret objects before writing them to etcd. It uses the configured encryption provider and key to transform the Secret data into ciphertext. This ciphertext is stored in etcd instead of plain text. When a Secret is requested, the API server decrypts the ciphertext back into plain text using the correct key before returning it to the requester. The encryption keys are managed in-memory by the API server and can be rotated by updating the configuration. The etcd database itself remains unaware of encryption; it simply stores the encrypted blobs.

Why designed this way?

Kubernetes separates encryption responsibility to the API server to keep etcd simple and fast. This design allows flexible encryption methods and key management without modifying etcd. It also enables backward compatibility and gradual rollout of encryption. Alternatives like encrypting the entire etcd disk were less flexible and harder to manage securely. This approach balances security, performance, and operational complexity.

┌───────────────┐       ┌─────────────────────┐       ┌───────────────┐
│ User/Pod     │──────▶│ Kubernetes API Server│──────▶│ etcd Database │
│ requests     │       │ (encrypts/decrypts) │       │ (stores data) │
└───────────────┘       └─────────────────────┘       └───────────────┘
         ▲                        │                          ▲
         │                        │                          │
         │                        ▼                          │
         │               ┌─────────────────┐               │
         │               │ Encryption Keys │               │
         │               └─────────────────┘               │
         └──────────────────────────────────────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does enabling encryption at rest automatically encrypt all Kubernetes data? Commit yes or no.

Common Belief: Enabling encryption at rest encrypts all data in Kubernetes including ConfigMaps and PersistentVolumes.

Reality: Only Secrets are encrypted by Kubernetes encryption at rest; other data types remain unencrypted unless separately handled.

Why it matters: Assuming all data is encrypted can lead to accidental exposure of sensitive information stored outside Secrets.

Quick: Is encryption at rest enabled by default in Kubernetes? Commit yes or no.

Common Belief: Kubernetes encrypts Secrets at rest automatically without any configuration.

Reality: Encryption at rest is disabled by default and must be explicitly enabled and configured by the cluster administrator.

Why it matters: Relying on default settings can leave Secrets exposed in plain text, risking security breaches.

Quick: Does enabling encryption at rest encrypt existing Secrets immediately? Commit yes or no.

Common Belief: Once encryption is enabled, all existing Secrets in etcd become encrypted instantly.

Reality: Existing Secrets remain unencrypted until they are updated or re-created after encryption is enabled.

Why it matters: Believing otherwise can cause false confidence in security and delay necessary Secret rotations.

Quick: Can Kubernetes API server decrypt Secrets without the encryption key? Commit yes or no.

Common Belief: The API server can always decrypt Secrets regardless of key availability.

Reality: If the encryption key is lost or misconfigured, the API server cannot decrypt Secrets, causing data loss.

Why it matters: Mismanaging keys can lead to permanent loss of access to critical Secrets, impacting applications.

Expert Zone

1

Kubernetes supports multiple encryption providers in a prioritized list, allowing fallback decryption with older keys during rotation.

2

Using an external KMS provider improves security by separating key storage from the cluster but adds latency and complexity.

3

Encryption at rest protects data on disk but does not protect Secrets in memory or in transit; complementary security measures are needed.

When NOT to use

Encryption at rest is not a substitute for network security or access controls. For clusters with external secret management systems (like HashiCorp Vault), relying solely on Kubernetes encryption may be insufficient. Also, for very small or ephemeral clusters, the operational overhead might outweigh benefits.

Production Patterns

In production, teams combine encryption at rest with RBAC policies, audit logging, and external KMS integration. They automate key rotation and Secret updates using CI/CD pipelines. Monitoring etcd access and API server logs helps detect unauthorized attempts. Multi-tenant clusters often use namespaces and encryption to isolate sensitive data.

Connections

Disk encryption

Related security layer

Understanding disk encryption helps grasp why encrypting Secrets at the application layer adds an extra security boundary.

Access control (RBAC)

Complementary security mechanism

Knowing RBAC helps understand how encryption at rest fits into a layered defense by limiting who can request decrypted Secrets.

Cryptography key rotation

Builds on key management principles

Understanding key rotation in cryptography clarifies how Kubernetes manages multiple keys to maintain security without downtime.

Common Pitfalls

#1 Assuming encryption at rest is enabled by default and not configuring it.

Wrong approach:

kubectl get secrets
# Secrets appear in plain text in etcd, no encryption config applied

Correct approach:

Create encryption-config.yaml with providers and keys
Update API server to use --encryption-provider-config=encryption-config.yaml
Restart API server
# Secrets now encrypted in etcd

Root cause:

Misunderstanding default Kubernetes behavior and missing explicit encryption setup.

#2 Not rotating encryption keys leading to stale keys and potential security risks.

Wrong approach:

Using a single static key in encryption-config.yaml indefinitely without updates.

Correct approach:

Add new keys to encryption-config.yaml at the top of the providers list
Rotate keys regularly
Remove old keys after re-encryption

Root cause:

Lack of awareness about key rotation importance and Kubernetes support for multiple keys.

#3 Expecting existing Secrets to be encrypted immediately after enabling encryption.

Wrong approach:

Enable encryption and assume all Secrets in etcd are now encrypted without re-saving them.

Correct approach:

After enabling encryption, update or recreate existing Secrets to trigger encryption on save.

Root cause:

Not knowing that encryption applies only on write, not retroactively.

Key Takeaways

Kubernetes Secrets store sensitive data but are plain text in etcd by default, risking exposure.

Encryption at rest protects Secrets by encrypting them before saving to etcd, requiring explicit setup.

Proper key management and rotation are essential to maintain encryption security over time.

Encryption at rest only protects data on disk; other security layers like RBAC and network policies are also needed.

Misunderstanding encryption behavior can lead to false security and operational issues, so careful configuration and monitoring are critical.

Practice

1. What is the main purpose of enabling Secrets encryption at rest in Kubernetes?

easy

A. To protect sensitive data stored in etcd from unauthorized access

B. To speed up the retrieval of Secrets from the API server

C. To allow Secrets to be shared publicly across namespaces

D. To automatically rotate Secrets without manual intervention

Solution

1. Step 1: Understand what Secrets encryption at rest means

   It means encrypting sensitive data stored on disk, specifically in etcd, to prevent unauthorized access if someone gains access to the storage.

2. Step 2: Identify the main goal of this encryption

   The goal is to protect sensitive data like passwords or tokens stored in etcd, not to speed up access or share Secrets publicly.

3. Final Answer:

   To protect sensitive data stored in etcd from unauthorized access -> Option A

4. Quick Check:

   Secrets encryption = protect data at rest [OK]

Hint: Encryption at rest means protecting stored data, not speeding access [OK]

Common Mistakes:

• Confusing encryption at rest with encryption in transit
• Thinking encryption shares Secrets publicly
• Assuming encryption automatically rotates Secrets

2. Which of the following is the correct way to enable Secrets encryption at rest in Kubernetes EncryptionConfiguration file?

easy

A. apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
- aescbc:
keys:
- name: key1
secret:
- identity: {}

B. apiVersion: v1
kind: Secret
metadata:
name: encryption-config
stringData:
key:

C. apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- secrets
providers:
- identity: {}
- aescbc:
keys:
- name: key1
secret:

D. apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfig
resources:
- secrets
- aescbc:
keys:
- name: key1
secret:

Solution

1. Step 1: Review the correct structure of EncryptionConfiguration

   The file must have apiVersion, kind, and a resources list with nested resources and providers. The providers list includes encryption methods like aescbc and identity.

2. Step 2: Compare options for correct YAML syntax and structure

   apiVersion: apiserver.config.k8s.io/v1 kind: EncryptionConfiguration resources: - secrets providers: - identity: {} - aescbc: keys: - name: key1 secret: correctly nests resources and providers, uses aescbc with keys, and includes identity as fallback. apiVersion: apiserver.config.k8s.io/v1 kind: EncryptionConfiguration resources: - resources: - secrets providers: - aescbc: keys: - name: key1 secret: - identity: {} incorrectly nests 'resources' under 'resources'. Others have syntax errors or wrong kind names.

3. Final Answer:

   apiVersion: apiserver.config.k8s.io/v1 kind: EncryptionConfiguration resources: - secrets providers: - identity: {} - aescbc: keys: - name: key1 secret: -> Option C

4. Quick Check:

   Correct YAML structure = apiVersion: apiserver.config.k8s.io/v1 kind: EncryptionConfiguration resources: - secrets providers: - identity: {} - aescbc: keys: - name: key1 secret: [OK]

Hint: Look for 'resources' as a list of resource names and 'providers' as a list of encryption methods [OK]

Common Mistakes:

• Using wrong kind name like EncryptionConfig instead of EncryptionConfiguration
• Incorrect YAML indentation or missing nested keys
• Placing keys outside the providers list
• Nesting 'resources' under 'resources' incorrectly

3. Given this snippet from a Kubernetes API server log after enabling Secrets encryption at rest:

"Encryption provider aescbc is enabled for resource secrets"
"Using key named key1 for encryption"
"Secrets stored in etcd are now encrypted"

What is the expected effect when retrieving a Secret via kubectl get secret?

medium

A. The Secret data is shown as base64-encoded encrypted strings

B. The Secret data is automatically decrypted and shown in base64-encoded plain text

C. The Secret cannot be retrieved until manual decryption is done

D. The Secret is deleted from etcd after retrieval

Solution

1. Step 1: Understand encryption at rest vs API response

   Encryption at rest means data is encrypted in storage (etcd), but the API server decrypts it before sending to clients.

2. Step 2: Determine what kubectl get secret shows

   kubectl shows the decrypted Secret data in base64-encoded form, which is normal for Secrets, not encrypted ciphertext.

3. Final Answer:

   The Secret data is automatically decrypted and shown in base64-encoded plain text -> Option B

4. Quick Check:

   Encryption at rest decrypts before API response [OK]

Hint: Encryption at rest is transparent to kubectl output [OK]

Common Mistakes:

• Thinking Secrets remain encrypted when retrieved
• Confusing base64 encoding with encryption
• Assuming manual decryption is needed

4. You configured Secrets encryption at rest but notice that Secrets are still stored unencrypted in etcd. What is the most likely cause?

medium

A. The Secrets were created before enabling encryption and never updated

B. The encryption key is too short and rejected silently

C. The etcd cluster does not support encryption

D. The API server was not restarted after applying the encryption config

Solution

1. Step 1: Recall how encryption config is applied

   The API server must be restarted to load the new encryption configuration and apply encryption to new Secrets.

2. Step 2: Identify why Secrets remain unencrypted

   If the API server is not restarted, it continues to store Secrets unencrypted despite config changes.

3. Final Answer:

   The API server was not restarted after applying the encryption config -> Option D

4. Quick Check:

   Restart API server to apply encryption config [OK]

Hint: Always restart API server after changing encryption config [OK]

Common Mistakes:

• Assuming existing Secrets auto-encrypt without update
• Believing etcd cannot support encryption
• Ignoring the need to restart API server

5. You want to rotate the encryption key used for Secrets encryption at rest without downtime. Which approach correctly achieves this?

hard

A. Add the new key as the first provider in the encryption config, keep the old key second, then restart the API server

B. Replace the old key with the new key in the config and restart the API server immediately

C. Delete all Secrets, update the key, then recreate Secrets encrypted with the new key

D. Update the key in etcd directly without changing the API server config

Solution

1. Step 1: Understand key rotation in encryption config

   To rotate keys safely, add the new key first so new Secrets encrypt with it, and keep the old key to decrypt existing Secrets.

2. Step 2: Apply config and restart API server

   Restarting the API server loads the new config. Existing Secrets remain decryptable with the old key, allowing smooth rotation.

3. Final Answer:

   Add the new key as the first provider in the encryption config, keep the old key second, then restart the API server -> Option A

4. Quick Check:

   New key first, old key second, restart API server [OK]

Hint: New key first, old key second in config for smooth rotation [OK]

Common Mistakes:

• Replacing old key immediately causing decryption failures
• Deleting Secrets instead of rotating keys
• Modifying etcd data directly risking corruption