Secrets encryption at rest in Kubernetes - Mini Project: Build & Apply

Secrets Encryption at Rest in Kubernetes

📖 Scenario: You are managing a Kubernetes cluster that stores sensitive information like passwords and API keys in Secrets. To protect this data, you want to enable encryption at rest so that the secrets are stored encrypted on disk.This project will guide you step-by-step to configure Kubernetes to encrypt secrets at rest using a simple encryption key.

🎯 Goal: Enable encryption at rest for Kubernetes Secrets by creating an encryption configuration file, updating the API server to use it, and verifying that secrets are stored encrypted.

📋 What You'll Learn

Create an encryption configuration file with a specific encryption key

Add a config variable to specify the path to the encryption config file

Modify the Kubernetes API server manifest to use the encryption config file

Verify that secrets are encrypted at rest by checking the stored data

💡 Why This Matters

🌍 Real World

Encrypting secrets at rest protects sensitive data stored in Kubernetes clusters from unauthorized access if the storage is compromised.

💼 Career

Understanding how to enable and verify secrets encryption is a key skill for Kubernetes administrators and DevOps engineers to secure cluster data.

1

Create the encryption configuration file

Create a YAML file called encryption-config.yaml with the following content exactly:

apiVersion: apiserver.config.k8s.io/v1
encryptionConfiguration:
  resources:
  - resources:
    - secrets
    providers:
    - aescbc:
        keys:
        - name: key1
          secret: c2VjcmV0a2V5MTIzNDU2Nzg5MDEyMzQ1Ng==
    - identity: {}

Kubernetes

# Create the file encryption-config.yaml with the exact content
# Your code here

Hint

The encryption key is base64 encoded. Make sure the indentation and keys match exactly.

2

Set the encryption config file path variable

Create a variable called ENCRYPTION_CONFIG_PATH and set it to the string "/etc/kubernetes/encryption-config.yaml".

Kubernetes

apiVersion: apiserver.config.k8s.io/v1
encryptionConfiguration:
  resources:
  - resources:
    - secrets
    providers:
    - aescbc:
        keys:
        - name: key1
          secret: c2VjcmV0a2V5MTIzNDU2Nzg5MDEyMzQ1Ng==
    - identity: {}

# Create the variable ENCRYPTION_CONFIG_PATH with the config file path
# Your code here

Hint

Use an uppercase variable name and assign the exact string path.

3

Update the API server manifest to use encryption config

Add the following flag to the Kubernetes API server manifest command line:
--encryption-provider-config=${ENCRYPTION_CONFIG_PATH}
Use the variable ENCRYPTION_CONFIG_PATH you created in Step 2.

Kubernetes

apiVersion: apiserver.config.k8s.io/v1
encryptionConfiguration:
  resources:
  - resources:
    - secrets
    providers:
    - aescbc:
        keys:
        - name: key1
          secret: c2VjcmV0a2V5MTIzNDU2Nzg5MDEyMzQ1Ng==
    - identity: {}

ENCRYPTION_CONFIG_PATH = "/etc/kubernetes/encryption-config.yaml"

# Add the flag --encryption-provider-config=${ENCRYPTION_CONFIG_PATH} to the API server command
# Your code here

Hint

Simulate the API server command as a string variable including the flag with the variable.

4

Verify secrets are encrypted at rest

Print the exact string Secrets are now encrypted at rest using /etc/kubernetes/encryption-config.yaml to confirm the setup.

Kubernetes

apiVersion: apiserver.config.k8s.io/v1
encryptionConfiguration:
  resources:
  - resources:
    - secrets
    providers:
    - aescbc:
        keys:
        - name: key1
          secret: c2VjcmV0a2V5MTIzNDU2Nzg5MDEyMzQ1Ng==
    - identity: {}

ENCRYPTION_CONFIG_PATH = "/etc/kubernetes/encryption-config.yaml"

kube_apiserver_command = "kube-apiserver --encryption-provider-config=${ENCRYPTION_CONFIG_PATH}"

# Print the confirmation message below
# Your code here

Hint

Use a print statement with the exact message including the file path.

Practice

1. What is the main purpose of enabling Secrets encryption at rest in Kubernetes?

easy

A. To protect sensitive data stored in etcd from unauthorized access

B. To speed up the retrieval of Secrets from the API server

C. To allow Secrets to be shared publicly across namespaces

D. To automatically rotate Secrets without manual intervention

Solution

1. Step 1: Understand what Secrets encryption at rest means

   It means encrypting sensitive data stored on disk, specifically in etcd, to prevent unauthorized access if someone gains access to the storage.

2. Step 2: Identify the main goal of this encryption

   The goal is to protect sensitive data like passwords or tokens stored in etcd, not to speed up access or share Secrets publicly.

3. Final Answer:

   To protect sensitive data stored in etcd from unauthorized access -> Option A

4. Quick Check:

   Secrets encryption = protect data at rest [OK]

Hint: Encryption at rest means protecting stored data, not speeding access [OK]

Common Mistakes:

• Confusing encryption at rest with encryption in transit
• Thinking encryption shares Secrets publicly
• Assuming encryption automatically rotates Secrets

2. Which of the following is the correct way to enable Secrets encryption at rest in Kubernetes EncryptionConfiguration file?

easy

A. apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
- aescbc:
keys:
- name: key1
secret:
- identity: {}

B. apiVersion: v1
kind: Secret
metadata:
name: encryption-config
stringData:
key:

C. apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- secrets
providers:
- identity: {}
- aescbc:
keys:
- name: key1
secret:

D. apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfig
resources:
- secrets
- aescbc:
keys:
- name: key1
secret:

Solution

1. Step 1: Review the correct structure of EncryptionConfiguration

   The file must have apiVersion, kind, and a resources list with nested resources and providers. The providers list includes encryption methods like aescbc and identity.

2. Step 2: Compare options for correct YAML syntax and structure

   apiVersion: apiserver.config.k8s.io/v1 kind: EncryptionConfiguration resources: - secrets providers: - identity: {} - aescbc: keys: - name: key1 secret: correctly nests resources and providers, uses aescbc with keys, and includes identity as fallback. apiVersion: apiserver.config.k8s.io/v1 kind: EncryptionConfiguration resources: - resources: - secrets providers: - aescbc: keys: - name: key1 secret: - identity: {} incorrectly nests 'resources' under 'resources'. Others have syntax errors or wrong kind names.

3. Final Answer:

   apiVersion: apiserver.config.k8s.io/v1 kind: EncryptionConfiguration resources: - secrets providers: - identity: {} - aescbc: keys: - name: key1 secret: -> Option C

4. Quick Check:

   Correct YAML structure = apiVersion: apiserver.config.k8s.io/v1 kind: EncryptionConfiguration resources: - secrets providers: - identity: {} - aescbc: keys: - name: key1 secret: [OK]

Hint: Look for 'resources' as a list of resource names and 'providers' as a list of encryption methods [OK]

Common Mistakes:

• Using wrong kind name like EncryptionConfig instead of EncryptionConfiguration
• Incorrect YAML indentation or missing nested keys
• Placing keys outside the providers list
• Nesting 'resources' under 'resources' incorrectly

3. Given this snippet from a Kubernetes API server log after enabling Secrets encryption at rest:

"Encryption provider aescbc is enabled for resource secrets"
"Using key named key1 for encryption"
"Secrets stored in etcd are now encrypted"

What is the expected effect when retrieving a Secret via kubectl get secret?

medium

A. The Secret data is shown as base64-encoded encrypted strings

B. The Secret data is automatically decrypted and shown in base64-encoded plain text

C. The Secret cannot be retrieved until manual decryption is done

D. The Secret is deleted from etcd after retrieval

Solution

1. Step 1: Understand encryption at rest vs API response

   Encryption at rest means data is encrypted in storage (etcd), but the API server decrypts it before sending to clients.

2. Step 2: Determine what kubectl get secret shows

   kubectl shows the decrypted Secret data in base64-encoded form, which is normal for Secrets, not encrypted ciphertext.

3. Final Answer:

   The Secret data is automatically decrypted and shown in base64-encoded plain text -> Option B

4. Quick Check:

   Encryption at rest decrypts before API response [OK]

Hint: Encryption at rest is transparent to kubectl output [OK]

Common Mistakes:

• Thinking Secrets remain encrypted when retrieved
• Confusing base64 encoding with encryption
• Assuming manual decryption is needed

4. You configured Secrets encryption at rest but notice that Secrets are still stored unencrypted in etcd. What is the most likely cause?

medium

A. The Secrets were created before enabling encryption and never updated

B. The encryption key is too short and rejected silently

C. The etcd cluster does not support encryption

D. The API server was not restarted after applying the encryption config

Solution

1. Step 1: Recall how encryption config is applied

   The API server must be restarted to load the new encryption configuration and apply encryption to new Secrets.

2. Step 2: Identify why Secrets remain unencrypted

   If the API server is not restarted, it continues to store Secrets unencrypted despite config changes.

3. Final Answer:

   The API server was not restarted after applying the encryption config -> Option D

4. Quick Check:

   Restart API server to apply encryption config [OK]

Hint: Always restart API server after changing encryption config [OK]

Common Mistakes:

• Assuming existing Secrets auto-encrypt without update
• Believing etcd cannot support encryption
• Ignoring the need to restart API server

5. You want to rotate the encryption key used for Secrets encryption at rest without downtime. Which approach correctly achieves this?

hard

A. Add the new key as the first provider in the encryption config, keep the old key second, then restart the API server

B. Replace the old key with the new key in the config and restart the API server immediately

C. Delete all Secrets, update the key, then recreate Secrets encrypted with the new key

D. Update the key in etcd directly without changing the API server config

Solution

1. Step 1: Understand key rotation in encryption config

   To rotate keys safely, add the new key first so new Secrets encrypt with it, and keep the old key to decrypt existing Secrets.

2. Step 2: Apply config and restart API server

   Restarting the API server loads the new config. Existing Secrets remain decryptable with the old key, allowing smooth rotation.

3. Final Answer:

   Add the new key as the first provider in the encryption config, keep the old key second, then restart the API server -> Option A

4. Quick Check:

   New key first, old key second, restart API server [OK]

Hint: New key first, old key second in config for smooth rotation [OK]

Common Mistakes:

• Replacing old key immediately causing decryption failures
• Deleting Secrets instead of rotating keys
• Modifying etcd data directly risking corruption