Secrets encryption at rest in Kubernetes - Commands & Configuration

Introduction

When you store sensitive information like passwords or keys in Kubernetes, you want to keep them safe even if someone accesses the storage directly. Secrets encryption at rest means the data is stored encrypted on disk, so it stays protected.

When you want to protect sensitive data like API keys or passwords stored in Kubernetes Secrets from being read directly from disk.

When your Kubernetes cluster is running in a shared environment and you want to add an extra layer of security for stored secrets.

When compliance rules require encryption of sensitive data stored on persistent storage.

When you want to prevent attackers who gain access to etcd storage from reading secrets in plain text.

When you want to secure secrets without changing how your applications access them.

Config File - encryption-config.yaml

encryption-config.yaml

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: c2VjcmV0a2V5MTIzNDU2Nzg5MDEyMzQ1Njc4OTA=
      - identity: {}

This file tells Kubernetes to encrypt Secrets using AES-CBC with the provided key named key1. The secret is a base64-encoded 32-byte key. The identity provider means if encryption fails, data is stored as plain text (fallback).

The resources section specifies that only Secrets are encrypted.

Commands

Check existing secrets in the default namespace before enabling encryption to see current data.

Terminal

kubectl get secrets -n default

Expected Output

NAME                  TYPE                                  DATA   AGE

my-secret             Opaque                                1      10m

Place the encryption configuration file on the Kubernetes master node in the correct directory for the API server to use.

Terminal

sudo mv /path/to/encryption-config.yaml /etc/kubernetes/encryption-config.yaml

Expected Output

No output (command runs silently)

Add the environment variable to the kube-apiserver service to tell it where to find the encryption config file.

Terminal

sudo sed -i '/kube-apiserver.service/a Environment="KUBE_ENCRYPTION_PROVIDER_CONFIG=/etc/kubernetes/encryption-config.yaml"' /etc/systemd/system/kube-apiserver.service

Expected Output

No output (command runs silently)

Reload systemd to apply changes and restart the API server so it uses the new encryption settings.

Terminal

sudo systemctl daemon-reload && sudo systemctl restart kube-apiserver

Expected Output

No output (command runs silently)

Verify secrets are still accessible after enabling encryption at rest.

Terminal

kubectl get secrets -n default

Expected Output

NAME                  TYPE                                  DATA   AGE

my-secret             Opaque                                1      12m

Check the raw secret data stored in etcd to confirm it is encrypted (will show unreadable data).

Terminal

ETCDCTL_API=3 etcdctl get /registry/secrets/default/my-secret --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key

Expected Output

k8s:enc:aescbc:v1:key1

x12x34x56x78x90xabxcdxef...

→

--endpoints - Specify etcd server address

→

--cacert - CA certificate for etcd TLS

→

--cert - Client certificate for etcd TLS

→

--key - Client key for etcd TLS

Key Concept

If you remember nothing else from this pattern, remember: enabling encryption at rest protects Kubernetes Secrets stored in etcd from being read in plain text.

Common Mistakes

Not restarting the kube-apiserver after adding the encryption config.

The API server won't use the new encryption settings until it restarts, so secrets remain unencrypted.

Always reload systemd and restart kube-apiserver after changing encryption configuration.

Using a base64 key that is not 32 bytes long for AES-CBC encryption.

The encryption key must be exactly 32 bytes (256 bits) base64-encoded, or encryption will fail.

Generate a proper 32-byte base64 key using a secure method like openssl.

Not backing up existing secrets before enabling encryption.

If something goes wrong, you could lose access to secrets or corrupt data.

Always backup etcd data and secrets before enabling encryption at rest.

Summary

Create an encryption configuration file specifying AES-CBC encryption for Kubernetes Secrets.

Configure the kube-apiserver to use this encryption config and restart it to apply changes.

Verify secrets remain accessible and confirm data is encrypted in etcd storage.