etcd backup and recovery in Kubernetes - Mini Project: Build & Apply

etcd Backup and Recovery in Kubernetes

📖 Scenario: You are managing a Kubernetes cluster. The cluster's key-value store, etcd, holds all the important data about your cluster state. To keep your cluster safe, you want to create a backup of the etcd data and learn how to restore it if something goes wrong.

🎯 Goal: Learn how to create a backup of the etcd data using the etcdctl command and how to restore the cluster from that backup.

📋 What You'll Learn

Access to a Kubernetes control plane node

etcdctl installed and configured with correct environment variables

Basic command line knowledge

💡 Why This Matters

🌍 Real World

etcd stores all Kubernetes cluster data. Backing it up regularly protects your cluster from data loss due to failures or mistakes.

💼 Career

Kubernetes administrators and DevOps engineers must know how to backup and restore etcd to maintain cluster reliability and recover from disasters.

1

Set up environment variables for etcdctl

Set the environment variables ETCDCTL_API=3, ETCDCTL_ENDPOINTS=https://127.0.0.1:2379, ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt, ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt, and ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key to configure etcdctl for secure communication with the etcd server.

Kubernetes

# Set environment variables for etcdctl
# Your code here

Hint

Use export to set environment variables in the shell.

2

Create a snapshot backup of etcd data

Use the etcdctl snapshot save command to create a backup file named backup.db in the current directory.

Kubernetes

export ETCDCTL_API=3
export ETCDCTL_ENDPOINTS=https://127.0.0.1:2379
export ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt
export ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt
export ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key
# Create etcd snapshot backup named backup.db
# Your code here

Hint

Use etcdctl snapshot save backup.db to save the snapshot.

3

Restore etcd data from the snapshot

Use the etcdctl snapshot restore command with the snapshot file backup.db and specify the restore directory as /var/lib/etcd-from-backup.

Kubernetes

export ETCDCTL_API=3
export ETCDCTL_ENDPOINTS=https://127.0.0.1:2379
export ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt
export ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt
export ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key
etcdctl snapshot save backup.db
# Restore etcd data from backup.db to /var/lib/etcd-from-backup
# Your code here

Hint

Use etcdctl snapshot restore backup.db --data-dir /var/lib/etcd-from-backup to restore.

4

Verify the snapshot backup file exists

Use the ls command to list the file backup.db in the current directory and print the output.

Kubernetes

export ETCDCTL_API=3
export ETCDCTL_ENDPOINTS=https://127.0.0.1:2379
export ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt
export ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt
export ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key
etcdctl snapshot save backup.db
etcdctl snapshot restore backup.db --data-dir /var/lib/etcd-from-backup
# List the backup.db file and print the output
# Your code here

Hint

Use ls backup.db to check the file exists.

Practice

1. What is the primary purpose of taking an etcd backup in Kubernetes?

easy

A. To save the current state of the cluster data safely

B. To update the Kubernetes version automatically

C. To monitor cluster performance metrics

D. To delete old cluster data permanently

Solution

1. Step 1: Understand etcd role in Kubernetes

   etcd stores all cluster data including configuration and state.

2. Step 2: Purpose of backup

   Backing up etcd saves this data so it can be restored if lost or corrupted.

3. Final Answer:

   To save the current state of the cluster data safely -> Option A

4. Quick Check:

   Backup = Save cluster data [OK]

Hint: Backup means saving cluster data safely [OK]

Common Mistakes:

• Confusing backup with updating Kubernetes
• Thinking backup monitors performance
• Assuming backup deletes data

2. Which of the following is the correct command to create an etcd snapshot backup?

easy

A. etcdctl save snapshot backup.db

B. etcdctl backup create backup.db

C. etcdctl snapshot create backup.db

D. etcdctl snapshot save backup.db

Solution

1. Step 1: Recall etcdctl snapshot save syntax

   The correct command to save a snapshot is etcdctl snapshot save <file>.

2. Step 2: Compare options

   Only etcdctl snapshot save backup.db matches the exact syntax.

3. Final Answer:

   etcdctl snapshot save backup.db -> Option D

4. Quick Check:

   Snapshot save = create backup [OK]

Hint: Use 'etcdctl snapshot save' to backup [OK]

Common Mistakes:

• Using 'backup create' instead of 'snapshot save'
• Mixing 'create' and 'save' commands
• Incorrect command order

3. What will be the output of the following command if the backup file backup.db exists and is valid?

etcdctl snapshot restore backup.db --data-dir restored-etcd

medium

A. Restores the snapshot data into the directory 'restored-etcd'

B. Creates a new snapshot named 'restored-etcd'

C. Deletes the existing backup.db file

D. Shows an error that the file does not exist

Solution

1. Step 1: Understand snapshot restore command

   The command restores data from a snapshot file into a specified data directory.

2. Step 2: Analyze given command

   It uses backup.db as source and restores into restored-etcd directory.

3. Final Answer:

   Restores the snapshot data into the directory 'restored-etcd' -> Option A

4. Quick Check:

   Restore command = recover data to directory [OK]

Hint: Restore puts data into given directory [OK]

Common Mistakes:

• Thinking it creates a new snapshot
• Assuming it deletes backup files
• Expecting error when file exists

4. You ran etcdctl snapshot save backup.db but the command failed with an error: etcdctl: command not found. What is the most likely cause?

medium

A. The command syntax is incorrect

B. The etcdctl tool is not installed or not in the system PATH

C. The etcd server is down and cannot create a snapshot

D. The backup.db file already exists and cannot be overwritten

Solution

1. Step 1: Analyze error message

   The error 'command not found' means the system cannot find the etcdctl program.

2. Step 2: Identify cause

   This usually happens if etcdctl is not installed or not in the PATH environment variable.

3. Final Answer:

   The etcdctl tool is not installed or not in the system PATH -> Option B

4. Quick Check:

   Command not found = tool missing or PATH issue [OK]

Hint: Command not found means tool missing or PATH error [OK]

Common Mistakes:

• Assuming file overwrite causes command not found
• Blaming etcd server status for command not found
• Thinking syntax error causes command not found

5. You want to recover your Kubernetes cluster after a failure using an etcd snapshot. Which sequence of commands correctly restores the cluster data and starts etcd with the restored data?

hard

A. systemctl restart etcd && etcdctl snapshot restore backup.db --data-dir /var/lib/etcd-restored

B. etcdctl snapshot save backup.db && systemctl stop etcd

C. etcdctl snapshot restore backup.db --data-dir /var/lib/etcd-restored && systemctl restart etcd

D. etcdctl snapshot restore backup.db --data-dir /var/lib/etcd-restored && systemctl stop etcd

Solution

1. Step 1: Restore snapshot to a new data directory

   Use etcdctl snapshot restore backup.db --data-dir /var/lib/etcd-restored to recover data safely without overwriting live data.

2. Step 2: Restart etcd service to use restored data

   Restarting etcd with systemctl restart etcd applies the restored data directory.

3. Final Answer:

   etcdctl snapshot restore backup.db --data-dir /var/lib/etcd-restored && systemctl restart etcd -> Option C

4. Quick Check:

   Restore then restart etcd = recovery [OK]

Hint: Restore snapshot first, then restart etcd service [OK]

Common Mistakes:

• Restarting etcd before restoring snapshot
• Stopping etcd without restarting after restore
• Saving snapshot instead of restoring