Secrets management (Vault, AWS Secrets Manager) in Microservices - System Design Exercise

Design: Secrets Management System for Microservices

Design covers secret storage, retrieval, rotation, access control, and audit logging. Out of scope are microservice application logic and network infrastructure setup.

Functional Requirements

FR1: Securely store and manage secrets such as API keys, database credentials, and certificates.

FR2: Allow microservices to retrieve secrets dynamically at runtime.

FR3: Support secret rotation without downtime.

FR4: Provide audit logs for secret access and changes.

FR5: Ensure least privilege access control for secrets.

FR6: Integrate with existing microservices architecture.

Non-Functional Requirements

NFR1: Handle up to 10,000 microservice instances accessing secrets concurrently.

NFR2: API response latency for secret retrieval should be under 100ms p99.

NFR3: System availability must be at least 99.9% uptime.

NFR4: Secrets must be encrypted at rest and in transit.

NFR5: Support multi-region deployment for disaster recovery.

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

❓ Question 6

Key Components

Secret storage backend (encrypted database or cloud KMS)

Authentication and authorization service

Secret retrieval API

Secret rotation scheduler

Audit logging system

Cache layer for secrets

Design Patterns

Token-based authentication

Lease and renewal pattern for secrets

Write-ahead logging for audit

Sidecar pattern for secret injection

Cache aside pattern for secret caching

Reference Architecture

                    +---------------------+
                    |  Microservice Apps   |
                    +----------+----------+
                               |
                               | Request secret
                               v
                    +----------+----------+
                    |  Secret Retrieval API|
                    +----------+----------+
                               |
          +--------------------+--------------------+
          |                                         |
+---------v---------+                     +---------v---------+
| Authentication &  |                     |  Cache Layer      |
| Authorization     |                     |  (Redis or Memcached) |
+---------+---------+                     +---------+---------+
          |                                         |
          +--------------------+--------------------+
                               |
                    +----------v----------+
                    | Secret Storage Backend|
                    | (Vault or AWS Secrets |
                    | Manager with KMS)     |
                    +----------+----------+
                               |
                    +----------v----------+
                    | Audit Logging System |
                    +---------------------+

Components

Secret Retrieval API

REST/gRPC service

Handles requests from microservices to fetch secrets securely.

Authentication & Authorization

OAuth2 tokens, IAM roles, or Vault tokens

Verifies identity and permissions of microservices requesting secrets.

Cache Layer

Redis or Memcached

Stores recently accessed secrets to reduce latency and load on storage backend.

Secret Storage Backend

HashiCorp Vault or AWS Secrets Manager with KMS encryption

Securely stores encrypted secrets and manages secret lifecycle.

Audit Logging System

Centralized logging (e.g., ELK stack, CloudWatch Logs)

Records all secret access and modification events for compliance and monitoring.

Secret Rotation Scheduler

Cron jobs or managed rotation features

Automates periodic secret updates without downtime.

Request Flow

1. 1. Microservice sends authenticated request to Secret Retrieval API to get a secret.

2. 2. API verifies the microservice's identity and permissions via Authentication & Authorization component.

3. 3. API checks Cache Layer for the requested secret.

4. 4. If secret is in cache and valid, return it immediately.

5. 5. If not cached, API fetches secret from Secret Storage Backend.

6. 6. Secret Storage Backend decrypts and returns the secret.

7. 7. API stores secret in Cache Layer for future requests.

8. 8. API returns secret to the microservice securely over TLS.

9. 9. Audit Logging System records the access event.

10. 10. Secret Rotation Scheduler periodically updates secrets in the storage backend and invalidates cache.

Database Schema

Entities: - Secret: id (PK), name, encrypted_value, version, created_at, updated_at, rotation_policy - AccessLog: id (PK), secret_id (FK), microservice_id, access_time, action (read/write), success_flag - Microservice: id (PK), name, authentication_credentials, permissions Relationships: - One Microservice can access many Secrets (many-to-many via permissions) - AccessLog links Microservice and Secret with access details

Scaling Discussion

Bottlenecks

Secret Retrieval API becomes overloaded with high concurrent requests.

Cache Layer may have cache misses causing high latency.

Secret Storage Backend throughput limits under heavy load.

Audit Logging System storage and query performance degradation.

Secret Rotation causing temporary unavailability or stale secrets.

Solutions

Scale Secret Retrieval API horizontally behind a load balancer.

Optimize cache hit ratio by tuning TTL and pre-warming cache.

Use highly available and scalable secret storage solutions with multi-region replication.

Archive old logs and use scalable log storage with indexing for audit logs.

Implement zero-downtime secret rotation with versioning and gradual rollout.

Interview Tips

Time: Spend 10 minutes understanding requirements and clarifying constraints, 20 minutes designing architecture and data flow, 10 minutes discussing scaling and trade-offs, 5 minutes summarizing.

Emphasize security best practices: encryption, least privilege, audit logging.

Explain how caching reduces latency and load.

Discuss secret rotation strategies to avoid downtime.

Highlight authentication and authorization importance.

Address scalability and availability with concrete solutions.

Mention real-world tools like Vault and AWS Secrets Manager.