Encryption at rest and in transit in HLD - System Design Exercise

Design: Encryption at rest and in transit

Focus on designing encryption mechanisms for data storage and communication channels. Key management and rotation included. Out of scope: detailed cryptographic algorithm design and hardware security modules.

Functional Requirements

FR1: Protect sensitive data stored on disk or in databases (encryption at rest)

FR2: Protect data while it moves between clients, servers, and services (encryption in transit)

FR3: Ensure encryption keys are securely managed and rotated

FR4: Support compliance with common security standards (e.g., GDPR, HIPAA)

FR5: Minimize performance impact on system operations

Non-Functional Requirements

NFR1: System must handle up to 100,000 concurrent users securely

NFR2: Encryption and decryption latency should not exceed 50ms per request

NFR3: Availability target of 99.9% uptime

NFR4: Key management must prevent unauthorized access and support audit logging

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

Key Components

Encryption libraries or services for data at rest

TLS/SSL for encryption in transit

Key Management Service (KMS)

Authentication and authorization systems

Audit logging system for key usage

Design Patterns

End-to-end encryption

Envelope encryption (data encrypted with data key, data key encrypted with master key)

Mutual TLS for client-server authentication

Key rotation and versioning

Secure key storage (e.g., Hardware Security Module or cloud KMS)

Reference Architecture

  +-------------+          +----------------+          +-------------+
  |   Client    | <------> |  Load Balancer | <------> | Application |
  +-------------+   TLS    +----------------+   TLS    +-------------+
                                      |                        |
                                      |                        |
                                +-------------+          +-------------+
                                |  Key Mgmt   |          |   Database  |
                                |   Service   |          | (Encrypted) |
                                +-------------+          +-------------+

Components

Client

Web/Mobile app with TLS support

Initiates secure connections using TLS to protect data in transit

Load Balancer

Nginx/HAProxy with TLS termination or passthrough

Distributes incoming requests securely and maintains encrypted channels

Application

Backend service with encryption libraries (e.g., OpenSSL, libsodium)

Encrypts/decrypts data at rest and communicates securely

Key Management Service (KMS)

Cloud KMS (AWS KMS, Google KMS) or on-prem HSM

Generates, stores, rotates, and audits encryption keys securely

Database

Relational or NoSQL DB with encryption support

Stores encrypted data ensuring encryption at rest

Request Flow

1. Client establishes a TLS connection to the Load Balancer to secure data in transit.

2. Load Balancer forwards the encrypted request to the Application server over TLS.

3. Application requests encryption keys from the Key Management Service to encrypt sensitive data before storing.

4. Application encrypts data using data keys and stores encrypted data in the Database.

5. When reading data, Application retrieves encrypted data from Database, requests keys from KMS, decrypts data, and sends response over TLS back to Client.

6. Key Management Service handles key rotation and audit logging transparently.

Database Schema

Entities: - EncryptedData: stores encrypted blobs with metadata (id, encrypted_content, encryption_key_version, timestamp) - KeyMetadata: stores key versions, creation date, status (active, retired) Relationships: - EncryptedData references KeyMetadata by encryption_key_version to track which key encrypted the data

Scaling Discussion

Bottlenecks

Key Management Service becoming a single point of failure or performance bottleneck

Encryption and decryption operations adding latency under high load

Database I/O overhead due to encryption increasing storage and retrieval time

TLS handshake overhead for large numbers of concurrent connections

Solutions

Use distributed and highly available KMS with caching of keys at application layer

Offload encryption/decryption to dedicated hardware or use asynchronous processing where possible

Use database encryption features optimized for performance and compress encrypted data

Implement TLS session reuse and keep-alive connections to reduce handshake overhead

Interview Tips

Time: Spend 10 minutes understanding requirements and constraints, 20 minutes designing architecture and data flow, 10 minutes discussing scaling and security trade-offs, 5 minutes summarizing.

Explain difference and importance of encryption at rest vs in transit

Discuss secure key management and rotation strategies

Highlight use of TLS for protecting data in transit

Describe how encryption impacts system performance and how to mitigate

Mention compliance and audit logging as critical non-functional requirements