Multi-level caching in HLD - System Design Exercise

Design: Multi-level Caching System

Design focuses on read path caching with multiple cache layers and cache consistency. Write path and cache invalidation strategies are included at a high level. Out of scope: detailed database schema design and write-heavy workload optimization.

Functional Requirements

FR1: Serve data requests with minimal latency

FR2: Support multiple cache layers (e.g., in-memory, distributed cache, disk cache)

FR3: Ensure cache consistency and freshness

FR4: Handle cache misses by fetching data from the primary database

FR5: Support high read throughput with low latency

FR6: Provide fallback mechanisms if a cache layer fails

Non-Functional Requirements

NFR1: System must handle 100,000 read requests per second

NFR2: P99 latency for cache hits should be under 5 milliseconds

NFR3: Availability target of 99.9% uptime

NFR4: Cache layers should have configurable expiration policies

NFR5: Data consistency should be eventual between cache layers and database

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

❓ Question 6

Key Components

Client application or service

In-memory cache (e.g., local cache like LRU cache)

Distributed cache (e.g., Redis, Memcached)

Persistent cache or disk cache layer

Primary database

Cache invalidation and refresh mechanism

Monitoring and metrics system

Design Patterns

Cache-aside pattern

Write-through and write-back caching

Cache invalidation strategies (time-based, event-based)

Multi-level cache hierarchy

Fallback and retry mechanisms

Reference Architecture

Client
  |
  v
Local In-Memory Cache (Level 1)
  |
  v
Distributed Cache (Level 2)
  |
  v
Persistent Cache / Disk Cache (Level 3)
  |
  v
Primary Database

Monitoring & Metrics system observes all layers

Components

Client

Any application or service

Sends data requests and receives responses

Local In-Memory Cache (Level 1)

LRU Cache or similar in-memory cache

Fastest cache layer to serve frequent requests with minimal latency

Distributed Cache (Level 2)

Redis or Memcached cluster

Shared cache across multiple clients or servers to reduce database load

Persistent Cache / Disk Cache (Level 3)

SSD-based cache or local disk cache

Stores less frequently accessed data with higher latency but larger capacity

Primary Database

Relational or NoSQL database

Source of truth for all data

Cache Invalidation and Refresh Mechanism

Background jobs or event-driven triggers

Keeps cache data fresh and consistent with the database

Monitoring and Metrics System

Prometheus, Grafana, or similar

Tracks cache hit rates, latencies, errors, and system health

Request Flow

1. Client sends a data request.

2. Request checks Local In-Memory Cache (Level 1). If hit, return data immediately.

3. If miss, request goes to Distributed Cache (Level 2). If hit, return data and update Level 1 cache.

4. If miss again, request goes to Persistent Cache (Level 3). If hit, return data and update Level 2 and Level 1 caches.

5. If still miss, fetch data from Primary Database.

6. Update all cache layers with fresh data.

7. Return data to client.

8. Cache invalidation jobs run periodically or are triggered by data changes to refresh or remove stale cache entries.

Database Schema

Entities: - DataItem(id, value, last_updated) Relationships: - No complex relationships needed for caching; primary key 'id' used for cache keys. Cache entries map to DataItem by id with expiration timestamps.

Scaling Discussion

Bottlenecks

Local in-memory cache size limits and eviction under high load

Distributed cache network latency and throughput limits

Cache consistency delays causing stale data

Database overload on cache misses

Cache invalidation complexity at scale

Solutions

Use efficient eviction policies and increase local cache size with memory optimization

Scale distributed cache horizontally with sharding and clustering

Implement event-driven cache invalidation and TTLs to reduce staleness

Use read replicas and database sharding to handle load

Automate cache invalidation with messaging systems and monitor cache freshness

Interview Tips

Time: Spend 10 minutes understanding requirements and clarifying constraints, 20 minutes designing the multi-level cache architecture and data flow, 10 minutes discussing scaling and trade-offs, and 5 minutes summarizing.

Explain why multi-level caching reduces latency and load

Describe cache hit and miss flows clearly

Discuss cache consistency and invalidation strategies

Highlight scalability challenges and solutions

Mention monitoring importance for cache health