Distributed caching (Redis, Memcached) in HLD - Scalability & System Analysis

Scalability Analysis - Distributed caching (Redis, Memcached)

Growth Table: Distributed Caching

Users / Traffic	Cache Size	Cache Hit Rate	Latency	Cache Nodes	Eviction Policy
100 users	Small (MBs)	High (~90%)	Low (~1ms)	1 node	LRU or TTL
10K users	Medium (GBs)	High (~85-90%)	Low (~1-2ms)	3-5 nodes	LRU + TTL
1M users	Large (10s GBs)	Moderate (~80%)	Low (~2-3ms)	10-20 nodes, sharded	LRU + TTL + LFU
100M users	Very Large (100s GBs to TBs)	Moderate (~75-80%)	Low (~3-5ms)	50+ nodes, sharded + replication	Advanced eviction + tiered caching

First Bottleneck

At small scale, the cache easily handles requests with a single node. As traffic grows beyond 10K users, the first bottleneck is the cache node memory and CPU. A single Redis or Memcached instance can handle around 100K operations per second but limited memory (~tens of GBs). When cache size or QPS grows beyond this, the node becomes CPU or memory bound.

Also, network bandwidth between app servers and cache nodes can become a bottleneck at very high QPS (millions of ops/sec).

Scaling Solutions

• Horizontal scaling: Add more cache nodes and shard keys across them to distribute load and memory usage.
• Replication: Use read replicas for read-heavy workloads to increase throughput and availability.
• Eviction policies: Use LRU, LFU, TTL to keep cache size manageable and remove stale data.
• Client-side sharding: Distribute keys at the client or proxy layer to avoid single node overload.
• Tiered caching: Combine in-memory cache with local caches or slower caches for large datasets.
• Network optimization: Use fast networks and colocate cache nodes near app servers to reduce latency.
• Monitoring and autoscaling: Track cache hit rates, latency, and resource usage to scale nodes dynamically.

Back-of-Envelope Cost Analysis

• At 1M users, assume 10 requests per second per user -> 10M QPS total.
• Redis single node handles ~100K ops/sec -> need ~100 nodes for caching all requests.
• Cache size depends on data size; e.g., 1MB per user cached data -> 1TB total cache needed.
• Network bandwidth: 100 nodes x 1 Gbps each -> 100 Gbps total bandwidth needed.
• Storage cost: RAM is expensive; consider cost vs benefit of caching vs DB load.

Interview Tip

Start by explaining what distributed caching is and why it helps reduce database load and latency.

Discuss scaling limits of a single cache node (memory, CPU, network).

Explain horizontal scaling with sharding and replication.

Mention eviction policies and cache consistency challenges.

Use numbers to justify when to add nodes or optimize network.

Self Check

Your database handles 1000 QPS. Traffic grows 10x to 10,000 QPS. What do you do first?

Answer: Add a distributed cache layer (Redis/Memcached) to offload read requests from the database, reducing DB load and latency. Then monitor cache hit rate and scale cache nodes as needed.

Key Result

Distributed caching scales well by adding more cache nodes and sharding data, but memory and CPU limits per node require horizontal scaling and eviction policies as traffic grows beyond 100K ops/sec per node.

Practice

1. What is the primary purpose of using a distributed cache like Redis or Memcached in a system?

easy

A. To reduce database load by storing frequently accessed data in memory

B. To permanently store all application data for backup

C. To replace the database entirely for all data storage needs

D. To increase the size of the database by adding more servers

Solution

1. Step 1: Understand the role of distributed cache

   Distributed caches store frequently accessed data in fast memory to reduce the number of database queries.

2. Step 2: Identify the main benefit

   This reduces the load on the database and speeds up data retrieval for repeated requests.

3. Final Answer:

   To reduce database load by storing frequently accessed data in memory -> Option A

4. Quick Check:

   Distributed cache reduces DB load = C [OK]

Hint: Cache stores frequent data to reduce DB hits [OK]

Common Mistakes:

• Thinking cache permanently stores all data
• Confusing cache with database backup
• Assuming cache replaces the database

2. Which of the following is the correct way to set a key-value pair in Redis using its command syntax?

easy

A. SET key value

B. GET key value

C. STORE key value

D. PUT key value

Solution

1. Step 1: Recall Redis command for setting data

   Redis uses the command SET key value to store a value under a key.

2. Step 2: Verify other options

   GET retrieves data, STORE and PUT are not valid Redis commands.

3. Final Answer:

   SET key value -> Option A

4. Quick Check:

   Redis set command = SET key value [OK]

Hint: Remember Redis uses SET to store key-value [OK]

Common Mistakes:

• Confusing GET with SET
• Using non-existent commands like STORE or PUT
• Mixing Redis commands with SQL syntax

3. Consider this simplified Redis usage in code:

cache.set('user:1', 'Alice')
name = cache.get('user:1')
print(name)

What will be printed?

medium

A. None

B. 'Alice'

C. user:1

D. Error

Solution

1. Step 1: Analyze the cache set operation

   The key 'user:1' is set with the value 'Alice' in the cache.

2. Step 2: Analyze the cache get operation

   Retrieving 'user:1' returns the stored value 'Alice', which is then printed.

3. Final Answer:

   'Alice' -> Option B

4. Quick Check:

   Set then get returns stored value = 'Alice' [OK]

Hint: Get returns value set by set command [OK]

Common Mistakes:

• Expecting key name instead of value
• Assuming None if not careful with key format
• Thinking it causes an error without context

4. A developer tries to use Memcached but notices cache misses even after setting keys. Which of these is a likely cause?

medium

A. The database is down, so cache cannot be used

B. Memcached does not support key-value storage

C. The cache keys are expiring too quickly due to a low TTL setting

D. Memcached requires SQL queries to retrieve data

Solution

1. Step 1: Understand Memcached key expiration

   Memcached keys have a TTL (time to live). If TTL is too low, keys expire quickly causing cache misses.

2. Step 2: Eliminate incorrect options

   Memcached supports key-value storage, does not depend on database status, and does not use SQL queries.

3. Final Answer:

   The cache keys are expiring too quickly due to a low TTL setting -> Option C

4. Quick Check:

   Low TTL causes cache misses = A [OK]

Hint: Check TTL settings if cache misses occur [OK]

Common Mistakes:

• Thinking Memcached lacks key-value support
• Assuming database status affects cache directly
• Confusing Memcached with SQL databases

5. You are designing a high-traffic web app that uses Redis as a distributed cache. To ensure data consistency when multiple servers update the same cache key, which approach is best?

hard

A. Use Memcached instead because it handles concurrency automatically

B. Allow all servers to update cache keys without coordination

C. Disable caching for keys that are frequently updated

D. Use Redis transactions or Lua scripts to perform atomic updates

Solution

1. Step 1: Identify concurrency challenges in distributed cache

   When multiple servers update the same key, race conditions can cause inconsistent data.

2. Step 2: Choose atomic update methods in Redis

   Redis supports transactions and Lua scripts to ensure updates happen atomically, preventing conflicts.

3. Step 3: Evaluate other options

   Allowing uncoordinated updates risks inconsistency; disabling caching reduces performance; Memcached lacks built-in atomic scripting.

4. Final Answer:

   Use Redis transactions or Lua scripts to perform atomic updates -> Option D

5. Quick Check:

   Atomic updates ensure consistency = D [OK]

Hint: Use Redis atomic commands for safe concurrent updates [OK]

Common Mistakes:

• Ignoring concurrency causing stale data
• Thinking Memcached handles concurrency automatically
• Disabling cache unnecessarily reduces speed