Distributed caching (Redis, Memcached) in HLD - Deep Dive

Overview - Distributed caching (Redis, Memcached)

What is it?

Distributed caching is a way to store data temporarily across multiple servers to make data retrieval faster. It helps applications quickly access frequently used information without always going to the main database. Redis and Memcached are popular tools that manage this temporary storage efficiently. They work by keeping data in memory, which is much faster than reading from disk.

Why it matters

Without distributed caching, applications would rely heavily on slower databases for every data request, causing delays and poor user experience. This would be like waiting in a long line every time you want a simple item. Distributed caching speeds up responses, reduces load on databases, and helps systems handle more users smoothly. It is essential for websites, apps, and services that need to be fast and scalable.

Where it fits

Before learning distributed caching, you should understand basic caching concepts and how databases work. After this, you can explore advanced topics like cache invalidation strategies, consistency models, and how caching fits into microservices and cloud architectures.

Mental Model

Core Idea

Distributed caching stores copies of data across multiple servers to speed up access and reduce load on the main database.

Think of it like...

Imagine a busy library where many people want the same popular book. Instead of everyone going to the main shelf, copies of the book are placed in several smaller reading rooms nearby. This way, readers get the book faster without crowding the main shelf.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│ Application 1 │──────▶│ Cache Server 1│       │               │
└───────────────┘       └───────────────┘       │               │
                            │                   │               │
┌───────────────┐       ┌───────────────┐       │               │
│ Application 2 │──────▶│ Cache Server 2│──────▶│   Database    │
└───────────────┘       └───────────────┘       │               │
                            │                   │               │
                            ▼                   └───────────────┘
                     ┌───────────────┐
                     │ Cache Server 3│
                     └───────────────┘

Build-Up - 7 Steps

1

Foundation

What is caching and why use it

Concept: Caching stores data temporarily to speed up repeated access.

Caching means keeping a copy of data in a place that is faster to reach than the original source. For example, a web browser stores images from websites so it doesn't have to download them again. This reduces waiting time and saves resources.

Result

Data requests are faster because the system can get data from the cache instead of the slower original source.

Understanding caching is key because it explains why storing data closer to where it is needed improves performance.

2

Foundation

Single-server cache basics

Concept: A cache can be a single server storing data in memory for quick access.

A single cache server holds data in its memory. When an application asks for data, the cache checks if it has it (cache hit). If yes, it returns the data quickly. If not (cache miss), it fetches from the database and stores it for next time.

Result

Applications get faster responses for repeated data requests, reducing database load.

Knowing how a single cache works helps understand the challenges when scaling to multiple servers.

3

Intermediate

Why distributed caching is needed

🤔Before reading on: do you think a single cache server can handle millions of users alone? Commit to yes or no.

Concept: Distributed caching spreads cached data across multiple servers to handle large scale and avoid single points of failure.

As applications grow, one cache server can't handle all requests or store all data. Distributed caching uses many cache servers working together. This improves capacity, reliability, and speed by sharing the load and data.

Result

Systems can serve many users quickly without crashing or slowing down due to cache overload.

Understanding the limits of single caches reveals why distributing cache is essential for real-world large systems.

4

Intermediate

How Redis and Memcached work

🤔Before reading on: do you think Redis and Memcached store data on disk or only in memory? Commit to your answer.

Concept: Redis and Memcached are in-memory data stores used as distributed caches with different features and use cases.

Memcached is a simple, fast cache storing key-value pairs in memory. Redis also stores data in memory but supports more complex data types and persistence options. Both allow multiple servers to share cached data and handle many requests.

Result

Applications can choose the right tool based on their needs for speed, data types, and durability.

Knowing the differences helps pick the best cache for specific system requirements.

5

Intermediate

Cache consistency and invalidation

🤔Before reading on: do you think cached data always stays perfectly up-to-date with the database? Commit to yes or no.

Concept: Cache consistency means keeping cached data accurate; invalidation is removing or updating stale data.

When data changes in the database, caches must update or remove old copies to avoid serving wrong information. Strategies include time-based expiration, manual invalidation, or write-through caching where updates go to both cache and database.

Result

Users get fresh data while still benefiting from fast cache access.

Understanding cache invalidation is crucial because stale data can cause serious errors in applications.

6

Advanced

Scaling distributed caches with sharding

🤔Before reading on: do you think all cache servers store the same data or different parts? Commit to your answer.

Concept: Sharding splits cached data across servers so each holds a portion, improving capacity and speed.

Instead of every cache server storing all data, sharding assigns keys to specific servers using hashing. This balances load and allows the system to grow by adding more servers. Clients know which server to ask for each key.

Result

Distributed caches handle more data and requests efficiently without duplication.

Knowing sharding helps design caches that scale horizontally and avoid bottlenecks.

7

Expert

Handling cache failures and data loss

🤔Before reading on: do you think losing cache data always breaks the application? Commit to yes or no.

Concept: Distributed caches must handle server failures gracefully and recover data without affecting application correctness.

Caches are temporary, so losing data is expected sometimes. Systems use replication to copy data across servers and fallback to the database on cache misses. Techniques like consistent hashing reduce data movement when servers fail or join. Monitoring and alerting detect problems early.

Result

Applications remain reliable and performant even if some cache servers fail or restart.

Understanding failure handling prevents downtime and data inconsistency in production systems.

Under the Hood

Distributed caching works by storing data in RAM across multiple servers. When an application requests data, it uses a hashing function to find which cache server holds the data. If the data is there (cache hit), it returns immediately. If not (cache miss), the cache fetches from the database, stores it, and returns it. Redis supports data persistence by writing snapshots or logs to disk, while Memcached keeps data only in memory. Both use network protocols to communicate with clients and other cache nodes. Sharding and replication manage data distribution and fault tolerance.

Why designed this way?

Caches were designed to be in-memory for speed, as disk access is slower. Distributed caches evolved to handle growing data and traffic beyond single servers. Redis added persistence and rich data types to support more use cases. Memcached focused on simplicity and speed. The tradeoff is complexity in managing consistency and failures, but this design balances performance and scalability for modern applications.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│ Application  │──────▶│ Hash Function │──────▶│ Cache Servers │
└───────────────┘       └───────────────┘       ┌───────┬───────┐
                                                │       │       │
                                         ┌──────▼────┐ ┌▼───────┐
                                         │ Cache 1   │ │ Cache 2│
                                         └───────────┘ └────────┘
                                                │
                                                ▼
                                          ┌───────────┐
                                          │ Database  │
                                          └───────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does caching guarantee data is always fresh and up-to-date? Commit to yes or no.

Common Belief: Caching always returns the latest data from the database.

Reality: Caches can serve stale data if not properly invalidated or updated after database changes.

Why it matters: Relying on stale cache data can cause incorrect application behavior and user confusion.

Quick: Do you think distributed caching eliminates the need for a database? Commit to yes or no.

Common Belief: Distributed caching replaces the database entirely.

Reality: Caches are temporary stores and must be backed by a persistent database for reliable data storage.

Why it matters: Treating cache as the main data store risks data loss and inconsistency.

Quick: Is adding more cache servers always a simple way to improve performance? Commit to yes or no.

Common Belief: More cache servers always mean better performance without extra complexity.

Reality: Adding servers requires careful data distribution and can introduce complexity like re-sharding and consistency challenges.

Why it matters: Ignoring these complexities can cause cache misses, data loss, or uneven load.

Quick: Does Memcached support data persistence by default? Commit to yes or no.

Common Belief: Memcached saves cached data to disk to survive restarts.

Reality: Memcached stores data only in memory and loses it on restart or failure.

Why it matters: Expecting persistence from Memcached can lead to unexpected data loss in production.

Expert Zone

1

Redis supports various eviction policies like LRU (Least Recently Used) and LFU (Least Frequently Used), allowing fine control over which data to remove when memory is full.

2

Consistent hashing minimizes cache data movement when servers are added or removed, reducing cache misses and improving stability.

3

Write-through and write-back caching strategies affect data consistency and performance tradeoffs; choosing the right one depends on application needs.

When NOT to use

Distributed caching is not suitable when data must always be strongly consistent or when data size is too large to fit in memory efficiently. In such cases, consider database indexing, search engines, or specialized storage systems like CDN for static content.

Production Patterns

In production, distributed caches are often deployed in clusters with replication and failover. Applications use client libraries that handle sharding and retries. Cache warming preloads data to avoid cold starts. Monitoring tools track cache hit rates and latency to optimize performance.

Connections

Content Delivery Networks (CDNs)

Both cache data closer to users to reduce latency and load on origin servers.

Understanding distributed caching helps grasp how CDNs speed up web content delivery by caching static files globally.

Database Indexing

Caching and indexing both speed up data retrieval but at different layers; caching stores copies in memory, indexing organizes data on disk.

Knowing caching complements indexing clarifies how databases and caches work together for fast queries.

Human Memory Systems

Caching is similar to how human short-term memory stores recent information for quick recall.

Recognizing this analogy reveals why caching improves system responsiveness by keeping frequently used data 'top of mind.'

Common Pitfalls

#1 Serving stale data due to missing cache invalidation.

Wrong approach:

Cache data is never updated or removed after database changes, e.g., always returning cached value without checks.

Correct approach:

Implement cache invalidation by deleting or updating cache entries when the database changes, or use time-to-live (TTL) settings.

Root cause:

Misunderstanding that cache automatically stays fresh leads to outdated information being served.

#2 Treating cache as the primary data store.

Wrong approach:

Relying solely on cache for data persistence without backing database, e.g., no database writes.

Correct approach:

Always write data to the database and use cache as a temporary fast-access layer.

Root cause:

Confusing cache purpose with permanent storage causes data loss on cache failures.

#3 Ignoring cache server failures and data loss.

Wrong approach:

No replication or fallback; application crashes or returns errors when cache server fails.

Correct approach:

Use replication, fallback to database on cache miss, and monitor cache health.

Root cause:

Assuming cache is always available leads to fragile systems.

Key Takeaways

Distributed caching stores data in memory across multiple servers to speed up access and reduce database load.

Redis and Memcached are popular tools with different features suited for various caching needs.

Cache invalidation is critical to prevent serving stale data and maintain application correctness.

Sharding and replication enable distributed caches to scale and handle failures gracefully.

Understanding the limits and tradeoffs of caching helps design reliable and high-performance systems.