Why caching reduces latency in HLD - Why This Architecture

Problem Statement

When every request must fetch data from a slow or distant source like a database or external API, users experience long wait times. This delay causes poor user experience and can overload the backend systems with repeated identical requests.

Solution

Caching stores frequently requested data closer to the user or application, so future requests can be served instantly without reaching the slow source. This reduces the time spent waiting for data retrieval and lowers the load on backend systems.

Architecture

┌─────────────┐       ┌───────────────┐       ┌───────────────┐
│   Client    │──────▶│    Cache      │──────▶│   Database    │
└─────────────┘       └───────────────┘       └───────────────┘
        │                    │                      │
        │                    │                      │
        │◀───────────────────┘                      │
        │                                           │
        │◀──────────────────────────────────────────┘

This diagram shows the client first checking the cache for data. If the cache has it (cache hit), the data returns immediately. If not (cache miss), the cache fetches data from the database and returns it to the client.

Trade-offs

✓ Pros

→

Significantly reduces response time for repeated requests by avoiding slow data sources.

→

Decreases load on backend databases or services, improving overall system stability.

→

Improves user experience with faster data access.

✗ Cons

→

Cache data can become stale if not updated properly, leading to outdated responses.

→

Adds complexity to the system with cache invalidation and consistency management.

→

Requires additional memory or storage resources to hold cached data.

Use caching when read requests are frequent and data changes relatively slowly, especially if backend data sources have high latency or limited capacity.

Avoid caching when data changes very rapidly or must always be fresh, or when the system handles very low traffic where caching overhead outweighs benefits.

Real World Examples

Netflix

Caches user viewing history and recommendations at edge servers to deliver instant responses without querying central databases.

Amazon

Uses caching for product details and pricing to reduce database load during high traffic sales events.

Twitter

Caches user timelines and tweets to serve millions of read requests quickly without hitting the main database every time.

Alternatives

CDN (Content Delivery Network)

Caches static content like images and videos geographically closer to users rather than dynamic data.

Use when: Use CDN when the main latency issue is delivering large static files rather than database query speed.

Database Replication

Creates multiple copies of the database to distribute read load instead of caching data separately.

Use when: Use replication when data freshness is critical and caching complexity is undesirable.

Summary

Caching stores frequently accessed data closer to the user to reduce wait times.

It lowers backend load and improves user experience by serving repeated requests quickly.

Proper cache management is essential to avoid stale data and system complexity.

Practice

1. Why does caching reduce latency in a system?

easy

A. Because it stores frequently accessed data closer to the user, reducing access time.

B. Because it increases the number of servers in the system.

C. Because it compresses data to make it smaller.

D. Because it deletes old data from the database.

Solution

1. Step 1: Understand caching location

   Caching stores data near the user or application, such as in memory or edge servers.

2. Step 2: Effect on access time

   Accessing data from cache is faster than fetching from the main database or server, reducing waiting time.

3. Final Answer:

   Because it stores frequently accessed data closer to the user, reducing access time. -> Option A

4. Quick Check:

   Caching reduces latency by storing data close [OK]

Hint: Caching stores data near users to speed up access [OK]

Common Mistakes:

• Thinking caching increases servers
• Confusing caching with data compression
• Believing caching deletes data

2. Which of the following is the correct way to describe caching in system design?

easy

A. Caching permanently deletes old data to save space.

B. Caching is a process of storing data in a remote backup server.

C. Caching temporarily stores data in a fast-access location to reduce retrieval time.

D. Caching encrypts data to improve security.

Solution

1. Step 1: Define caching purpose

   Caching temporarily holds data in a fast-access place like memory or local storage.

2. Step 2: Identify correct description

   Only caching temporarily stores data in a fast-access location to reduce retrieval time correctly states caching reduces retrieval time by temporary storage.

3. Final Answer:

   Caching temporarily stores data in a fast-access location to reduce retrieval time. -> Option C

4. Quick Check:

   Caching = temporary fast storage [OK]

Hint: Caching is temporary fast storage, not permanent or remote [OK]

Common Mistakes:

• Confusing caching with backup storage
• Thinking caching deletes data
• Mixing caching with encryption

3. Consider a system where fetching data from the main database takes 100ms, and fetching from cache takes 10ms. If 80% of requests hit the cache, what is the average latency per request?

medium

A. 18 ms

B. 90 ms

C. 50 ms

D. 28 ms

Solution

1. Step 1: Calculate cache hit and miss times

   Cache hit latency = 10ms (80% requests), cache miss latency = 100ms (20% requests).

2. Step 2: Compute weighted average latency

   Average latency = (0.8 * 10) + (0.2 * 100) = 8 + 20 = 28 ms.

3. Final Answer:

   28 ms -> Option D

4. Quick Check:

   Average latency = hit%*hit_time + miss%*miss_time [OK]

Hint: Use weighted average: hit%*hit_time + miss%*miss_time [OK]

Common Mistakes:

• Ignoring cache miss latency
• Adding times instead of weighted average
• Using wrong percentages

4. A system uses caching but users still experience high latency. Which of the following is a likely cause?

medium

A. Cache encrypts data, increasing retrieval time.

B. Cache hit rate is very low, so most requests go to the slow database.

C. Cache size is too large, slowing down the system.

D. Cache stores data permanently, causing outdated results.

Solution

1. Step 1: Analyze cache hit rate impact

   If cache hit rate is low, most requests bypass cache and hit slow database, increasing latency.

2. Step 2: Evaluate other options

   Permanent storage or encryption are not typical causes of high latency; large cache size usually improves speed.

3. Final Answer:

   Cache hit rate is very low, so most requests go to the slow database. -> Option B

4. Quick Check:

   Low cache hit rate = high latency [OK]

Hint: Low cache hit rate means slow database access dominates [OK]

Common Mistakes:

• Assuming cache size slows system
• Confusing cache permanence with latency
• Thinking encryption is main latency cause

5. In a large-scale web application, which caching strategy best reduces latency while keeping data fresh?

hard

A. Use a cache with a short expiration time and update cache asynchronously on data changes.

B. Use a cache that never expires and manually clear it once a month.

C. Disable caching and always fetch fresh data from the database.

D. Cache only rarely accessed data to save memory.

Solution

1. Step 1: Understand freshness and latency tradeoff

   Short expiration keeps data fresh; asynchronous updates reduce latency by avoiding blocking calls.

2. Step 2: Evaluate other options

   Never expiring cache risks stale data; disabling cache increases latency; caching rarely accessed data wastes resources.

3. Final Answer:

   Use a cache with a short expiration time and update cache asynchronously on data changes. -> Option A

4. Quick Check:

   Short expiry + async update = fresh data + low latency [OK]

Hint: Short expiry plus async update balances freshness and speed [OK]

Common Mistakes:

• Choosing never expiring cache causing stale data
• Disabling cache increases latency
• Caching rarely accessed data wastes resources