Why choosing the right storage matters in HLD - Architecture Impact

System Overview - Why choosing the right storage matters

This system demonstrates how choosing the right storage affects performance, scalability, and reliability. It shows a typical web application using cache, database, and file storage to handle different data types efficiently.

Architecture Diagram

User
  |
  v
Load Balancer
  |
  v
API Gateway
  |
  v
+-------------------+
|    Application    |
|      Server       |
+-------------------+
  |          |          
  v          v          v
Cache      Database   File Storage
(Redis)   (SQL DB)    (Object Store)

Components

User

client

Sends requests to the system

Load Balancer

load_balancer

Distributes incoming requests evenly to servers

API Gateway

api_gateway

Routes requests to appropriate services and handles security

Application Server

service

Processes business logic and coordinates data access

Cache (Redis)

cache

Stores frequently accessed data for fast retrieval

Database (SQL DB)

database

Stores structured data with strong consistency

File Storage (Object Store)

storage

Stores large unstructured files like images and videos

Request Flow - 12 Hops

User → Load Balancer

Load Balancer → API Gateway

API Gateway → Application Server

Application Server → Cache (Redis)

Cache (Redis) → Application Server

Application Server → Database (SQL DB)

Database (SQL DB) → Application Server

Application Server → File Storage (Object Store)

File Storage (Object Store) → Application Server

Application Server → API Gateway

API Gateway → Load Balancer

Load Balancer → User

Failure Scenario

Component Fails:Database (SQL DB)

Impact:Structured data queries fail; cache may serve stale data; writes are blocked

Mitigation:Use database replication and failover; rely on cache for read availability; queue writes for retry

Architecture Quiz - 3 Questions

Why does the system check the cache before querying the database?

A To avoid using the load balancer

B To store large files efficiently

C To reduce latency by serving frequent data faster

D To handle user authentication

Design Principle

Choosing the right storage type for different data ensures the system is fast, scalable, and reliable. Cache speeds up frequent reads, databases provide consistency for structured data, and file storage handles large files efficiently.

Practice

1. Why is it important to choose the right storage type for a system?

easy

A. Because storage type changes the programming language used

B. Because it affects data safety and system performance

C. Because storage type determines the user interface design

D. Because storage type controls the network bandwidth

Solution

1. Step 1: Understand the role of storage in a system

   Storage holds data, so its choice impacts how safe and fast data access is.

2. Step 2: Connect storage choice to system outcomes

   Choosing the right storage ensures data is protected and system runs efficiently.

3. Final Answer:

   Because it affects data safety and system performance -> Option B

4. Quick Check:

   Storage choice = data safety and speed [OK]

Hint: Storage choice impacts data safety and speed [OK]

Common Mistakes:

• Confusing storage with programming language
• Thinking storage affects UI design
• Assuming storage controls network bandwidth

2. Which of the following storage types is best suited for fast read and write operations in a caching system?

easy

A. Cold storage like tape drives

B. Magnetic hard drives only

C. In-memory storage like RAM

D. Optical storage like CDs

Solution

1. Step 1: Identify storage needs for caching

   Caching requires very fast read and write speeds to improve performance.

2. Step 2: Match storage type to speed requirements

   RAM is the fastest storage type suitable for caching compared to others listed.

3. Final Answer:

   In-memory storage like RAM -> Option C

4. Quick Check:

   Fast cache = RAM [OK]

Hint: Cache needs fastest storage: choose RAM [OK]

Common Mistakes:

• Choosing slow storage like tape or optical for caching
• Ignoring speed differences between storage types
• Assuming magnetic drives are fastest

3. Consider a system that stores user photos and metadata. Which storage combination best balances cost and performance?

medium

A. Store photos in tape drives and metadata in in-memory storage

B. Store photos and metadata both in relational database

C. Store photos in in-memory storage and metadata in flat files

D. Store photos in object storage and metadata in a relational database

Solution

1. Step 1: Analyze storage needs for photos and metadata

   Photos are large files suited for object storage; metadata is structured data suited for databases.

2. Step 2: Evaluate cost and performance trade-offs

   Object storage is cost-effective for large files; relational DB offers fast queries for metadata.

3. Final Answer:

   Store photos in object storage and metadata in a relational database -> Option D

4. Quick Check:

   Photos = object storage, Metadata = DB [OK]

Hint: Large files in object storage, metadata in DB [OK]

Common Mistakes:

• Storing large files in relational DB (costly and slow)
• Using in-memory for large photos (expensive, volatile)
• Using tape drives for active data (slow access)

4. A system uses a NoSQL database for all data storage but faces slow complex queries. What is the likely cause?

medium

A. NoSQL databases are not optimized for complex relational queries

B. NoSQL databases always have slow writes

C. NoSQL databases cannot store large data

D. NoSQL databases require more network bandwidth

Solution

1. Step 1: Understand NoSQL database strengths and weaknesses

   NoSQL excels at flexible, scalable storage but is not optimized for complex joins or relational queries.

2. Step 2: Identify cause of slow complex queries

   Using NoSQL for relational queries causes slow performance due to lack of join optimizations.

3. Final Answer:

   NoSQL databases are not optimized for complex relational queries -> Option A

4. Quick Check:

   NoSQL + complex queries = slow [OK]

Hint: NoSQL is weak for complex joins [OK]

Common Mistakes:

• Assuming NoSQL is slow for all operations
• Thinking NoSQL can't store large data
• Blaming network bandwidth for query slowness

5. You are designing a system expected to handle millions of user transactions daily with strict latency requirements. Which storage strategy best fits this scenario?

hard

A. Use distributed in-memory cache for active data and persistent NoSQL for durability

B. Use a single relational database for all transactions

C. Store all data on slow cold storage to save cost

D. Use flat files on local disk for all transaction data

Solution

1. Step 1: Analyze system requirements for high volume and low latency

   Millions of transactions require fast access and scalable storage to meet latency needs.

2. Step 2: Choose storage types that balance speed and durability

   Distributed in-memory cache offers fast access; NoSQL provides scalable, durable storage for persistence.

3. Step 3: Evaluate other options for suitability

   Single DB may bottleneck; cold storage is too slow; flat files lack scalability and reliability.

4. Final Answer:

   Use distributed in-memory cache for active data and persistent NoSQL for durability -> Option A

5. Quick Check:

   High volume + low latency = cache + NoSQL [OK]

Hint: Combine fast cache with durable NoSQL storage [OK]

Common Mistakes:

• Relying on single DB for high load
• Using slow cold storage for active data
• Storing transactions in flat files