Fan-out on write vs fan-out on read in HLD - Design Approaches Compared

Design: Fan-out Messaging System

Design the message delivery mechanism focusing on fan-out strategies. Out of scope: message content creation, user authentication, and UI design.

Functional Requirements

FR1: Deliver messages from one sender to many receivers efficiently

FR2: Support real-time or near real-time message delivery

FR3: Handle up to 1 million users with 100,000 concurrent active users

FR4: Ensure message delivery latency under 200ms for 99th percentile

FR5: Allow users to read their message feeds on demand

Non-Functional Requirements

NFR1: System availability of 99.9% uptime

NFR2: Support horizontal scaling as user base grows

NFR3: Minimize storage and compute costs

NFR4: Handle spikes in message volume gracefully

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

Key Components

Message queue or broker

Datastore for messages and user feeds

Cache layer for fast reads

Worker services for processing fan-out

API servers for client interaction

Design Patterns

Fan-out on write (push updates to receivers on message arrival)

Fan-out on read (store messages once, fan-out when receivers read)

Caching strategies

Batch processing for fan-out

Load balancing and rate limiting

Reference Architecture

Client
  |
  v
API Servers
  |
  v
Message Queue / Broker
  |
  +-------------------+
  |                   |
Fan-out Workers    Datastore (Messages, User Feeds)
  |                   |
  +-------------------+
          |
          v
       Cache
          |
          v
        Clients (Receivers)

Components

API Servers

REST/HTTP servers

Receive messages from senders and serve message feeds to receivers

Message Queue / Broker

Kafka or RabbitMQ

Buffer incoming messages and decouple producers from consumers

Fan-out Workers

Microservices

Process messages and distribute them to receivers depending on fan-out strategy

Datastore

NoSQL database (e.g., Cassandra, DynamoDB)

Store messages and user feeds for persistence

Cache

Redis or Memcached

Serve frequently accessed message feeds with low latency

Request Flow

1. Sender sends message to API Server

2. API Server publishes message to Message Queue

3. Fan-out Workers consume message from queue

4. For fan-out on write: Workers push message copies to each receiver's feed in Datastore

5. For fan-out on read: Workers store message once; feeds are generated on demand

6. Receivers request their feeds from API Server

7. API Server checks Cache; if miss, reads from Datastore

8. API Server returns messages to receivers

Database Schema

Entities: - User: user_id (PK), name - Message: message_id (PK), sender_id (FK User), content, timestamp - UserFeed (for fan-out on write): user_id (FK User), message_id (FK Message), read_status Relationships: - One User can send many Messages (1:N) - UserFeed stores many messages per user (1:N) For fan-out on read, UserFeed may be virtual or generated on demand by querying Messages from followed users.

Scaling Discussion

Bottlenecks

Fan-out on write: High write amplification when sender has many receivers

Fan-out on read: High read latency and compute when many receivers read simultaneously

Message Queue throughput limits

Datastore write/read capacity

Cache size and eviction policies

Solutions

Use batch writes and asynchronous processing to reduce write load in fan-out on write

Implement caching and pre-computed feeds to reduce read latency in fan-out on read

Partition message queue topics by sender or user groups to increase throughput

Use scalable NoSQL databases with horizontal partitioning

Employ multi-level caching and TTL strategies to optimize cache usage

Interview Tips

Time: Spend 10 minutes clarifying requirements and constraints, 15 minutes designing architecture and data flow, 10 minutes discussing scaling and trade-offs, 10 minutes answering questions.

Explain difference between fan-out on write and fan-out on read with examples

Discuss trade-offs in latency, storage, and compute costs

Describe components and their roles clearly

Address scaling challenges and mitigation strategies

Show understanding of real-world constraints and user behavior

Practice

1. What is the main advantage of using fan-out on write in system design?

easy

A. Simpler read logic by fetching data on demand

B. Faster read operations by duplicating data during write

C. Reduced storage usage by avoiding data duplication

D. Faster write operations by delaying data duplication

Solution

1. Step 1: Understand fan-out on write behavior

   Fan-out on write duplicates data to multiple places during the write operation.

2. Step 2: Analyze impact on read speed

   This duplication allows reads to be faster because data is already pre-distributed and ready to access.

3. Final Answer:

   Faster read operations by duplicating data during write -> Option B

4. Quick Check:

   Fan-out on write = Faster reads [OK]

Hint: Fan-out on write means write duplicates data for fast reads [OK]

Common Mistakes:

• Confusing fan-out on write with fan-out on read
• Thinking fan-out on write reduces storage
• Assuming writes are faster with fan-out on write

2. Which of the following best describes fan-out on read?

easy

A. Data is compressed during write to save storage

B. Data is duplicated during write to speed up reads

C. Data is cached permanently to reduce read latency

D. Data is fetched and combined during read to keep writes fast

Solution

1. Step 1: Define fan-out on read

   Fan-out on read means data is not duplicated during write but fetched from multiple sources during read.

2. Step 2: Understand write speed impact

   This keeps writes fast because no extra duplication work is done during write time.

3. Final Answer:

   Data is fetched and combined during read to keep writes fast -> Option D

4. Quick Check:

   Fan-out on read = Fast writes, complex reads [OK]

Hint: Fan-out on read delays data gathering until read time [OK]

Common Mistakes:

• Mixing fan-out on read with fan-out on write
• Assuming fan-out on read duplicates data during write
• Confusing caching with fan-out on read

3. Consider a system using fan-out on write. If a user updates their profile, what happens during the write operation?

medium

A. The update is duplicated to multiple storage locations immediately

B. The update is written once and read fetches combine data later

C. The update is cached temporarily and written later asynchronously

D. The update is compressed and stored in a single location

Solution

1. Step 1: Recall fan-out on write behavior

   Fan-out on write duplicates data during the write operation to multiple places.

2. Step 2: Apply to user profile update

   When a user updates their profile, the system writes the update to all relevant storage locations immediately.

3. Final Answer:

   The update is duplicated to multiple storage locations immediately -> Option A

4. Quick Check:

   Fan-out on write = Immediate duplication on write [OK]

Hint: Fan-out on write duplicates data immediately on update [OK]

Common Mistakes:

• Thinking update is written once and combined later
• Confusing caching with fan-out on write
• Assuming asynchronous write in fan-out on write

4. A system using fan-out on read is experiencing slow response times. What is a likely cause?

medium

A. Writes are slow due to data duplication

B. Storage is overloaded due to duplicated data

C. Reads are slow because data is fetched from multiple sources on demand

D. Data is compressed causing decompression delays

Solution

1. Step 1: Understand fan-out on read read behavior

   Fan-out on read fetches data from multiple sources during read, which can add latency.

2. Step 2: Analyze slow response cause

   Because reads combine data on demand, slow response times are likely due to this complex read process.

3. Final Answer:

   Reads are slow because data is fetched from multiple sources on demand -> Option C

4. Quick Check:

   Fan-out on read = Slow reads if sources are many [OK]

Hint: Fan-out on read can cause slow reads due to multiple fetches [OK]

Common Mistakes:

• Blaming slow writes in fan-out on read
• Assuming storage overload in fan-out on read
• Confusing compression delays with fan-out issues

5. You are designing a social media feed system. Which approach is better if you want instant feed updates but can tolerate higher storage costs?

hard

A. Fan-out on write to duplicate feed data for fast reads

B. Fan-out on read to keep writes fast and storage low

C. Use caching only without fan-out

D. Compress data on write to save storage

Solution

1. Step 1: Identify system needs

   Instant feed updates require fast reads with up-to-date data.

2. Step 2: Match approach to needs

   Fan-out on write duplicates feed data during write, enabling fast reads and instant updates but uses more storage.

3. Step 3: Evaluate other options

   Fan-out on read delays data gathering to read time, causing slower reads. Caching alone may not guarantee instant updates. Compression saves storage but slows access.

4. Final Answer:

   Fan-out on write to duplicate feed data for fast reads -> Option A

5. Quick Check:

   Instant updates + higher storage = Fan-out on write [OK]

Hint: Instant reads with more storage? Choose fan-out on write [OK]

Common Mistakes:

• Choosing fan-out on read for instant updates
• Ignoring storage cost impact
• Assuming caching replaces fan-out needs