News feed generation in HLD - Architecture Diagram

System Overview - News feed generation

This system generates personalized news feeds for users by collecting posts from their connections and relevant sources. It must deliver fresh content quickly and handle many users simultaneously.

Architecture Diagram

User
  |
  v
Load Balancer
  |
  v
API Gateway
  |
  v
Feed Service <--> Cache
  |
  v
Database
  |
  v
Message Queue
  |
  v
Feed Generator Service
  |
  v
Content Sources

Components

User

client

Requests personalized news feed

Load Balancer

load_balancer

Distributes incoming user requests evenly to API Gateway instances

API Gateway

api_gateway

Handles authentication, routing, and request validation

Feed Service

service

Serves news feed data to users, checks cache before database

Cache

cache

Stores recently generated feeds for fast retrieval

Database

database

Stores user data, posts, and feed metadata

Message Queue

queue

Queues feed generation tasks asynchronously

Feed Generator Service

service

Processes queued tasks to generate and update user feeds

Content Sources

external_service

Provides new posts and updates from users and publishers

Request Flow - 12 Hops

User → Load Balancer

Load Balancer → API Gateway

API Gateway → Feed Service

Feed Service → Cache

Cache → Feed Service

Feed Service → Database

Database → Feed Service

Feed Service → User

Content Sources → Feed Generator Service

Feed Generator Service → Message Queue

Feed Generator Service → Database

Feed Generator Service → Cache

Failure Scenario

Component Fails:Cache

Impact:Feed Service cannot retrieve cached feeds, causing increased database load and slower response times

Mitigation:System falls back to database queries for feeds; cache is rebuilt asynchronously by Feed Generator Service

Architecture Quiz - 3 Questions

Which component handles user authentication and routes requests to services?

A Load Balancer

B Feed Service

C API Gateway

D Message Queue

Design Principle

This design uses caching to speed up feed retrieval and asynchronous processing with message queues to handle feed updates efficiently. It separates user request handling from feed generation to improve scalability and responsiveness.

Practice

1. What is the main purpose of a news feed generation system in social media platforms?

easy

A. To store user passwords securely

B. To manage user account settings

C. To show personalized and timely content to users

D. To handle payment transactions

Solution

1. Step 1: Understand the role of news feed generation

   The news feed system is designed to deliver content that is relevant and timely to each user.

2. Step 2: Identify the correct purpose among options

   The other options relate to security, settings, and payments, which are unrelated to news feed generation.

3. Final Answer:

   To show personalized and timely content to users -> Option C

4. Quick Check:

   News feed = personalized timely content [OK]

Hint: News feed = personalized content display [OK]

Common Mistakes:

• Confusing news feed with user authentication
• Mixing news feed with payment processing
• Thinking news feed manages account settings

2. Which of the following is a common method used in news feed generation to deliver updates efficiently?

easy

A. Manual refresh by users only

B. Push model where updates are sent to users proactively

C. Storing all data in a single database without caching

D. Using FTP to transfer news feed data

Solution

1. Step 1: Identify common delivery methods in news feed systems

   Push model proactively sends updates to users, improving latency and experience.

2. Step 2: Evaluate other options for efficiency

   Manual refresh is user-driven and less efficient; no caching slows performance; FTP is unrelated to real-time feed delivery.

3. Final Answer:

   Push model where updates are sent to users proactively -> Option B

4. Quick Check:

   Push model = efficient update delivery [OK]

Hint: Push model proactively sends updates [OK]

Common Mistakes:

• Assuming manual refresh is efficient
• Ignoring caching benefits
• Confusing FTP with real-time data delivery

3. Consider a news feed system using a pull model where users request their feed on demand. What is a likely outcome when many users request feeds simultaneously?

medium

A. Users receive outdated feeds only

B. Instant delivery with no server load

C. System automatically caches all feeds without delay

D. High latency and increased load on backend servers

Solution

1. Step 1: Understand pull model behavior under load

   Pull model requires backend to generate feeds on request, causing high load if many users request simultaneously.

2. Step 2: Analyze other options for feasibility

   Instant delivery with no load is unrealistic; outdated feeds depend on caching, not pull model alone; automatic caching without delay is ideal but not guaranteed.

3. Final Answer:

   High latency and increased load on backend servers -> Option D

4. Quick Check:

   Pull model + many requests = high load [OK]

Hint: Pull model causes backend load spikes [OK]

Common Mistakes:

• Assuming pull model has no latency
• Confusing caching with pull model behavior
• Believing feeds are always instantly cached

4. A news feed system uses a push model but users report seeing stale content. Which is the most likely cause?

medium

A. Cache not invalidated after new content is pushed

B. Users are not refreshing their browsers

C. Backend servers are down

D. Users have slow internet connections

Solution

1. Step 1: Identify push model behavior and caching role

   Push model sends updates, but if cache is not invalidated, users see old content.

2. Step 2: Evaluate other options for staleness cause

   Browser refresh is less relevant in push; backend down causes no updates; slow internet delays but does not cause stale cached data.

3. Final Answer:

   Cache not invalidated after new content is pushed -> Option A

4. Quick Check:

   Push + stale feed = cache invalidation issue [OK]

Hint: Stale feed often means cache invalidation failed [OK]

Common Mistakes:

• Blaming user refresh instead of cache
• Ignoring cache invalidation importance
• Assuming slow internet causes stale cache

5. You are designing a news feed system for a platform with 100 million users. Which approach best balances scalability and freshness of content?

hard

A. Hybrid model: push important updates and pull less critical content

B. Pure pull model: generate feed on every user request

C. Pure push model: push all updates to all users immediately

D. No caching: always fetch fresh data from database

Solution

1. Step 1: Analyze scalability and freshness needs for large user base

   Pure pull causes high load; pure push is costly and complex; no caching is inefficient.

2. Step 2: Evaluate hybrid model benefits

   Hybrid model pushes critical updates for freshness and uses pull for less urgent content, balancing load and latency.

3. Final Answer:

   Hybrid model: push important updates and pull less critical content -> Option A

4. Quick Check:

   Hybrid model balances scale and freshness [OK]

Hint: Hybrid push-pull balances scale and freshness [OK]

Common Mistakes:

• Choosing pure pull causing backend overload
• Choosing pure push causing network overload
• Ignoring caching and load balancing