News feed generation in HLD - System Design Guide

Problem Statement

When millions of users follow thousands of others, generating a personalized news feed in real-time becomes a huge challenge. Without an efficient system, users face slow load times, outdated content, or incomplete feeds, leading to poor user experience and engagement loss.

Solution

The system precomputes or dynamically generates personalized feeds by aggregating posts from followed users and ranking them based on relevance and freshness. It uses a combination of push and pull models, caching, and distributed storage to serve feeds quickly and keep them updated.

Architecture

User Posts

→Feed Service

↓

↓

Follower Graph

This diagram shows how user posts flow into the feed service, which interacts with follower data and ranking logic to generate personalized feeds stored in cache and database, then served to users.

Trade-offs

✓ Pros

→

Supports personalized and timely feeds by combining push and pull strategies.

→

Scales horizontally by distributing feed generation and storage.

→

Improves user experience with caching and ranking for relevance.

✗ Cons

→

Complexity increases with maintaining consistency between cache and database.

→

High storage and compute costs for precomputing feeds for millions of users.

→

Latency can increase if feed generation is done fully on-demand.

Use when user base exceeds hundreds of thousands with frequent content updates and personalized feed expectations.

Avoid if user base is small (under 10,000) or if real-time personalization is not critical, as simpler pull-only models suffice.

Real World Examples

Facebook

Uses a combination of push and pull models to generate personalized news feeds that rank posts based on user interactions and freshness.

Twitter

Precomputes timelines for active users to serve feeds with low latency, while also supporting on-demand fetching for less active users.

LinkedIn

Generates feeds by aggregating posts from connections and ranking them using machine learning models for relevance and engagement.

Alternatives

Pull-based feed generation

Feeds are generated on-demand by querying followed users' posts at request time without precomputation.

Use when: Choose when user base is small or content updates are infrequent, reducing storage and compute overhead.

Push-based feed generation

Feeds are precomputed and pushed to users' feed storage immediately after content creation.

Use when: Choose when low latency feed delivery is critical and user follow graph is relatively stable.

Hybrid feed generation

Combines push for active users and pull for less active users to balance latency and resource usage.

Use when: Choose when user activity varies widely and system needs to optimize resource usage.

Summary

News feed generation solves the challenge of delivering personalized content to millions of users efficiently.

It uses a mix of push and pull models, caching, and ranking to balance latency, freshness, and resource use.

Choosing the right feed generation approach depends on user scale, activity patterns, and latency requirements.

Practice

1. What is the main purpose of a news feed generation system in social media platforms?

easy

A. To store user passwords securely

B. To manage user account settings

C. To show personalized and timely content to users

D. To handle payment transactions

Solution

1. Step 1: Understand the role of news feed generation

   The news feed system is designed to deliver content that is relevant and timely to each user.

2. Step 2: Identify the correct purpose among options

   The other options relate to security, settings, and payments, which are unrelated to news feed generation.

3. Final Answer:

   To show personalized and timely content to users -> Option C

4. Quick Check:

   News feed = personalized timely content [OK]

Hint: News feed = personalized content display [OK]

Common Mistakes:

• Confusing news feed with user authentication
• Mixing news feed with payment processing
• Thinking news feed manages account settings

2. Which of the following is a common method used in news feed generation to deliver updates efficiently?

easy

A. Manual refresh by users only

B. Push model where updates are sent to users proactively

C. Storing all data in a single database without caching

D. Using FTP to transfer news feed data

Solution

1. Step 1: Identify common delivery methods in news feed systems

   Push model proactively sends updates to users, improving latency and experience.

2. Step 2: Evaluate other options for efficiency

   Manual refresh is user-driven and less efficient; no caching slows performance; FTP is unrelated to real-time feed delivery.

3. Final Answer:

   Push model where updates are sent to users proactively -> Option B

4. Quick Check:

   Push model = efficient update delivery [OK]

Hint: Push model proactively sends updates [OK]

Common Mistakes:

• Assuming manual refresh is efficient
• Ignoring caching benefits
• Confusing FTP with real-time data delivery

3. Consider a news feed system using a pull model where users request their feed on demand. What is a likely outcome when many users request feeds simultaneously?

medium

A. Users receive outdated feeds only

B. Instant delivery with no server load

C. System automatically caches all feeds without delay

D. High latency and increased load on backend servers

Solution

1. Step 1: Understand pull model behavior under load

   Pull model requires backend to generate feeds on request, causing high load if many users request simultaneously.

2. Step 2: Analyze other options for feasibility

   Instant delivery with no load is unrealistic; outdated feeds depend on caching, not pull model alone; automatic caching without delay is ideal but not guaranteed.

3. Final Answer:

   High latency and increased load on backend servers -> Option D

4. Quick Check:

   Pull model + many requests = high load [OK]

Hint: Pull model causes backend load spikes [OK]

Common Mistakes:

• Assuming pull model has no latency
• Confusing caching with pull model behavior
• Believing feeds are always instantly cached

4. A news feed system uses a push model but users report seeing stale content. Which is the most likely cause?

medium

A. Cache not invalidated after new content is pushed

B. Users are not refreshing their browsers

C. Backend servers are down

D. Users have slow internet connections

Solution

1. Step 1: Identify push model behavior and caching role

   Push model sends updates, but if cache is not invalidated, users see old content.

2. Step 2: Evaluate other options for staleness cause

   Browser refresh is less relevant in push; backend down causes no updates; slow internet delays but does not cause stale cached data.

3. Final Answer:

   Cache not invalidated after new content is pushed -> Option A

4. Quick Check:

   Push + stale feed = cache invalidation issue [OK]

Hint: Stale feed often means cache invalidation failed [OK]

Common Mistakes:

• Blaming user refresh instead of cache
• Ignoring cache invalidation importance
• Assuming slow internet causes stale cache

5. You are designing a news feed system for a platform with 100 million users. Which approach best balances scalability and freshness of content?

hard

A. Hybrid model: push important updates and pull less critical content

B. Pure pull model: generate feed on every user request

C. Pure push model: push all updates to all users immediately

D. No caching: always fetch fresh data from database

Solution

1. Step 1: Analyze scalability and freshness needs for large user base

   Pure pull causes high load; pure push is costly and complex; no caching is inefficient.

2. Step 2: Evaluate hybrid model benefits

   Hybrid model pushes critical updates for freshness and uses pull for less urgent content, balancing load and latency.

3. Final Answer:

   Hybrid model: push important updates and pull less critical content -> Option A

4. Quick Check:

   Hybrid model balances scale and freshness [OK]

Hint: Hybrid push-pull balances scale and freshness [OK]

Common Mistakes:

• Choosing pure pull causing backend overload
• Choosing pure push causing network overload
• Ignoring caching and load balancing