Search and recommendation in HLD - System Design Exercise

Design: Search and Recommendation System

Design covers backend architecture for search and recommendation features including data ingestion, indexing, query processing, recommendation engine, and APIs. Frontend UI and detailed machine learning model design are out of scope.

Functional Requirements

FR1: Allow users to search for items using keywords with fast response times

FR2: Provide personalized recommendations based on user behavior and preferences

FR3: Support at least 1 million daily active users

FR4: Search results should be relevant and ranked by relevance score

FR5: Recommendations should update in near real-time as user data changes

FR6: Support filtering and sorting options in search results

FR7: Provide analytics on search queries and recommendation effectiveness

Non-Functional Requirements

NFR1: Search API latency p99 < 200ms

NFR2: Recommendation updates within 5 minutes of user activity

NFR3: System availability 99.9% uptime

NFR4: Handle peak traffic of 10,000 concurrent search requests

NFR5: Data storage must support fast read and write operations

NFR6: Ensure user privacy and data security compliance

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

❓ Question 6

❓ Question 7

Key Components

Search index (e.g., Elasticsearch, Solr)

User behavior tracking and data storage

Recommendation engine (collaborative filtering, content-based)

API gateway and load balancer

Caching layer for frequent queries

Analytics and monitoring tools

Design Patterns

Inverted index for search

Batch and real-time data processing

Personalization with user profiles

Cache-aside pattern for search results

Event-driven architecture for updates

A/B testing for recommendation algorithms

Reference Architecture

Client
  |
  v
API Gateway / Load Balancer
  |
  +-----------------------------+
  |                             |
Search Service             Recommendation Service
  |                             |
Search Index (Elasticsearch)  User Profile DB
  |                             |
Cache Layer (Redis)          Behavior Data Store
  |                             |
Analytics & Monitoring System

Components

API Gateway / Load Balancer

Nginx / AWS ALB

Distribute incoming requests and route to appropriate services

Search Service

Elasticsearch

Process search queries using inverted index and return ranked results

Recommendation Service

Custom microservice with ML models

Generate personalized recommendations based on user data

Cache Layer

Redis

Cache frequent search queries and recommendation results for low latency

User Profile DB

PostgreSQL / NoSQL (e.g., Cassandra)

Store user preferences and profile data for personalization

Behavior Data Store

Kafka + HDFS / Data Lake

Collect and store user activity data for batch and real-time processing

Analytics & Monitoring System

Prometheus + Grafana

Track system health, query patterns, and recommendation effectiveness

Request Flow

1. User sends search query or requests recommendations via client app.

2. API Gateway routes request to Search Service or Recommendation Service.

3. Search Service checks cache for query results; if miss, queries Elasticsearch index.

4. Search results are ranked and returned to client; results cached for future requests.

5. Recommendation Service fetches user profile and recent behavior data.

6. Runs recommendation algorithms to generate personalized item list.

7. Recommendations cached and returned to client.

8. User interactions are logged and sent to Behavior Data Store via event stream.

9. Batch jobs update user profiles and retrain recommendation models periodically.

10. Analytics system collects metrics and logs for monitoring and improvements.

Database Schema

Entities: - User: user_id (PK), name, preferences, demographics - Item: item_id (PK), title, description, category, metadata - SearchIndex: inverted index structure managed by Elasticsearch - UserBehavior: event_id (PK), user_id (FK), item_id (FK), action_type, timestamp - RecommendationModel: model_id (PK), version, parameters, last_trained Relationships: - UserBehavior links User and Item with actions - UserProfile stores preferences linked to User - SearchIndex indexes Item data for fast retrieval

Scaling Discussion

Bottlenecks

Search index query latency under high concurrent load

Recommendation engine compute time for large user base

Cache invalidation and consistency with frequent data updates

Data ingestion pipeline throughput for user behavior events

Database read/write throughput for user profiles

Solutions

Shard and replicate search index to distribute query load

Use approximate nearest neighbor search and model caching for recommendations

Implement cache-aside pattern with TTL and event-driven invalidation

Use scalable message queues (Kafka) and stream processing for data ingestion

Partition user profile DB and use read replicas to scale reads

Interview Tips

Time: Spend 10 minutes clarifying requirements and constraints, 20 minutes designing architecture and data flow, 10 minutes discussing scaling and trade-offs, 5 minutes summarizing.

Clarify user needs and data freshness requirements

Explain choice of search index and recommendation algorithms

Describe caching strategy to meet latency goals

Discuss data pipeline for user behavior and model updates

Highlight scalability challenges and solutions

Mention monitoring and analytics for continuous improvement

Practice

1. What is the primary purpose of a search system in a large-scale application?

easy

A. To help users quickly find relevant content from a large dataset

B. To store user passwords securely

C. To manage user account settings

D. To display advertisements randomly

Solution

1. Step 1: Understand the role of search systems

   Search systems are designed to help users find information efficiently from large amounts of data.

2. Step 2: Match the purpose with options

   Only To help users quickly find relevant content from a large dataset describes helping users find relevant content quickly, which is the core function of search.

3. Final Answer:

   To help users quickly find relevant content from a large dataset -> Option A

4. Quick Check:

   Search system purpose = find relevant content [OK]

Hint: Search systems focus on finding relevant data fast [OK]

Common Mistakes:

• Confusing search with unrelated features like password storage
• Thinking search manages user settings
• Assuming search is for random content display

2. Which component is essential in a recommendation system to personalize suggestions?

easy

A. Database backup scripts

B. User behavior tracking

C. Static HTML pages

D. Load balancer configuration

Solution

1. Step 1: Identify personalization needs

   Recommendation systems personalize suggestions based on user data and behavior.

2. Step 2: Match components to personalization

   User behavior tracking collects data needed to tailor recommendations, unlike static pages or infrastructure tasks.

3. Final Answer:

   User behavior tracking -> Option B

4. Quick Check:

   Personalization needs user data = User behavior tracking [OK]

Hint: Personalization needs user data collection [OK]

Common Mistakes:

• Confusing infrastructure tasks with personalization
• Thinking static pages can personalize content
• Ignoring the role of user data

3. Consider a search system that indexes 1 million documents. If the system uses an inverted index, what is the main advantage?

medium

A. Automatically deleting old documents

B. Storing documents in a single large file

C. Encrypting all documents for security

D. Faster search queries by mapping words to document lists

Solution

1. Step 1: Understand inverted index concept

   An inverted index maps each word to the list of documents containing it, enabling quick lookups.

2. Step 2: Identify the advantage for search speed

   This mapping allows the system to find relevant documents quickly without scanning all documents.

3. Final Answer:

   Faster search queries by mapping words to document lists -> Option D

4. Quick Check:

   Inverted index = fast word-to-doc lookup [OK]

Hint: Inverted index speeds up word-based search [OK]

Common Mistakes:

• Confusing indexing with storage format
• Thinking encryption is the main index benefit
• Assuming index deletes documents automatically

4. A recommendation system is returning irrelevant suggestions. Which issue is most likely causing this?

medium

A. Using HTTPS instead of HTTP

B. Too many servers in the cluster

C. Incorrect or missing user behavior data

D. Database backup frequency is too high

Solution

1. Step 1: Analyze cause of irrelevant recommendations

   Recommendations depend on accurate user data; missing or wrong data leads to poor suggestions.

2. Step 2: Evaluate other options

   Server count, protocol choice, or backup frequency do not directly affect recommendation relevance.

3. Final Answer:

   Incorrect or missing user behavior data -> Option C

4. Quick Check:

   Bad recommendations = bad user data [OK]

Hint: Check user data quality for recommendation issues [OK]

Common Mistakes:

• Blaming infrastructure instead of data quality
• Confusing network protocols with recommendation logic
• Ignoring data collection importance

5. You are designing a scalable recommendation system for millions of users. Which approach best balances personalization and system performance?

hard

A. Use a hybrid model combining collaborative filtering and content-based filtering with offline batch processing and online updates

B. Only recommend the most popular items to all users without personalization

C. Run real-time deep learning models for every user request without caching

D. Store all user data in a single database server for simplicity

Solution

1. Step 1: Understand scalability and personalization needs

   Millions of users require efficient processing; personalization improves user experience.

2. Step 2: Evaluate approaches

   Hybrid models combine strengths of different methods. Offline batch processing reduces load, while online updates keep recommendations fresh.

3. Step 3: Reject less scalable or less personalized options

   Popular-only recommendations lack personalization. Real-time deep learning per request is costly. Single DB server is a bottleneck.

4. Final Answer:

   Use a hybrid model combining collaborative filtering and content-based filtering with offline batch processing and online updates -> Option A

5. Quick Check:

   Hybrid + batch + online = scalable personalized system [OK]

Hint: Combine offline and online methods for scalable personalization [OK]

Common Mistakes:

• Ignoring scalability by doing all processing online
• Sacrificing personalization for simplicity
• Using single server for massive data