Search and metadata in HLD - System Design Exercise

Design: Search and Metadata System

Design covers search functionality and metadata management for content items. Out of scope are user authentication, content creation UI, and analytics.

Functional Requirements

FR1: Allow users to search content using keywords and filters

FR2: Store and manage metadata for each content item (e.g., title, author, date, tags)

FR3: Support fast search response times (p99 < 300ms)

FR4: Enable filtering search results by metadata fields

FR5: Handle up to 100,000 concurrent search requests

FR6: Support adding, updating, and deleting content and metadata

FR7: Provide relevance ranking for search results

Non-Functional Requirements

NFR1: System must be highly available (99.9% uptime)

NFR2: Search index should be updated within 5 seconds of content changes

NFR3: Support horizontal scaling for both search and metadata storage

NFR4: Latency for search queries should be under 300ms at p99

NFR5: Metadata storage must ensure consistency for updates

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

❓ Question 6

Key Components

Search index engine (e.g., Elasticsearch, OpenSearch)

Metadata database (relational or NoSQL)

API gateway or search service

Indexing pipeline for content and metadata updates

Cache layer for frequent queries

Load balancer for scaling search requests

Design Patterns

Inverted index for full-text search

Event-driven indexing for near real-time updates

Cache aside pattern for metadata caching

Sharding and replication for scaling search index

Pagination and filtering in search results

Reference Architecture

Client
  |
  v
API Gateway / Search Service
  |
  +--> Cache (Redis)
  |
  +--> Search Engine (Elasticsearch Cluster)
  |
  +--> Metadata DB (PostgreSQL Cluster)

Indexing Pipeline:
Metadata DB --> Change Events --> Indexer --> Search Engine

Components

API Gateway / Search Service

Node.js or Java Spring Boot

Handles client search requests, applies filters, queries cache and search engine, returns results

Search Engine

Elasticsearch or OpenSearch

Stores inverted index for full-text search and metadata fields, performs fast search queries with ranking

Metadata Database

PostgreSQL

Stores authoritative metadata records with strong consistency

Cache

Redis

Caches frequent search queries and metadata to reduce latency and load

Indexing Pipeline

Kafka + Worker Services

Processes metadata changes from DB events and updates search engine index within 5 seconds

Request Flow

1. Client sends search request with keywords and filters to API Gateway

2. API Gateway checks Redis cache for cached results

3. If cache miss, API Gateway queries Elasticsearch with search and filter parameters

4. Elasticsearch returns ranked search results

5. API Gateway returns results to client and caches them in Redis

6. When metadata changes, DB emits change event to Kafka

7. Indexer service consumes event, updates Elasticsearch index accordingly

8. Metadata updates are stored in PostgreSQL with strong consistency

Database Schema

Entities: - ContentItem(id PK, title, author, created_at, updated_at, ...) - Metadata(id PK, content_item_id FK, key, value) Relationships: - ContentItem 1:N Metadata Metadata stores key-value pairs for flexible metadata fields. ContentItem stores core attributes.

Scaling Discussion

Bottlenecks

Search engine cluster CPU and memory limits under high query load

Metadata database write throughput for frequent updates

Cache size and eviction policy for popular queries

Indexer pipeline lag causing stale search results

Solutions

Scale search engine horizontally by adding shards and replicas

Use read replicas and partition metadata DB to improve write/read throughput

Implement cache eviction policies and increase Redis cluster size

Optimize indexing pipeline with parallel consumers and batch updates

Interview Tips

Time: 10 minutes for requirements and clarifications, 15 minutes for architecture and components, 10 minutes for data flow and database design, 10 minutes for scaling and trade-offs discussion

Clarify search and metadata requirements and constraints

Explain choice of search engine and metadata storage

Describe how indexing pipeline keeps search index fresh

Discuss caching strategy to reduce latency

Address scaling challenges and solutions

Mention trade-offs between consistency and availability

Practice

1. What is the primary purpose of metadata in a search system?

easy

A. To display images on the website

B. To store user passwords securely

C. To describe data and make search faster

D. To manage network connections

Solution

1. Step 1: Understand metadata role

   Metadata provides information about data, like tags or descriptions.

2. Step 2: Connect metadata to search

   Search engines use metadata to quickly find relevant data without scanning everything.

3. Final Answer:

   To describe data and make search faster -> Option C

4. Quick Check:

   Metadata = Data description for search [OK]

Hint: Metadata helps find data faster by describing it [OK]

Common Mistakes:

• Confusing metadata with user data
• Thinking metadata stores passwords
• Assuming metadata manages network

2. Which of the following is a correct example of metadata used in search?

easy

A. title = 'Introduction to Cats'

B. file_size = 2048

C. user_password = '1234'

D. connection_timeout = 30

Solution

1. Step 1: Identify metadata examples

   Metadata describes content, like titles, tags, or dates.

2. Step 2: Check each option

   title = 'Introduction to Cats' shows a title, which is metadata describing content. Others are config or sensitive data.

3. Final Answer:

   title = 'Introduction to Cats' -> Option A

4. Quick Check:

   Title is metadata for search [OK]

Hint: Metadata describes content, not configs or passwords [OK]

Common Mistakes:

• Choosing config values as metadata
• Confusing sensitive data with metadata
• Ignoring descriptive fields

3. Given a search system with metadata index, what is the expected output when searching for "apple" if metadata contains {"title": "apple pie", "tags": ["fruit", "dessert"]}?

medium

A. No results found

B. Returns item with title "apple pie"

C. Returns all items with tag "fruit" only

D. Returns items with tag "dessert" only

Solution

1. Step 1: Understand search with metadata

   Search looks for matches in metadata fields like title and tags.

2. Step 2: Check if "apple" matches metadata

   "apple" matches the title "apple pie", so the item is returned.

3. Final Answer:

   Returns item with title "apple pie" -> Option B

4. Quick Check:

   Search matches title containing "apple" [OK]

Hint: Search matches metadata fields containing query word [OK]

Common Mistakes:

• Ignoring title field in search
• Returning unrelated tags only
• Assuming no results if exact match missing

4. A search system's metadata index is not returning expected results. Which issue below is most likely the cause?

medium

A. Database password is incorrect

B. User interface colors are dull

C. Network cables are unplugged

D. Metadata is not updated after data changes

Solution

1. Step 1: Identify cause of search failure

   If metadata is stale, search index won't reflect latest data.

2. Step 2: Evaluate options

   Only Metadata is not updated after data changes relates to metadata and search correctness; others are unrelated.

3. Final Answer:

   Metadata is not updated after data changes -> Option D

4. Quick Check:

   Stale metadata breaks search results [OK]

Hint: Keep metadata updated to ensure correct search [OK]

Common Mistakes:

• Blaming UI or network for search logic errors
• Ignoring metadata update process
• Confusing unrelated system issues

5. You are designing a scalable search system for millions of users. Which approach best ensures fast search using metadata?

hard

A. Use distributed indexing with metadata shards and update indexes asynchronously

B. Store metadata in a centralized database and scan all records on each search

C. Keep metadata only on user devices and search locally

D. Disable metadata to reduce storage and search raw data only

Solution

1. Step 1: Understand scalability needs

   Millions of users require fast, distributed search to avoid bottlenecks.

2. Step 2: Evaluate options for scalability

   Use distributed indexing with metadata shards and update indexes asynchronously uses distributed indexing and async updates, which scales well and keeps search fast.

3. Final Answer:

   Use distributed indexing with metadata shards and update indexes asynchronously -> Option A

4. Quick Check:

   Distributed indexing + async updates = scalable search [OK]

Hint: Distribute metadata index and update asynchronously for scale [OK]

Common Mistakes:

• Scanning all data centrally causes slow search
• Relying on local device metadata limits scale
• Disabling metadata removes search efficiency