Search and filter design in LLD - System Design Guide

Problem Statement

Users trying to find specific items in large datasets face slow response times and irrelevant results when search and filter operations are not optimized. Without a structured approach, the system can become overwhelmed, leading to poor user experience and high resource consumption.

Solution

The system indexes data to enable fast search queries and applies filters to narrow down results efficiently. It separates concerns by handling search and filter logic in modular components, allowing scalable and maintainable code. The design uses query parsing, indexing, and filtering layers to process user requests quickly and accurately.

Architecture

User Query

→Query Parser

↓

↓

Filter Processor

↓

Result Formatter

Result Formatter

↓

User Display

User Display

This diagram shows the flow from user query input through parsing, searching indexed data, applying filters, formatting results, and finally displaying to the user.

Trade-offs

✓ Pros

→

Improves query response time by using indexed data structures.

→

Modular design allows easy addition or modification of filters.

→

Separates search and filter logic for better maintainability.

→

Supports complex queries with multiple filter conditions.

✗ Cons

→

Indexing large datasets requires additional storage and update overhead.

→

Complex filter logic can increase processing time if not optimized.

→

Initial implementation complexity is higher than simple linear search.

When the dataset size exceeds thousands of records and users require fast, relevant search results with multiple filtering options.

For small datasets under a few hundred records where simple linear search is sufficient and indexing overhead is unnecessary.

Real World Examples

Amazon

Uses advanced search and filter design to allow customers to quickly find products by keywords and apply multiple filters like price, brand, and ratings.

LinkedIn

Implements search and filter to help users find jobs or people by keywords and refine results with filters such as location, industry, and experience level.

Airbnb

Enables users to search listings and apply filters like price range, amenities, and property type to find suitable accommodations quickly.

Code Example

The before code mixes search and filter logic in a single loop, making it hard to extend or maintain. The after code separates search and filter into classes, allowing independent changes and clearer logic. This modular design supports adding more filters or changing search behavior easily.

LLD

### Before: Simple linear search and filter without modular design
items = [
    {"name": "Red Shirt", "color": "red", "size": "M"},
    {"name": "Blue Jeans", "color": "blue", "size": "L"},
    {"name": "Green Hat", "color": "green", "size": "S"}
]

query = "shirt"
color_filter = "red"

results = []
for item in items:
    if query.lower() in item["name"].lower() and item["color"] == color_filter:
        results.append(item)

print(results)

### After: Modular search and filter design
class SearchEngine:
    def __init__(self, items):
        self.items = items

    def search(self, query):
        return [item for item in self.items if query.lower() in item["name"].lower()]

class FilterProcessor:
    def __init__(self, filters):
        self.filters = filters

    def apply(self, items):
        filtered = items
        for key, value in self.filters.items():
            filtered = [item for item in filtered if item.get(key) == value]
        return filtered

items = [
    {"name": "Red Shirt", "color": "red", "size": "M"},
    {"name": "Blue Jeans", "color": "blue", "size": "L"},
    {"name": "Green Hat", "color": "green", "size": "S"}
]

search_engine = SearchEngine(items)
filter_processor = FilterProcessor({"color": "red"})

search_results = search_engine.search("shirt")
final_results = filter_processor.apply(search_results)

print(final_results)

Alternatives

Full-text search engines (e.g., Elasticsearch)

Uses distributed, scalable search clusters with advanced text analysis and ranking algorithms.

Use when: When the system requires high scalability, complex text search, and real-time indexing across massive datasets.

Database native filtering

Relies on database query capabilities without separate indexing or search layers.

Use when: When dataset size is moderate and the complexity of search and filter queries is low.

In-memory filtering

Loads data into memory and applies filters directly without persistent indexes.

Use when: When ultra-low latency is needed for small datasets that fit entirely in memory.

Summary

Search and filter design prevents slow and irrelevant results by structuring query processing.

It uses modular components to separate search logic from filtering for better scalability and maintainability.

This design is essential when datasets grow beyond thousands of records and users need fast, precise results.

Practice

1. What is the main purpose of adding filters in a search system?

easy

A. To slow down the search process for accuracy

B. To increase the total number of search results

C. To narrow down search results based on user preferences

D. To remove the search bar from the interface

Solution

1. Step 1: Understand the role of filters in search

   Filters help users reduce the number of results by selecting specific criteria.

2. Step 2: Identify the effect of filters on results

   Filters narrow results to match user preferences, making search faster and more relevant.

3. Final Answer:

   To narrow down search results based on user preferences -> Option C

4. Quick Check:

   Filters narrow results = C [OK]

Hint: Filters reduce results to match user needs quickly [OK]

Common Mistakes:

• Thinking filters increase results
• Assuming filters slow down search intentionally
• Confusing filters with UI removal

2. Which of the following is the correct way to represent a filter for price less than $100 in a query parameter?

easy

A. price>100

B. price<100

C. price=100

D. price!=100

Solution

1. Step 1: Understand comparison operators in queries

   The symbol '<' means less than, so 'price<100' filters prices below 100.

2. Step 2: Eliminate incorrect operators

   '>' means greater than, '=' means equal, '!=' means not equal, so they don't match 'less than 100'.

3. Final Answer:

   price<100 -> Option B

4. Quick Check:

   Less than operator = A [OK]

Hint: Use '<' for less than in filters [OK]

Common Mistakes:

• Using '>' instead of '<' for less than
• Confusing '=' with less than
• Using '!=' which means not equal

3. Consider a search system that indexes products by category and price. If a user searches with filters category='books' and price < 20, which data structure best supports fast filtering?

medium

A. A hash map keyed by category with sorted price lists

B. A single unsorted list of all products

C. A queue of products ordered by insertion time

D. A stack of products sorted by price

Solution

1. Step 1: Analyze filtering needs

   Filtering by category and price requires quick lookup by category and efficient price range queries.

2. Step 2: Choose data structure supporting these queries

   A hash map keyed by category allows fast category lookup; sorted price lists enable quick filtering by price.

3. Final Answer:

   A hash map keyed by category with sorted price lists -> Option A

4. Quick Check:

   Hash map + sorted list = B [OK]

Hint: Use hash map for categories and sorted lists for range filters [OK]

Common Mistakes:

• Using unsorted lists causing slow searches
• Using queue or stack which don't support efficient filtering
• Ignoring the need for sorting by price

4. A search filter system is returning incorrect results when filtering by date range. Which of the following is the most likely cause?

medium

A. Date values are stored as strings and compared lexicographically

B. The filter uses numeric comparison on date objects

C. The database index is on the wrong column

D. The search query is missing a filter parameter

Solution

1. Step 1: Understand date comparison issues

   Comparing dates stored as strings can cause wrong order because string comparison is lexicographic.

2. Step 2: Identify why this causes incorrect results

   Dates like '12/01/2023' and '02/12/2023' compared as strings may not sort correctly, causing wrong filter results.

3. Final Answer:

   Date values are stored as strings and compared lexicographically -> Option A

4. Quick Check:

   String date comparison causes errors = A [OK]

Hint: Store dates as date objects, not strings [OK]

Common Mistakes:

• Assuming numeric comparison works on strings
• Ignoring index relevance
• Thinking missing filter param causes wrong filtered results

5. You are designing a scalable search and filter system for an e-commerce site with millions of products. Which approach best balances fast search and flexible filtering?

hard

A. Load all products into memory and filter using loops

B. Store all products in a single SQL table and scan it for every search

C. Use a simple key-value store without indexes

D. Use a distributed search engine with inverted indexes and faceted filters

Solution

1. Step 1: Consider scalability and performance needs

   Millions of products require fast, scalable search with flexible filters.

2. Step 2: Evaluate options for search and filtering

   Distributed search engines with inverted indexes enable fast text search; faceted filters allow flexible attribute filtering efficiently.

3. Step 3: Eliminate inefficient approaches

   Scanning large SQL tables or in-memory filtering is slow and not scalable; key-value stores lack complex search capabilities.

4. Final Answer:

   Use a distributed search engine with inverted indexes and faceted filters -> Option D

5. Quick Check:

   Distributed search + faceted filters = D [OK]

Hint: Use distributed search with faceted filters for scale [OK]

Common Mistakes:

• Relying on full table scans for large data
• Ignoring indexing for search speed
• Using memory-heavy filtering for millions of items