Search and filter design in LLD - System Design Guide

Problem Statement

Users trying to find specific items in large datasets face slow response times and irrelevant results when search and filter operations are not optimized. Without a structured approach, the system can become overwhelmed, leading to poor user experience and high resource consumption.

Solution

The system indexes data to enable fast search queries and applies filters to narrow down results efficiently. It separates concerns by handling search and filter logic in modular components, allowing scalable and maintainable code. The design uses query parsing, indexing, and filtering layers to process user requests quickly and accurately.

Architecture

User Query

→Query Parser

↓

↓

Filter Processor

↓

Result Formatter

Result Formatter

↓

User Display

User Display

This diagram shows the flow from user query input through parsing, searching indexed data, applying filters, formatting results, and finally displaying to the user.

Trade-offs

✓ Pros

→

Improves query response time by using indexed data structures.

→

Modular design allows easy addition or modification of filters.

→

Separates search and filter logic for better maintainability.

→

Supports complex queries with multiple filter conditions.

✗ Cons

→

Indexing large datasets requires additional storage and update overhead.

→

Complex filter logic can increase processing time if not optimized.

→

Initial implementation complexity is higher than simple linear search.

When the dataset size exceeds thousands of records and users require fast, relevant search results with multiple filtering options.

For small datasets under a few hundred records where simple linear search is sufficient and indexing overhead is unnecessary.

Real World Examples

Amazon

Uses advanced search and filter design to allow customers to quickly find products by keywords and apply multiple filters like price, brand, and ratings.

LinkedIn

Implements search and filter to help users find jobs or people by keywords and refine results with filters such as location, industry, and experience level.

Airbnb

Enables users to search listings and apply filters like price range, amenities, and property type to find suitable accommodations quickly.

Code Example

The before code mixes search and filter logic in a single loop, making it hard to extend or maintain. The after code separates search and filter into classes, allowing independent changes and clearer logic. This modular design supports adding more filters or changing search behavior easily.

LLD

### Before: Simple linear search and filter without modular design
items = [
    {"name": "Red Shirt", "color": "red", "size": "M"},
    {"name": "Blue Jeans", "color": "blue", "size": "L"},
    {"name": "Green Hat", "color": "green", "size": "S"}
]

query = "shirt"
color_filter = "red"

results = []
for item in items:
    if query.lower() in item["name"].lower() and item["color"] == color_filter:
        results.append(item)

print(results)

### After: Modular search and filter design
class SearchEngine:
    def __init__(self, items):
        self.items = items

    def search(self, query):
        return [item for item in self.items if query.lower() in item["name"].lower()]

class FilterProcessor:
    def __init__(self, filters):
        self.filters = filters

    def apply(self, items):
        filtered = items
        for key, value in self.filters.items():
            filtered = [item for item in filtered if item.get(key) == value]
        return filtered

items = [
    {"name": "Red Shirt", "color": "red", "size": "M"},
    {"name": "Blue Jeans", "color": "blue", "size": "L"},
    {"name": "Green Hat", "color": "green", "size": "S"}
]

search_engine = SearchEngine(items)
filter_processor = FilterProcessor({"color": "red"})

search_results = search_engine.search("shirt")
final_results = filter_processor.apply(search_results)

print(final_results)

Alternatives

Full-text search engines (e.g., Elasticsearch)

Uses distributed, scalable search clusters with advanced text analysis and ranking algorithms.

Use when: When the system requires high scalability, complex text search, and real-time indexing across massive datasets.

Database native filtering

Relies on database query capabilities without separate indexing or search layers.

Use when: When dataset size is moderate and the complexity of search and filter queries is low.

In-memory filtering

Loads data into memory and applies filters directly without persistent indexes.

Use when: When ultra-low latency is needed for small datasets that fit entirely in memory.

Summary

Search and filter design prevents slow and irrelevant results by structuring query processing.

It uses modular components to separate search logic from filtering for better scalability and maintainability.

This design is essential when datasets grow beyond thousands of records and users need fast, precise results.