Database indexing in HLD - System Design Guide

Problem Statement

When a database grows large, searching for data without any shortcuts means scanning every record. This causes slow query responses and high resource use, making the system inefficient and frustrating for users.

Solution

Database indexing creates a special data structure that acts like a fast lookup guide. Instead of scanning all records, the database uses the index to quickly find the location of the desired data, speeding up queries significantly.

Architecture

User Query

↓

Index Structure

Index Structure

↓

Data Records

Data Records

This diagram shows how a user query is first directed to the database, which uses the index structure to quickly locate the data records without scanning the entire dataset.

Trade-offs

✓ Pros

→

Speeds up data retrieval by avoiding full table scans.

→

Reduces CPU and I/O load during queries.

→

Improves user experience with faster response times.

→

Supports efficient sorting and filtering operations.

✗ Cons

→

Consumes extra storage space for the index data structures.

→

Slows down write operations like inserts, updates, and deletes due to index maintenance.

→

Requires careful design to avoid unnecessary or redundant indexes.

Use indexing when the database has large tables with frequent read queries that filter or sort on specific columns, especially when query response time is critical.

Avoid indexing small tables with mostly write operations or when the query patterns do not benefit from indexes, as the overhead may outweigh the benefits.

Real World Examples

Amazon

Amazon uses indexing on product catalogs to quickly retrieve items matching search filters and sort orders, enabling fast and relevant search results.

Twitter

Twitter indexes tweets by user and hashtags to efficiently serve timelines and search queries without scanning all tweets.

LinkedIn

LinkedIn applies indexing on user profiles and connections to speed up complex queries for recommendations and network searches.

Alternatives

Full Table Scan

Reads every record sequentially without any shortcut or index.

Use when: Choose when tables are very small or queries retrieve most of the data, making indexing overhead unnecessary.

Materialized Views

Precomputes and stores query results for fast access instead of indexing base tables.

Use when: Choose when queries are complex aggregations or joins that benefit from precomputed results.

Caching

Stores frequently accessed query results in fast memory rather than indexing the database.

Use when: Choose when read patterns are highly repetitive and data freshness requirements allow caching.

Summary

Database indexing creates fast lookup structures to avoid scanning all records during queries.

Indexes improve read performance but add storage cost and slow down writes due to maintenance.

Proper indexing depends on data size, query patterns, and balancing read/write trade-offs.

Practice

1. What is the primary purpose of a database index?

easy

A. To increase the size of the database

B. To speed up data retrieval by allowing quick lookups

C. To store backup copies of data

D. To encrypt the database contents

Solution

1. Step 1: Understand what an index does

   An index helps the database find data faster by creating a quick lookup structure.

2. Step 2: Identify the main benefit

   This quick lookup reduces the time needed to search large tables.

3. Final Answer:

   To speed up data retrieval by allowing quick lookups -> Option B

4. Quick Check:

   Index = Faster data search [OK]

Hint: Indexes speed up searches by creating shortcuts [OK]

Common Mistakes:

• Thinking indexes store extra data copies
• Confusing indexes with backups
• Assuming indexes encrypt data

2. Which of the following is the correct syntax to create an index on the column user_id in SQL?

easy

A. CREATE INDEX idx_user ON users (user_id);

B. INDEX CREATE user_id ON users;

C. CREATE users INDEX ON user_id;

D. MAKE INDEX user_id IN users;

Solution

1. Step 1: Recall SQL syntax for creating indexes

   The standard syntax is CREATE INDEX index_name ON table_name (column_name);

2. Step 2: Match the syntax with options

   CREATE INDEX idx_user ON users (user_id); matches the correct syntax exactly.

3. Final Answer:

   CREATE INDEX idx_user ON users (user_id); -> Option A

4. Quick Check:

   CREATE INDEX ... ON ... (column) [OK]

Hint: Remember: CREATE INDEX name ON table(column) [OK]

Common Mistakes:

• Swapping table and column names
• Using wrong keyword order
• Missing parentheses around column

3. Given a table orders with an index on order_date, what will be the output of the query plan for SELECT * FROM orders WHERE order_date = '2023-01-01';?

medium

A. Full table scan without using the index

B. Ignoring the order_date column completely

C. Sorting all rows by order_date before filtering

D. Index seek on order_date to quickly find matching rows

Solution

1. Step 1: Understand how indexes affect query plans

   An index on order_date allows the database to quickly locate rows matching the date.

2. Step 2: Identify the expected query plan

   The database will perform an index seek to find matching rows efficiently, not scan the whole table.

3. Final Answer:

   Index seek on order_date to quickly find matching rows -> Option D

4. Quick Check:

   Indexed column = index seek [OK]

Hint: Indexed WHERE columns use index seek, not full scan [OK]

Common Mistakes:

• Assuming full table scan despite index
• Confusing sorting with filtering
• Ignoring index usage in query plan

4. A developer created an index on a column but notices no improvement in query speed. What could be a likely reason?

medium

A. The database automatically disables all indexes

B. The index was created with the wrong SQL syntax

C. The queries do not filter or sort using the indexed column

D. Indexes always slow down queries

Solution

1. Step 1: Analyze index usage conditions

   Indexes only help if queries use the indexed column in WHERE, JOIN, or ORDER BY clauses.

2. Step 2: Identify why no speed improvement occurs

   If queries don't use the indexed column, the index is ignored, so no speed gain.

3. Final Answer:

   The queries do not filter or sort using the indexed column -> Option C

4. Quick Check:

   Index helps only if column used in query [OK]

Hint: Index unused if queries ignore indexed columns [OK]

Common Mistakes:

• Assuming syntax error disables index
• Believing all indexes slow queries
• Thinking database disables indexes by default

5. You have a large table with frequent inserts and queries filtering by status and created_at. Which indexing strategy balances read speed and write overhead best?

hard

A. Create a composite index on (status, created_at)

B. Create separate indexes on status and created_at individually

C. Create no indexes to avoid write overhead

D. Create a full-text index on status and created_at

Solution

1. Step 1: Understand composite vs separate indexes

   A composite index on (status, created_at) helps queries filtering on both columns efficiently.

2. Step 2: Consider write overhead and query patterns

   Separate indexes increase write overhead more than one composite index and may be less efficient for combined filters.

3. Step 3: Exclude unsuitable options

   No indexes hurt read speed; full-text index is for text search, not filtering by status/date.

4. Final Answer:

   Create a composite index on (status, created_at) -> Option A

5. Quick Check:

   Composite index = balanced read/write [OK]

Hint: Composite indexes optimize multi-column filters with less overhead [OK]

Common Mistakes:

• Creating many separate indexes causing slow writes
• Skipping indexes hurting read performance
• Using full-text index for non-text filtering