Design a rate limiter in HLD - Scalability & System Analysis

Scalability Analysis - Design a rate limiter

Growth Table: Rate Limiter Scaling

Users	Requests per Second (RPS)	Storage Needs	Latency Impact	System Changes
100 users	~1,000 RPS	Minimal (in-memory counters)	Low latency	Single server with in-memory rate limiting
10,000 users	~100,000 RPS	Moderate (distributed cache)	Low to moderate latency	Distributed cache (e.g., Redis cluster), load balancer
1,000,000 users	~10,000,000 RPS	High (sharded cache and DB)	Moderate latency	Sharded caches, multiple rate limiter instances, consistent hashing
100,000,000 users	~1,000,000,000 RPS	Very high (multi-region, sharded storage)	Higher latency possible	Global distributed rate limiting, hierarchical limits, CDN edge enforcement

First Bottleneck

The first bottleneck is the storage and update of counters that track user requests. At low scale, in-memory counters on a single server work well. As users and requests grow, the rate limiter's data store (like Redis or database) becomes the bottleneck because it must handle many read/write operations per second with low latency.

Scaling Solutions

• Horizontal scaling: Add more rate limiter instances behind a load balancer to distribute traffic.
• Distributed caching: Use Redis clusters or similar to store counters with fast access.
• Sharding: Partition counters by user ID or API key to spread load across multiple nodes.
• Token bucket or leaky bucket algorithms: Use efficient algorithms to reduce storage and computation.
• Local caching with periodic sync: Cache counters locally and sync with central store to reduce writes.
• CDN edge enforcement: For global scale, enforce limits closer to users to reduce central load.

Back-of-Envelope Cost Analysis

• At 10,000 users with 10 RPS each = 100,000 RPS total.
• Each request requires 1 read + 1 write to counter -> 200,000 ops/sec on data store.
• Redis single instance handles ~100,000 ops/sec -> need 2+ Redis nodes or cluster.
• Storage: counters are small (few bytes each), but millions of users require efficient memory use.
• Network bandwidth: assuming 1 KB per request metadata, 100,000 RPS = ~100 MB/s bandwidth.

Interview Tip

Start by explaining the rate limiter's purpose and basic algorithm (fixed window, sliding window, token bucket). Then discuss expected traffic and identify bottlenecks. Propose scaling solutions step-by-step, focusing on data store limits and latency. Mention trade-offs like accuracy vs. performance. Finally, consider global scale and edge enforcement.

Self Check

Your database handles 1000 QPS. Traffic grows 10x to 10,000 QPS. What do you do first?

Answer: Add read replicas and implement caching to reduce database load. Also, consider sharding counters or moving counters to a faster in-memory store like Redis to handle increased QPS.

Key Result

The main bottleneck in a rate limiter is the fast, frequent update of counters in the data store. Scaling requires distributing counters across multiple nodes, using caching, and efficient algorithms to maintain low latency under high request rates.

Practice

1. What is the primary purpose of a rate limiter in system design?

easy

A. To control the number of requests a user can make in a given time

B. To increase the speed of database queries

C. To store user data securely

D. To balance load between multiple servers

Solution

1. Step 1: Understand the role of rate limiter

   A rate limiter restricts how many requests a user or client can send in a certain time to prevent overload.

2. Step 2: Identify the correct purpose

   Among the options, only controlling request rate matches the rate limiter's function.

3. Final Answer:

   To control the number of requests a user can make in a given time -> Option A

4. Quick Check:

   Rate limiter = control request rate [OK]

Hint: Rate limiter limits requests per time window [OK]

Common Mistakes:

• Confusing rate limiter with load balancer
• Thinking it speeds up database queries
• Assuming it stores user data

2. Which data structure is most suitable to implement a sliding window rate limiter?

easy

A. Stack

B. Hash Map

C. Queue

D. Binary Tree

Solution

1. Step 1: Recall sliding window mechanism

   Sliding window rate limiter tracks timestamps of requests in a time window, removing old ones as time moves.

2. Step 2: Choose data structure for efficient insert and remove

   A queue allows adding new timestamps at the end and removing old timestamps from the front efficiently, matching sliding window needs.

3. Final Answer:

   Queue -> Option C

4. Quick Check:

   Sliding window = queue for timestamps [OK]

Hint: Sliding window needs FIFO structure like queue [OK]

Common Mistakes:

• Using stack which is LIFO, not suitable
• Choosing hash map without order
• Picking binary tree which is complex here

3. Consider a rate limiter allowing 3 requests per 10 seconds using sliding window. If requests come at seconds 1, 3, 7, and 9, which request will be rejected?

medium

A. Request at second 9

B. Request at second 3

C. Request at second 7

D. Request at second 1

Solution

1. Step 1: Track requests in 10-second window

   Requests at 1, 3, 7 are allowed as they are within limit 3 per 10 seconds.

2. Step 2: Check request at second 9

   At second 9, previous requests at 1, 3, 7 are still within 10 seconds window (from -1 to 9). So 3 requests already made, this 4th request exceeds limit and is rejected.

3. Final Answer:

   Request at second 9 -> Option A

4. Quick Check:

   4th request in 10s window = rejected [OK]

Hint: Count requests in last 10 seconds; 4th exceeds limit [OK]

Common Mistakes:

• Ignoring requests older than 10 seconds
• Allowing all requests without limit
• Counting requests incorrectly

4. A rate limiter uses a fixed window counter but sometimes allows bursts of requests at window edges. What is the likely cause?

medium

A. Sliding window algorithm is used

B. Queue data structure is not used

C. Rate limit is set too low

D. Fixed window resets counters abruptly causing bursts

Solution

1. Step 1: Understand fixed window behavior

   Fixed window counts requests in fixed intervals, resetting count at window end.

2. Step 2: Identify burst cause

   Requests near end of one window and start of next can both be allowed, causing bursts.

3. Final Answer:

   Fixed window resets counters abruptly causing bursts -> Option D

4. Quick Check:

   Fixed window reset causes bursts [OK]

Hint: Fixed window resets cause bursts at edges [OK]

Common Mistakes:

• Confusing sliding window with fixed window
• Blaming rate limit value instead of algorithm
• Ignoring window reset behavior

5. You need to design a distributed rate limiter for millions of users with low latency. Which approach best balances accuracy and scalability?

hard

A. Centralized fixed window counter on a single server

B. Distributed sliding window using local caches and periodic sync

C. Per-user token bucket stored only in client devices

D. No rate limiting, rely on server hardware scaling

Solution

1. Step 1: Consider scalability and accuracy needs

   Millions of users require distributed design to avoid bottlenecks and reduce latency.

2. Step 2: Evaluate options

   Centralized fixed window causes bottleneck; client-only token bucket is insecure; no rate limiting risks overload. Distributed sliding window with local caches and sync balances accuracy and scalability.

3. Final Answer:

   Distributed sliding window using local caches and periodic sync -> Option B

4. Quick Check:

   Distributed sliding window = scalable + accurate [OK]

Hint: Use distributed sliding window with local caches [OK]

Common Mistakes:

• Choosing centralized approach causing bottlenecks
• Relying on client-only enforcement
• Ignoring rate limiting and risking overload