Bird
Raised Fist0

Rate Limiting at Different Layers - API Gateway, Service, Database

HLDmediumInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🏗️
Rate Limiting at Different Layers - API Gateway, Service, Database
Design a scalable rate limiting system enforcing limits at API Gateway, service, and database layers
mediumRate LimitingFreq: high
Prerequisites:Token Bucket AlgorithmRedisDistributed CachingAPI Gateway Concepts
⚙️
Functional Requirements
  • Enforce per-user and per-API rate limits at the API Gateway layer to reject excessive requests early.
  • Apply fine-grained rate limits within services for specific operations or user tiers.
  • Implement database-level rate limiting to prevent excessive writes or reads per user or entity.
  • Support burst traffic handling using token bucket algorithm with refill rates and burst capacity.
  • Provide real-time feedback to clients on rate limit status (e.g., headers with remaining quota).
  • Allow dynamic configuration of rate limits per API, user, or service.
📊
Non-Functional Requirements
  • Support 2 billion daily active users generating approximately 40 requests per user per day.
  • Achieve 99.99% availability for rate limiting checks with sub-100ms P99 latency on request path.
  • Handle peak write QPS of approximately 925,000 requests per second at API Gateway.
  • Ensure rate limit state storage supports TTL and automatic expiry to avoid stale data buildup.
  • Scale horizontally with minimal coordination overhead to avoid bottlenecks.
❓
Clarifying Questions to Ask
  • What are the exact rate limit policies? Are they per user, per API key, or per IP?
  • Should rate limits be global or vary by user tier or API endpoint?
  • Is burst traffic allowed, and what is the maximum burst size?
  • What is the expected read/write ratio for rate limit state queries and updates?
  • Are rate limits enforced strictly or is some leeway allowed (soft limits)?
  • How quickly must rate limit configuration changes propagate to all nodes?
Out of Scope
✗ Authentication and authorization mechanisms.
✗ Detailed billing or quota charging systems.
✗ Complex user behavior analytics or anomaly detection.
✗ Rate limiting for internal admin or monitoring APIs.
📌
Assumptions
  • 2 billion daily active users (DAU)
  • Each user generates 40 API requests per day on average
  • Average request size is 100 bytes
  • Retention period for rate limit logs is 7 days
⚡
QPS Calculation
write qps2B × 40 / 86400 ≈ 925,926 writes/sec
peak write qpsAssuming 3× peak factor: 925,926 × 3 ≈ 2.78M writes/sec
read qpsReads for rate limit checks assumed equal to writes: ~2.78M reads/sec at peak
💾
Storage Estimate
daily storage bytes2B × 40 × 100 bytes = 8 TB/day
retention storage bytes8 TB × 7 days = 56 TB
with replication56 TB × 3 replicas = 168 TB total storage
🖥️
Server Estimate
api gateway nodesAssuming 10k RPS per node, 2.78M / 10k ≈ 278 nodes
redis cluster nodesAssuming 100k ops/sec per Redis shard, 2.78M / 100k ≈ 28 shards
🏗️
Architecture Diagram
Architecture Diagram
          +----------------+
          |   Clients      |
          +-------+--------+
                  |
                  v
          +-------+--------+
          |   API Gateway  |
          | (Rate Limiter) |
          +-------+--------+
                  |
                  v
          +-------+--------+
          |   Service(s)   |
          | (Rate Limiter) |
          +-------+--------+
                  |
                  v
          +-------+--------+
          |   Database     |
          | (Rate Limiter) |
          +----------------+
                  |
                  v
          +----------------+
          |  Redis Cluster |
          | (Token Buckets)|
          +----------------+
🧩
Components
API Gateway Rate Limiter
Enforces coarse-grained per-user and per-API rate limits early to reject excessive requests
Why thisImplemented as a lightweight Redis-backed token bucket to achieve sub-10ms latency and reduce load downstream
Why notUsing in-memory local caches fails at scale due to lack of global state and causes inconsistent enforcement
Service Layer Rate Limiter
Applies fine-grained limits per operation or user tier, allowing differentiated throttling
Why thisUses Redis with Lua scripts for atomic token bucket updates ensuring consistency and low latency within service boundaries
Why notRelational DB rate limiting rejected due to high write latency and poor horizontal scalability at millions QPS
Database Layer Rate Limiter
Prevents abusive write/read patterns at the storage layer to protect database performance and integrity
Why thisImplemented via lightweight middleware with Redis caching to quickly reject over-limit requests before DB writes
Why notRelying solely on DB constraints or triggers is too slow and causes high contention under load
Redis Cluster
Stores token bucket states with TTL for automatic expiry, supporting atomic increments and decrements
Why thisRedis supports sub-millisecond lookups, native TTL, and Lua scripting for atomic operations at scale
Why notMemcached lacks TTL and atomic increment/decrement operations needed for token bucket algorithm
Configuration Management Service
Manages dynamic rate limit policies and propagates changes to API Gateway and services
Why thisUses distributed config store (e.g., Consul or etcd) for strong consistency and fast propagation
Why notManual config reload or polling causes stale policies and inconsistent enforcement
🔄
Data Flow
1
Client → API Gateway
Client sends API request
API Gateway performs initial rate limit check using Redis token bucket
2
API Gateway → Redis Cluster
Token bucket decrement for user/API key
Atomic Lua script ensures consistent token consumption
3
API Gateway → Service
Forward request if within limits
Reject early if limit exceeded to save downstream resources
4
Service → Redis Cluster
Fine-grained rate limit check per operation
Service-level token bucket enforces operation-specific limits
5
Service → Database
Request database operation if allowed
Database layer rate limiter middleware checks write/read limits
6
Database → Service
Return operation result
Database enforces data integrity and returns success/failure
7
Service → Client
Return response with rate limit headers
Headers include remaining tokens and reset time

Practice

(1/5)
1. In a fraud detection system ingesting thousands of transactions per second, which component is best suited to decouple the high-throughput transaction ingestion from downstream real-time scoring to avoid backpressure?
easy
A. Use a relational database with batch inserts to buffer transactions
B. Use direct synchronous HTTP calls from ingestion to scoring service
C. Use an in-memory cache like Redis to store incoming transactions temporarily
D. Use a distributed message queue like Kafka with partitioning by user ID

Solution

  1. Step 1: Identify decoupling mechanism

    Relational DB batch inserts (A) add latency and don't handle high write throughput well. Redis (C) lacks durability guarantees for buffering. Direct synchronous calls (D) cause backpressure and tight coupling.

  2. Step 2: Evaluate Kafka suitability

    Kafka supports high-throughput, partitioned by user ID to maintain ordering per user, and durable storage, enabling asynchronous processing and backpressure handling.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Kafka partitions decouple ingestion and scoring with durability and ordering [check]
Hint: Kafka partitions buffer high-throughput streams reliably [check]
Common Mistakes:
2. A messaging system must handle 925K message writes per second and fan-out delivery to millions of users with minimal latency. Which component best decouples message ingestion from fan-out delivery to ensure durability and scalability?
easy
A. Use a relational database with row-level locking for message storage and fan-out
B. Use HTTP polling from clients to fetch new messages periodically
C. Use Redis pub/sub for message delivery to all subscribers
D. Use a distributed log system like Kafka partitioned by conversation ID

Solution

  1. Step 1: Identify Kafka's role in decoupling ingestion and delivery

    Kafka partitions by conversation ID to guarantee ordering and durability, allowing high write throughput and asynchronous fan-out.

  2. Step 2: Eliminate other options

    Relational DB with locking can't handle high write concurrency; Redis pub/sub lacks durability and persistence; HTTP polling adds latency and load.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Kafka partitions -> durable, ordered, scalable message ingestion [check]
Hint: Kafka partitions ensure durable, ordered, scalable ingestion [OK]
Common Mistakes:
3. A global video streaming service wants to ensure uninterrupted content delivery even if one CDN provider experiences an outage. Which component in a multi-CDN strategy primarily manages failover between CDNs to maintain availability?
easy
A. Client-side DNS resolver that caches CDN IPs permanently
B. Global Traffic Manager that monitors CDN health and routes traffic accordingly
C. Origin server that redirects requests to alternate CDNs on failure
D. Local edge cache that stores content replicas from all CDNs

Solution

  1. Step 1: Identify role of Global Traffic Manager

    The Global Traffic Manager monitors health of multiple CDNs and dynamically routes user requests to healthy CDNs, enabling failover.

  2. Step 2: Eliminate other options

    Client-side DNS caching (A) is static and cannot adapt quickly to failures. Origin server redirects (C) add latency and single point of failure. Local edge caches (D) do not manage CDN selection or failover.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Global Traffic Manager dynamically routes traffic based on CDN health [check]
Hint: Global Traffic Manager controls CDN failover dynamically [OK]
Common Mistakes:
4. A distributed messaging system must handle 1 million writes per second and guarantee message durability despite node failures. Which component best handles this requirement?
easy
A. Use a distributed log system like Apache Kafka with replication and partitioning
B. Use an in-memory cache like Redis without persistence to speed up writes
C. Use a relational database with synchronous writes for durability
D. Use a simple message queue with no replication to reduce latency

Solution

  1. Step 1: Identify durability and high write throughput needs

    Redis without persistence (A) cannot guarantee durability on failure. Relational DB (C) cannot handle 1M writes/sec synchronously without bottlenecks. Simple queue without replication (D) risks message loss.

  2. Step 2: Recognize Kafka's replication and partitioning

    Kafka provides durable, replicated logs with partitioning for high throughput and fault tolerance.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Kafka replication + partitioning -> durability and scale [check]
Hint: Kafka replicates logs for durability and scales with partitions [check]
Common Mistakes:
5. At 10x traffic scale, which component of a global server load balancing system is most likely to fail first and why?
hard
A. The data center ingress routers, since network bandwidth scales linearly and saturates quickly
B. The health monitoring system, due to increased frequency of health checks overwhelming it
C. The GSLB control plane, as it struggles to process and update routing decisions for all data centers in real-time
D. The DNS servers, because they cannot handle the surge in query volume and caching limits failover speed

Solution

  1. Step 1: Analyze scaling challenges

    Health monitoring (The health monitoring system, due to increased frequency of health checks overwhelming it) scales with checks but can be optimized; control plane (The GSLB control plane, as it struggles to process and update routing decisions for all data centers in real-time) is distributed and designed for scale; ingress routers (The data center ingress routers, since network bandwidth scales linearly and saturates quickly) scale with network upgrades.

  2. Step 2: Identify DNS bottleneck

    DNS servers face query volume spikes and caching TTLs limit failover responsiveness, making them the first to fail under 10x load.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    DNS query volume and caching limits cause first failure at scale [check]
Hint: DNS query volume overloads servers first at scale [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard