Bird
Raised Fist0

Rate Limiting at Different Layers - API Gateway, Service, Database

HLDmediumInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Rate Limiting at Different Layers - API Gateway, Service, Database
45 min · senior
senior

You are building a rate limiting system for a 50 million daily active user (DAU) startup. The system must enforce rate limits at multiple layers: API Gateway, individual microservices, and the database layer to prevent abuse and ensure fair usage. The system must handle up to 1 million requests per second at peak, maintain P99 latency under 200ms, and encrypt all rate limit metadata in transit and at rest. Offline delivery of rate limit exceeded notifications is required. The system must be resilient to burst traffic and partial failures without causing cascading failures.

Constraints
  • Encrypt all rate limit metadata in transit (TLS) and at rest (AES-256).
  • P99 latency for rate limit checks must be under 200ms end-to-end.
  • Support offline delivery of rate limit exceeded notifications via push and email.
  • Handle burst traffic spikes up to 3x average QPS without dropping requests.
  • Rate limits must be configurable per user, API key, and service.
📋
Guided Steps
1/5
Step 1
Choosing the API Gateway Rate Limiting Strategy
How would you implement rate limiting at the API Gateway layer to handle 1M RPS with P99 < 200ms?
💡 Hint
Consider a distributed in-memory store with atomic counters and TTL for fast checks.
Key Decision
Use Redis Cluster with Lua scripts for atomic increment and TTL-based sliding window counters at API Gateway - Redis Cluster scales horizontally and Lua scripts ensure atomicity under high concurrency.
Interviewer Expectation
Mid: Names Redis Cluster for caching counters. Senior: Explains Lua scripts for atomic increments and TTL-based sliding window to avoid race conditions and stale counters.
⚠️Common Wrong Answer
Using a relational database like MySQL for rate limit counters - causes high latency and contention at 1M RPS, breaking P99 latency SLA.
Step 2
Service Layer Rate Limiting Enforcement
How should individual microservices enforce rate limits considering eventual consistency and burst tolerance?
💡 Hint
Think about asynchronous usage reporting and local caching to reduce latency and load on central stores.
Key Decision
Implement local in-memory token buckets per service instance combined with asynchronous usage reporting to Kafka - token buckets provide low-latency enforcement, Kafka decouples usage reporting for eventual consistency.
Interviewer Expectation
Mid: Mentions token bucket algorithm. Senior: Details local caching to handle bursts and Kafka for durable, scalable usage event streaming to central rate limit store.
⚠️Common Wrong Answer
Synchronous calls to central rate limit DB from every service request - causes high latency and system-wide bottlenecks under burst traffic.
Step 3
Database Layer Rate Limit Metadata Storage
Which database technology would you choose to store rate limit configurations and usage data, and why?
💡 Hint
Consider write-heavy workloads with range queries and encryption at rest.
Key Decision
Use Cassandra with partition key = user_id and clustering key = timestamp for time-series usage data - Cassandra’s LSM-tree design supports high write throughput and efficient range scans for usage aggregation; supports encryption at rest natively.
Interviewer Expectation
Mid: Names Cassandra for scalability. Senior: Explains partition key choice for data locality and range queries, and why relational DBs fail at write scale and latency.
⚠️Common Wrong Answer
Using MongoDB for time-series usage data - MongoDB’s document model and locking cause write amplification and higher latency at scale.
Step 4
Offline Delivery of Rate Limit Exceeded Notifications
How would you design offline delivery of rate limit exceeded notifications to users with guaranteed delivery?
💡 Hint
Consider durable queues and push notification services with retry mechanisms.
Key Decision
Use Kafka for durable notification event streaming combined with AWS SNS for push notifications and SES for email - Kafka ensures no lost notifications, SNS/SES handle multi-channel delivery with retries.
Interviewer Expectation
Mid: Mentions Kafka for event streaming. Senior: Explains Kafka’s durability and partitioning for ordering, plus SNS/SES integration for multi-channel offline delivery and retry policies.
⚠️Common Wrong Answer
Directly sending notifications synchronously during request processing - increases latency and risks dropped notifications on failure.
Step 5
Ensuring Security and Encryption Compliance
How do you ensure encryption of rate limit metadata in transit and at rest without impacting latency?
💡 Hint
Look for TLS termination points and database encryption features with hardware acceleration.
Key Decision
Terminate TLS at API Gateway and service ingress using NGINX with hardware TLS acceleration; enable Cassandra’s native AES-256 encryption at rest with transparent data encryption (TDE) - balances security and low latency.
Interviewer Expectation
Mid: Mentions TLS and encryption at rest. Senior: Details hardware TLS acceleration to meet latency SLA and Cassandra TDE configuration for compliance.
⚠️Common Wrong Answer
Encrypting data manually in application layer causing CPU overhead and increased latency beyond P99 SLA.
🔌
API Design Challenge
Sketch 3 core endpoints for the rate limiting system: (1) client request submission with rate limit check, (2) service reporting usage metrics, (3) client fetching rate limit exceeded notifications.
Hints
  • How does the client send requests and receive immediate rate limit decisions?
  • How do services report usage metrics asynchronously for enforcement?
  • How does the client sync and acknowledge offline rate limit exceeded notifications?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Use Redis Cluster with Lua scripts for atomic counters at API Gateway to meet 1M RPS and P99 < 200ms.
  • Implement local token buckets in services with Kafka for asynchronous usage reporting to handle bursts.
  • Store rate limit configs and usage in Cassandra with appropriate partitioning for write scalability.
  • Design offline notification delivery using Kafka plus SNS/SES for guaranteed multi-channel delivery.
Good to Have - 30%
  • Use hardware TLS acceleration at ingress points to maintain encryption without latency impact.
  • Enable Cassandra transparent data encryption for compliance with AES-256 at rest.
  • Configurable rate limits per user, API key, and service for flexible enforcement.
Bonus - 10%
  • Explain failure modes such as Redis cluster partition loss causing temporary rate limit bypass and mitigation.
Anti-patterns to Avoid
✅
Model Answer
▼ Reveal

This design implements a multi-layer rate limiting system for a 50M DAU startup handling 1M RPS with strict latency and security requirements. It enforces rate limits at the API Gateway using Redis Cluster with Lua scripts for atomic counters and sliding window TTLs, at the service layer with local in-memory token buckets and asynchronous Kafka reporting, and stores metadata in Cassandra with user_id partitioning and timestamp clustering for efficient time-series writes. Offline notifications are delivered reliably via Kafka event streaming combined with AWS SNS and SES for push and email with retries. Security is ensured by TLS termination with hardware acceleration at ingress points and AES-256 encryption at rest using Cassandra's transparent data encryption.

Key Decisions
  • Use Redis Cluster with Lua scripts to implement atomic increment and TTL-based sliding window counters at API Gateway to meet 1M RPS and P99 < 200ms latency.
  • Implement local in-memory token buckets per service instance to handle bursts with asynchronous usage reporting to Kafka for eventual consistency.
  • Store rate limit metadata and usage data in Cassandra using user_id as partition key and timestamp as clustering key to support high write throughput and efficient range queries.
  • Use Kafka for durable notification event streaming ensuring no lost notifications and integrate with AWS SNS and SES for reliable multi-channel offline delivery with retry policies.
  • Terminate TLS at API Gateway and service ingress using NGINX with hardware TLS acceleration to maintain encryption without latency penalty.
  • Enable Cassandra's native AES-256 encryption at rest with transparent data encryption for compliance.
  • Redis Cluster with Lua scripts for atomic, low-latency rate limit counters at API Gateway.
  • Local in-memory token buckets in microservices combined with Kafka for asynchronous usage reporting.
  • Cassandra with user_id partition key and timestamp clustering key for scalable time-series usage data storage.
  • Kafka for durable notification event streaming ensuring no lost offline notifications.
  • AWS SNS and SES for multi-channel offline delivery with retry and backoff mechanisms.
  • NGINX with hardware TLS acceleration for encrypted transport without latency penalty.
  • Cassandra native AES-256 encryption at rest with transparent data encryption for compliance.
  • Configurable rate limits per user, API key, and service stored in Cassandra for dynamic enforcement.

Practice

(1/5)
1. For storing feature metadata and versioning in the registry, which storage approach is best when the system requires strong consistency and auditability for compliance?
medium
A. Use a relational database with ACID transactions and audit logging
B. Use a cloud object store with versioned files and eventual consistency
C. Use a distributed log system like Kafka to append feature metadata changes
D. Use a distributed NoSQL store with eventual consistency and high availability

Solution

  1. Step 1: Identify compliance needs for strong consistency and auditability

    Compliance requires strict transactional guarantees and immutable audit trails.

  2. Step 2: Evaluate storage options

    NoSQL and object stores offer eventual consistency, risking stale reads. Kafka provides append-only logs but lacks transactional ACID guarantees for metadata queries. Relational DBs provide ACID transactions and built-in audit logging, ensuring compliance.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Relational DBs ensure ACID and audit logging for compliance [check]
Hint: Compliance needs ACID and audit logs -> relational DB [check]
Common Mistakes:
2. Without a centralized coordination service (like ZooKeeper) in a horizontally scaled cluster, what exact failure occurs during leader election?
medium
A. All nodes elect themselves leader simultaneously, causing split-brain and inconsistent writes
B. The cluster halts because no node can become leader, blocking all writes
C. Leader election proceeds normally but with increased latency due to retries
D. Nodes elect multiple leaders but automatically resolve conflicts without data loss

Solution

  1. Step 1: Understand role of centralized coordination in leader election

    Coordination service ensures single leader election and consensus.

  2. Step 2: Analyze failure without coordination

    Without it, leader election retries occur causing latency but eventual leader election succeeds via timeouts.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Leader election delays but does not halt cluster without coordination [check]
Hint: No coordination causes leader election delays, not total failure [check]
Common Mistakes:
3. For a distributed key-value store that must handle frequent node additions and removals with minimal key remapping, which approach is best?
medium
A. Use modular hashing with a fixed bucket count and rehash keys on node changes
B. Use a centralized directory service mapping keys to nodes updated on every node change
C. Use consistent hashing with virtual nodes to balance load and minimize remapping
D. Use rendezvous hashing to assign keys to nodes without a ring structure

Solution

  1. Step 1: Compare modular hashing and consistent hashing

    Modular hashing requires full remapping on node changes, causing high overhead.

  2. Step 2: Evaluate alternatives

    Centralized directory adds latency and single point of failure; rendezvous hashing is good but less common and more complex than consistent hashing with virtual nodes.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Consistent hashing with virtual nodes balances load and minimizes remapping [check]
Hint: Virtual nodes smooth load, minimize remap on node changes [check]
Common Mistakes:
4. At 10x scale, which component in a GraphQL API gateway is most likely to become the bottleneck and why?
hard
A. The underlying database, because it cannot handle 10x query volume
B. The GraphQL query parsing and validation layer, due to CPU-intensive query complexity checks on every request
C. The network load balancer, because it cannot distribute traffic evenly at high scale
D. The client SDK, because it caches too much data locally

Solution

  1. Step 1: Identify components stressed by 10x scale

    Database scaling is often handled by sharding or replicas; load balancers scale horizontally; client SDKs are local.

  2. Step 2: Recognize GraphQL parsing and validation cost

    Parsing and complexity validation are CPU-bound and happen per request, becoming CPU bottlenecks at high request rates.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    GraphQL validation CPU cost scales linearly with requests [check]
Hint: GraphQL validation CPU cost limits scale first [check]
Common Mistakes:
5. At 10x scale, which component in your design explanation is most likely to fail first due to stateful connection bottlenecks, and why?
hard
A. Stateful WebSocket servers because connections are bound to server memory and cannot be easily resharded
B. Stateless API servers because CPU usage spikes with more requests
C. Load balancers because they cannot handle increased DNS queries
D. Databases because they run out of disk space first

Solution

  1. Step 1: Identify stateful components at scale

    WebSocket servers maintain persistent connections bound to server memory, limiting horizontal scaling.

  2. Step 2: Compare with stateless components

    Stateless API servers and load balancers scale horizontally more easily; databases fail later due to storage.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Stateful WebSocket servers bottleneck first due to connection affinity [check]
Hint: Stateful WebSocket servers bottleneck first at scale [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard