Bird
Raised Fist0

Load Balancer vs API Gateway vs Reverse Proxy - Key Differences

HLDmediumInterview freq: common
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Load Balancer vs API Gateway vs Reverse Proxy - Key Differences
45 min · senior
senior

You are designing the edge infrastructure for a 50 million daily active user (DAU) startup that provides a suite of microservices. The system must handle encrypted traffic, maintain P99 latency under 200ms, and support flexible routing and authentication. The engineering team is debating when to use a Load Balancer, an API Gateway, or a Reverse Proxy. Your task is to design a system that clearly distinguishes these components, their roles, and trade-offs under this scale and constraints.

Constraints
  • All inbound traffic must be encrypted with TLS termination at the edge
  • P99 end-to-end latency must be under 200ms including routing overhead
  • Support for authentication, rate limiting, and request transformation is required
  • Must handle 50M DAU with peak QPS of 1M requests/sec
  • System must support routing to multiple backend microservices with versioning
📋
Guided Steps
1/5
Step 1
Define the role of the Load Balancer
What is the primary function of the Load Balancer in this architecture and which technology would you choose?
💡 Hint
Think about distributing traffic evenly and health checking backend nodes at high scale.
Key Decision
Use AWS Elastic Load Balancer (ALB) for Layer 7 routing with built-in health checks and TLS termination to distribute 1M QPS evenly across backend API Gateway instances.
Interviewer Expectation
Mid: Names ELB or NGINX as load balancer. Senior: Explains ALB’s Layer 7 routing, TLS termination benefits, and health check mechanisms to maintain availability at 1M QPS.
⚠️Common Wrong Answer
Using DNS round-robin for load balancing - leads to uneven traffic distribution and no health checks, causing backend overload and downtime.
Step 2
Clarify the API Gateway responsibilities
What core features must the API Gateway provide and which technology fits best for 50M DAU with P99 < 200ms?
💡 Hint
Focus on authentication, rate limiting, request transformation, and routing to microservices.
Key Decision
Use Kong API Gateway for its native support of OAuth2 authentication, rate limiting plugins, request/response transformation, and horizontal scalability to handle 1M QPS with <200ms latency.
Interviewer Expectation
Mid: Mentions API Gateway for routing and auth. Senior: Details Kong’s plugin architecture, horizontal scaling, and how it enforces SLA with low latency under high load.
⚠️Common Wrong Answer
Using a simple reverse proxy as API Gateway - lacks built-in auth and rate limiting, causing security and scalability issues.
Step 3
Explain the Reverse Proxy role and technology choice
How does the Reverse Proxy fit between the API Gateway and backend services, and what technology would you pick?
💡 Hint
Consider caching, SSL offloading, and request buffering close to backend services.
Key Decision
Use NGINX as Reverse Proxy to provide SSL offloading, caching static content, request buffering, and connection multiplexing to backend microservices, reducing backend load and improving latency.
Interviewer Expectation
Mid: Names NGINX or Envoy as reverse proxy. Senior: Explains NGINX’s caching, SSL offload, and connection reuse benefits that reduce backend resource consumption and improve P99 latency.
⚠️Common Wrong Answer
Skipping reverse proxy and letting API Gateway connect directly to backend - increases backend load and latency, reducing system resilience.
Step 4
Design the request flow and encryption strategy
How do you design the end-to-end encrypted request flow across Load Balancer, API Gateway, and Reverse Proxy while meeting latency constraints?
💡 Hint
Consider TLS termination points and re-encryption between components.
Key Decision
Terminate TLS at the Load Balancer (ALB) for offloading, then re-encrypt traffic between ALB and Kong API Gateway using mutual TLS, and again between Kong and NGINX Reverse Proxy to backend services to maintain end-to-end encryption and security.
Interviewer Expectation
Mid: Mentions TLS termination at edge. Senior: Explains multi-hop TLS termination with mutual TLS between components to balance security and latency under 200ms P99.
⚠️Common Wrong Answer
Terminating TLS only at backend services - causes high CPU load on backend and increases latency beyond SLA.
Step 5
API design challenge: define 3 core endpoints
Sketch 3 core API endpoints illustrating client interaction with Load Balancer, API Gateway, and Reverse Proxy layers.
💡 Hint
How does client send requests and how are they routed at each layer? How does the system handle authentication and rate limiting at the API Gateway? How does the backend service acknowledge and respond through the Reverse Proxy?
Key Decision
Define endpoints: (1) /api/v1/resource routed by ALB to Kong with OAuth2 auth and rate limiting, (2) Kong routes to NGINX reverse proxy which caches static assets at /static/*, (3) Backend service endpoint /service/data responds via NGINX proxy with connection reuse and buffering.
Interviewer Expectation
Mid: Lists endpoints with basic routing. Senior: Details how each endpoint flows through layers with auth, caching, and connection optimizations to meet latency and security constraints.
⚠️Common Wrong Answer
Designing endpoints without considering routing layers or security enforcement - leads to insecure and inefficient request handling.
🔌
API Design Challenge
Sketch 3 core API endpoints illustrating how clients interact with the Load Balancer, API Gateway, and Reverse Proxy layers.
Hints
  • How does the client send requests and how are they routed at each layer?
  • How does the system handle authentication and rate limiting at the API Gateway?
  • How does the backend service acknowledge and respond through the Reverse Proxy?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Clear distinction of Load Balancer, API Gateway, and Reverse Proxy roles with real tech choices
  • Explanation of TLS termination strategy balancing security and latency
  • API Gateway features including authentication and rate limiting at scale
  • Load Balancer choice supporting 1M QPS with health checks
  • Reverse Proxy benefits for caching and connection reuse
Good to Have - 30%
  • Detailed request flow with encryption hops
  • API design showing interaction through layers with security and performance considerations
Bonus - 10%
  • Discussion of trade-offs in technology choices under scale
  • Consideration of versioning and routing flexibility
Anti-patterns to Avoid
✅
Model Answer
▼ Reveal

This design clearly separates the roles of Load Balancer, API Gateway, and Reverse Proxy to handle 50M DAU with 1M QPS under strict latency and security constraints. AWS ALB is chosen for Layer 7 routing and TLS termination with health checks to distribute traffic evenly. Kong API Gateway manages authentication (OAuth2), rate limiting, and request transformation with horizontal scalability to maintain P99 latency under 200ms. NGINX Reverse Proxy provides SSL offloading, caching, request buffering, and connection multiplexing to reduce backend load and improve latency. Multi-hop TLS termination with mutual TLS between components ensures end-to-end encryption and security. The API design includes three core endpoints illustrating client requests flowing through these layers with clear responsibility separation.

Key Decisions
  • Use AWS ALB for Layer 7 routing, TLS termination, and health checks to handle 1M QPS
  • Use Kong API Gateway for OAuth2 authentication, rate limiting, request transformation, and horizontal scaling under 200ms P99 latency
  • Use NGINX Reverse Proxy for SSL offloading, caching static content, request buffering, and connection multiplexing to reduce backend load
  • Implement multi-hop TLS termination with mutual TLS between ALB, Kong, and NGINX to maintain end-to-end encryption
  • Design API endpoints to flow through Load Balancer → API Gateway → Reverse Proxy → Backend with clear separation of concerns
  • AWS Elastic Load Balancer (ALB) for Layer 7 routing, TLS termination, and health checks to distribute 1M QPS evenly
  • Kong API Gateway for OAuth2 authentication, rate limiting, request transformation, and horizontal scalability under 200ms P99
  • NGINX Reverse Proxy for SSL offloading, caching static content, request buffering, and connection multiplexing to reduce backend load
  • Multi-hop TLS termination with mutual TLS between ALB, Kong, and NGINX to maintain end-to-end encryption and security
  • API endpoints designed to flow through Load Balancer → API Gateway → Reverse Proxy → Backend with clear responsibility separation
  • Reject DNS round-robin load balancing due to lack of health checks and uneven traffic distribution
  • Reject using simple reverse proxy as API Gateway due to missing auth and rate limiting features
  • Reject skipping reverse proxy layer to avoid backend overload and increased latency
  • Reject terminating TLS only at backend services to prevent backend CPU overload and SLA violations
  • Use Kong’s plugin architecture to enforce SLA and security policies dynamically at scale

Practice

(1/5)
1. A social media platform needs to handle 1 million vote updates per second with low latency and durability. Which component is best suited to buffer and reliably process these vote events before updating the main datastore?
easy
A. Use a batch processing system that aggregates votes every hour
B. Use a relational database with immediate synchronous writes for each vote
C. Use an in-memory cache like Redis without persistence to handle votes
D. Use a distributed log system like Kafka to buffer and order vote events

Solution

  1. Step 1: Identify the need for durability and ordering

    Relational DB synchronous writes (Use a relational database with immediate synchronous writes for each vote) cause high latency and don't scale to 1M writes/sec. Redis without persistence (Use an in-memory cache like Redis without persistence to handle votes) risks data loss on failure. Batch processing (Use a batch processing system that aggregates votes every hour) introduces unacceptable latency.

  2. Step 2: Recognize Kafka's role

    Kafka provides durable, ordered, high-throughput buffering of vote events, decoupling ingestion from processing.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Kafka buffers high-write streams with durability and ordering [check]
Hint: Kafka buffers high-write streams with durability and ordering [check]
Common Mistakes:
2. For a distributed database with high write throughput and large keyspace, which approach to Bloom filter design best balances memory usage and false positive rate?
medium
A. Use a single global Bloom filter for all keys to minimize memory overhead
B. Use partitioned Bloom filters per data shard to localize false positives and reduce memory per filter
C. Use counting Bloom filters everywhere to allow deletions despite higher memory cost
D. Use a cache of recently accessed keys instead of Bloom filters to reduce false positives

Solution

  1. Step 1: Analyze memory and false positive trade-offs

    A single global Bloom filter grows large and has higher false positive rates due to keyspace size; partitioning per shard reduces filter size and localizes false positives.

  2. Step 2: Evaluate alternatives

    Counting Bloom filters allow deletions but consume more memory, which is costly at large scale; caching recent keys does not guarantee absence and can miss cold keys.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Partitioned Bloom filters balance memory and false positives at scale [check]
Hint: Partition filters localize false positives, save memory [check]
Common Mistakes:
3. If the Gossip Protocol component fails to propagate membership updates, what is the most immediate failure observed in the key-value store?
medium
A. Consistent hashing ring fails to rebalance keys on node failure
B. Nodes continue to send requests to failed nodes causing timeouts
C. Clients receive stale data due to vector clock version conflicts
D. Write operations block indefinitely waiting for quorum acknowledgments

Solution

  1. Step 1: Understand gossip failure impact

    Without membership updates, nodes are unaware of failed nodes and continue routing requests to them.

  2. Step 2: Eliminate other failure modes

    Vector clocks handle versioning, unrelated to gossip failure. Consistent hashing ring updates rely on membership info but do not fail immediately. Write blocking on quorum is unrelated to gossip propagation.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Missing membership updates cause requests to failed nodes causing timeouts [check]
Hint: Gossip failure -> nodes unaware of failures, keep sending requests [OK]
Common Mistakes:
4. If Redis becomes unavailable in a centralized rate limiting system, what is the immediate failure mode for incoming requests?
medium
A. API servers allow all requests without rate limiting, risking overload
B. Requests are queued locally until Redis recovers, causing increased latency
C. API servers reject all requests immediately to avoid exceeding limits
D. API servers switch to a backup Redis instance automatically without delay

Solution

  1. Step 1: Understand Redis unavailability impact

    Without Redis, the centralized counter is unreachable, so rate limiting cannot be enforced.

  2. Step 2: Check fallback behavior

    Most implementations allow requests to proceed without rate limiting to avoid blocking, risking overload.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Redis down -> no rate limiting -> requests allowed unchecked [check]
Hint: No Redis means no rate limiting, requests pass unchecked [check]
Common Mistakes:
5. At 10x scale, which component in the search ranking system is most likely to fail first and why?
hard
A. Model Serving, because CPU-intensive scoring cannot be parallelized beyond a point
B. Update Stream (Kafka), because partition count limits throughput and causes backpressure
C. A/B Testing Controller, because routing logic becomes too complex to handle high traffic
D. Feature Store, because maintaining consistent, low-latency feature retrieval under massive read load saturates memory bandwidth

Solution

  1. Step 1: Analyze scaling bottlenecks

    Model Serving can be horizontally scaled; A/B controller logic is lightweight; Kafka partitions can be increased.

  2. Step 2: Identify Feature Store bottleneck

    Feature Store must serve millions of low-latency reads with consistency, saturating memory bandwidth and cache, making it the first to fail.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Feature Store memory bandwidth saturates first at 10x scale [check]
Hint: Feature Store memory bandwidth saturates before compute or routing [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard