API Gateway pattern in Microservices - Scalability & System Analysis

Scalability Analysis - API Gateway pattern

Growth Table: API Gateway Pattern

Users	Requests per Second (RPS)	API Gateway Load	Microservices Load	Network Traffic	Notes
100 users	~50 RPS	Single instance handles easily	Microservices handle requests directly	Low	Simple setup, no scaling needed
10,000 users	~5,000 RPS	API Gateway needs horizontal scaling	Microservices start to see increased load	Moderate	Introduce load balancer, caching at gateway
1,000,000 users	~500,000 RPS	Multiple API Gateway instances behind LB	Microservices require scaling and partitioning	High	Use caching, rate limiting, and circuit breakers
100,000,000 users	~50,000,000 RPS	Global distributed API Gateways with CDN	Microservices sharded and geo-distributed	Very High	Advanced routing, edge caching, and autoscaling

First Bottleneck

The API Gateway becomes the first bottleneck as it handles all incoming requests. At moderate to high traffic (around 10,000 users or 5,000 RPS), a single gateway instance struggles with CPU and network limits. This causes increased latency and potential request drops.

Scaling Solutions

• Horizontal Scaling: Add multiple API Gateway instances behind a load balancer to distribute traffic evenly.
• Caching: Implement response caching at the gateway to reduce calls to microservices.
• Rate Limiting: Protect backend services by limiting request rates per user or IP.
• Sharding Microservices: Partition microservices by user region or function to reduce load.
• CDN Integration: Use Content Delivery Networks for static content and edge caching to reduce gateway load.
• Circuit Breakers: Prevent cascading failures by stopping calls to failing microservices.

Back-of-Envelope Cost Analysis

• At 10,000 users (~5,000 RPS):
  • API Gateway CPU: ~50% utilization per instance (assuming 2,000 RPS per instance)
  • Network bandwidth: ~500 Mbps (assuming 100 KB per request/response)
  • Storage: Minimal at gateway, microservices storage depends on data
• At 1,000,000 users (~500,000 RPS):
  • Need ~250 API Gateway instances (500,000 / 2,000 RPS per instance)
  • Network bandwidth: ~50 Gbps
  • Microservices require database scaling and caching layers

Interview Tip

Start by explaining the role of the API Gateway as a single entry point. Discuss how it simplifies client interactions but can become a bottleneck. Then, outline scaling steps: horizontal scaling, caching, rate limiting, and microservices partitioning. Always justify why each step is needed based on traffic growth.

Self Check

Your database handles 1000 QPS. Traffic grows 10x to 10,000 QPS. What do you do first?

Answer: Add read replicas and implement caching to reduce direct database load before scaling vertically or sharding.

Key Result

API Gateway handles all client requests and becomes the first bottleneck as traffic grows; horizontal scaling and caching at the gateway are key to maintaining performance.

Practice

1. What is the primary role of an API Gateway in a microservices architecture?

easy

A. It acts as a single entry point to route requests to multiple microservices.

B. It stores all the data for the microservices.

C. It replaces the database in the system.

D. It directly manages the internal logic of each microservice.

Solution

1. Step 1: Understand the role of API Gateway

   An API Gateway serves as a single entry point that routes client requests to the appropriate microservices.

2. Step 2: Eliminate incorrect roles

   It does not store data, replace databases, or manage internal microservice logic; those are handled by other components.

3. Final Answer:

   It acts as a single entry point to route requests to multiple microservices. -> Option A

4. Quick Check:

   API Gateway = Single entry point [OK]

Hint: API Gateway routes requests, it does not store data [OK]

Common Mistakes:

• Confusing API Gateway with database or service logic
• Thinking API Gateway manages microservice internals
• Assuming API Gateway stores data

2. Which of the following is the correct way to describe the API Gateway's function in handling client requests?

easy

A. API Gateway directly executes business logic for each microservice.

B. API Gateway replaces the need for microservices.

C. API Gateway stores client data permanently.

D. API Gateway routes requests, handles authentication, and aggregates responses.

Solution

1. Step 1: Identify API Gateway responsibilities

   API Gateway routes requests, manages security like authentication, and can combine responses from multiple services.

2. Step 2: Remove incorrect options

   It does not execute business logic itself, store data permanently, or replace microservices.

3. Final Answer:

   API Gateway routes requests, handles authentication, and aggregates responses. -> Option D

4. Quick Check:

   Routing + Security + Aggregation = API Gateway [OK]

Hint: API Gateway routes and secures, does not store data [OK]

Common Mistakes:

• Thinking API Gateway runs business logic
• Confusing API Gateway with data storage
• Assuming API Gateway replaces microservices

3. Consider this simplified request flow: A client sends a request to the API Gateway, which then calls Service A and Service B. The API Gateway combines their responses and sends back a single response to the client. What is the main benefit of this approach?

medium

A. It increases the number of client requests to microservices.

B. It reduces the number of client requests by aggregating responses.

C. It forces clients to call each microservice separately.

D. It eliminates the need for microservices.

Solution

1. Step 1: Analyze the request flow

   The API Gateway receives one client request and internally calls multiple services, then combines their responses.

2. Step 2: Understand the benefit

   This reduces the number of requests the client must make, simplifying client logic and improving efficiency.

3. Final Answer:

   It reduces the number of client requests by aggregating responses. -> Option B

4. Quick Check:

   Response aggregation reduces client calls [OK]

Hint: API Gateway aggregates responses to reduce client calls [OK]

Common Mistakes:

• Thinking client must call each service separately
• Believing API Gateway increases client requests
• Confusing aggregation with service removal

4. A developer implemented an API Gateway but notices that clients receive errors when calling multiple microservices through it. Which of the following is the most likely cause?

medium

A. The client is bypassing the API Gateway and calling microservices directly.

B. The microservices do not have any APIs.

C. The API Gateway is not properly routing requests to the correct microservices.

D. The API Gateway is storing all client data incorrectly.

Solution

1. Step 1: Identify the error source

   If clients get errors when calling multiple services via the API Gateway, routing issues are a common cause.

2. Step 2: Exclude other causes

   Microservices usually have APIs; clients bypassing the gateway would not cause errors through it; storing data is not the gateway's role.

3. Final Answer:

   The API Gateway is not properly routing requests to the correct microservices. -> Option C

4. Quick Check:

   Routing errors cause client failures [OK]

Hint: Check routing rules if clients get errors via API Gateway [OK]

Common Mistakes:

• Blaming microservices for missing APIs
• Assuming clients bypass the gateway
• Thinking API Gateway stores client data

5. You are designing a system with multiple microservices and want to use an API Gateway. Which of the following is the best reason to include response aggregation in the API Gateway?

hard

A. To reduce client complexity by combining data from multiple services into one response.

B. To increase the number of network calls clients must make.

C. To allow clients to manage authentication for each microservice separately.

D. To store all microservice data centrally in the API Gateway.

Solution

1. Step 1: Understand response aggregation purpose

   Response aggregation combines data from multiple microservices into a single response, simplifying client handling.

2. Step 2: Evaluate other options

   Increasing network calls or forcing clients to manage authentication per service adds complexity; storing data centrally is not the gateway's role.

3. Final Answer:

   To reduce client complexity by combining data from multiple services into one response. -> Option A

4. Quick Check:

   Aggregation simplifies client responses [OK]

Hint: Aggregate responses to simplify client communication [OK]

Common Mistakes:

• Thinking aggregation increases client calls
• Assuming clients handle all authentications
• Confusing API Gateway with data storage