API gateway concept in HLD - Scalability & System Analysis

Scalability Analysis - API gateway concept

Growth Table: API Gateway Scaling

Users/Traffic	API Gateway Load	Latency Impact	Security & Routing	Infrastructure Changes
100 users	Low requests per second (RPS), single instance handles well	Minimal latency, simple routing	Basic authentication and rate limiting	Single server or cloud function
10,000 users	Moderate RPS, single instance may start to saturate	Latency slightly increases, need optimized routing	Enhanced security policies, throttling	Load balancer added, possible multiple instances
1 million users	High RPS, single instance insufficient	Latency sensitive, need caching and optimized paths	Advanced security (OAuth, JWT), API versioning	Horizontal scaling, distributed gateway cluster
100 million users	Very high RPS, requires global distribution	Latency critical, edge caching and CDN integration	Multi-tenant security, dynamic routing, throttling	Global load balancers, multi-region clusters, CDN

First Bottleneck

The API gateway server CPU and memory become the first bottleneck as traffic grows. It must handle all incoming requests, perform routing, authentication, rate limiting, and sometimes transformation. At moderate to high traffic, a single gateway instance cannot keep up, causing increased latency and dropped requests.

Scaling Solutions

• Horizontal Scaling: Add multiple API gateway instances behind a load balancer to distribute traffic.
• Caching: Use response caching at the gateway or integrate with CDN to reduce backend load.
• Rate Limiting & Throttling: Protect backend services by limiting requests per user or IP.
• Edge Deployment: Deploy gateways closer to users globally to reduce latency.
• Service Mesh Integration: For internal microservices, use service mesh to offload routing and security.
• API Versioning & Routing Optimization: Efficient routing rules reduce processing time.

Back-of-Envelope Cost Analysis

• At 1 million users, assuming 1 request per second per user, API gateway handles ~1 million RPS.
• One server handles ~3000-5000 concurrent connections; need ~200-300 gateway instances.
• Network bandwidth: 1 Gbps ~125 MB/s; estimate average request size to calculate total bandwidth.
• Storage is minimal at gateway level, mostly logs and cache; scale storage for logs accordingly.
• Cost grows with number of instances, bandwidth, and caching infrastructure.

Interview Tip

Start by explaining the API gateway role and its responsibilities. Discuss traffic growth impact on CPU, memory, and network. Identify the first bottleneck clearly. Then propose scaling solutions step-by-step: horizontal scaling, caching, edge deployment. Mention trade-offs and cost implications. Use real numbers to show understanding.

Self Check

Your API gateway handles 1000 QPS. Traffic grows 10x to 10,000 QPS. What do you do first?

Answer: Add more API gateway instances behind a load balancer to horizontally scale and distribute the increased load, preventing CPU/memory saturation and reducing latency.

Key Result

API gateway first breaks at CPU/memory under high request load; horizontal scaling and caching are key to handle millions of users efficiently.

Practice

1. What is the primary role of an API gateway in a system architecture?

easy

A. It replaces the need for backend services.

B. It stores all the data for the application.

C. It directly manages the database connections.

D. It acts as a single entry point for client requests to backend services.

Solution

1. Step 1: Understand the role of API gateway

   An API gateway serves as the main entry point for clients to access multiple backend services, simplifying communication.

2. Step 2: Eliminate incorrect roles

   Storing data, managing database connections, or replacing backend services are not responsibilities of an API gateway.

3. Final Answer:

   It acts as a single entry point for client requests to backend services. -> Option D

4. Quick Check:

   API gateway = single entry point [OK]

Hint: API gateway = one door for all client requests [OK]

Common Mistakes:

• Confusing API gateway with database or storage
• Thinking API gateway replaces backend services
• Assuming API gateway manages data directly

2. Which of the following is the correct function of an API gateway in handling client requests?

easy

A. Directly querying the database for each request.

B. Serving static files like images and videos.

C. Routing requests to appropriate backend services.

D. Replacing the client application.

Solution

1. Step 1: Identify API gateway functions

   An API gateway routes incoming client requests to the correct backend services based on the request type or URL.

2. Step 2: Remove unrelated functions

   Serving static files, querying databases directly, or replacing clients are not typical API gateway roles.

3. Final Answer:

   Routing requests to appropriate backend services. -> Option C

4. Quick Check:

   API gateway routes requests = C [OK]

Hint: API gateway directs traffic to backend services [OK]

Common Mistakes:

• Thinking API gateway queries databases directly
• Confusing API gateway with CDN or static server
• Assuming API gateway replaces client apps

3. Consider this simplified API gateway flow:

Client -> API Gateway -> Service A (auth) -> Service B (data)

If the API gateway receives a request and Service A rejects it due to failed authentication, what will the client receive?

medium

A. Authentication error response

B. Data from Service B

C. No response (timeout)

D. Success response with empty data

Solution

1. Step 1: Trace request through services

   The request first goes to Service A for authentication. If Service A rejects it, the request does not proceed further.

2. Step 2: Determine client response

   Since authentication failed, the API gateway returns an authentication error to the client instead of data or success.

3. Final Answer:

   Authentication error response -> Option A

4. Quick Check:

   Failed auth = error response [OK]

Hint: Failed auth stops flow, returns error [OK]

Common Mistakes:

• Assuming data is returned despite failed auth
• Thinking client gets no response (timeout)
• Believing success with empty data is sent

4. An API gateway is configured to route requests to multiple services. However, clients report 404 errors for valid endpoints. What is the most likely cause?

medium

A. Backend services are down.

B. API gateway routing rules are misconfigured.

C. Client requests are malformed.

D. API gateway is overloaded.

Solution

1. Step 1: Analyze 404 error meaning

   A 404 error means the requested resource was not found, often due to incorrect routing or URL mapping.

2. Step 2: Identify likely cause in API gateway

   If backend services are up and clients send correct requests, misconfigured routing rules in the API gateway cause 404 errors.

3. Final Answer:

   API gateway routing rules are misconfigured. -> Option B

4. Quick Check:

   404 = routing misconfiguration [OK]

Hint: 404 usually means wrong routing path [OK]

Common Mistakes:

• Assuming backend services are down without checking
• Blaming client requests without validation
• Thinking overload causes 404 errors

5. You want to design an API gateway that aggregates data from three backend services before responding to the client. Which approach best fits this requirement?

hard

A. API gateway calls all three services, combines their responses, then sends one response to client.

B. API gateway caches data and never calls backend services.

C. API gateway sends separate responses from each service to the client.

D. API gateway forwards client requests directly to one backend service only.

Solution

1. Step 1: Understand aggregation requirement

   The API gateway must collect data from multiple services and combine it into a single response for the client.

2. Step 2: Evaluate options for aggregation

   Forwarding to one service or sending separate responses does not aggregate data. Caching alone does not fulfill aggregation if fresh data is needed.

3. Final Answer:

   API gateway calls all three services, combines their responses, then sends one response to client. -> Option A

4. Quick Check:

   Aggregation = combine multiple service responses [OK]

Hint: Aggregate by combining multiple service responses [OK]

Common Mistakes:

• Thinking forwarding to one service aggregates data
• Sending multiple responses instead of one combined
• Assuming caching replaces aggregation