API gateway concept in HLD - System Design Exercise

Design: API Gateway System

Design focuses on the API Gateway component and its interaction with clients and backend services. Backend service internal design and database details are out of scope.

Functional Requirements

FR1: Serve as a single entry point for multiple backend services

FR2: Route client requests to appropriate backend services

FR3: Handle authentication and authorization for incoming requests

FR4: Perform request and response transformations (e.g., protocol translation, data format changes)

FR5: Implement rate limiting to protect backend services from overload

FR6: Provide caching to improve response times for frequent requests

FR7: Log requests and responses for monitoring and debugging

FR8: Support load balancing across multiple instances of backend services

Non-Functional Requirements

NFR1: Must handle 10,000 concurrent client connections

NFR2: API response latency p99 should be under 150ms

NFR3: Availability target of 99.9% uptime (less than 8.77 hours downtime per year)

NFR4: Scalable to add more backend services without downtime

NFR5: Secure handling of sensitive data and credentials

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

❓ Question 6

Key Components

Load balancer in front of API gateway instances

Authentication and authorization module

Routing and service discovery component

Rate limiter and throttling mechanism

Caching layer

Logging and monitoring system

Backend service registry

Design Patterns

Reverse proxy pattern

Circuit breaker pattern for backend service failures

Token-based authentication

Request throttling and rate limiting

Caching proxy pattern

Reference Architecture

Load Balancer

↓

API Gateway Cluster

↓

Auth

↓

Cache

↓

Backend Service 1

Components

Load Balancer

Nginx or AWS ELB

Distributes incoming client requests evenly across API Gateway instances

API Gateway

Kong, AWS API Gateway, or custom Node.js service

Main entry point that handles routing, authentication, rate limiting, caching, and logging

Authentication Module

JWT validation library or OAuth server integration

Verifies client identity and permissions before forwarding requests

Routing and Service Discovery

Consul, Eureka, or static config

Determines which backend service to forward the request to

Rate Limiter

Redis-based token bucket algorithm

Prevents clients from overwhelming backend services by limiting request rates

Cache

Redis or Memcached

Stores frequent responses to reduce backend load and improve latency

Logging and Monitoring

ELK stack (Elasticsearch, Logstash, Kibana) or Prometheus + Grafana

Collects request logs and metrics for analysis and alerting

Request Flow

1. Client sends request to Load Balancer

2. Load Balancer forwards request to one API Gateway instance

3. API Gateway authenticates the request using Authentication Module

4. If authentication fails, return error to client

5. API Gateway checks rate limits for the client

6. If rate limit exceeded, return error to client

7. API Gateway checks cache for a stored response

8. If cache hit, return cached response to client

9. If cache miss, API Gateway uses Routing component to find backend service

10. API Gateway forwards request to backend service

11. Backend service processes request and returns response

12. API Gateway caches the response if applicable

13. API Gateway logs request and response details

14. API Gateway returns response to client

Database Schema

Not applicable as API Gateway typically does not store persistent data but may store configuration and rate limit counters in fast key-value stores like Redis.

Scaling Discussion

Bottlenecks

API Gateway instance CPU and memory limits under high concurrent connections

Rate limiter storage becoming a hotspot under heavy traffic

Cache size and eviction policies limiting hit rate

Service discovery delays causing routing failures

Logging system overwhelmed by high request volume

Solutions

Scale API Gateway horizontally by adding more instances behind load balancer

Use distributed rate limiting with sharded Redis clusters or token bucket algorithms

Implement cache sharding and tune eviction policies based on usage patterns

Use highly available and fast service discovery systems with health checks

Aggregate logs and use sampling to reduce logging volume; use scalable log storage solutions

Interview Tips

Time: Spend 10 minutes understanding requirements and clarifying questions, 20 minutes designing the architecture and data flow, 10 minutes discussing scaling and trade-offs, 5 minutes summarizing.

Explain the role of API Gateway as a single entry point

Discuss how authentication and rate limiting protect backend services

Describe caching benefits and how it improves latency

Highlight how routing and service discovery enable flexible backend integration

Mention scalability strategies and handling failures gracefully

Show awareness of monitoring and logging importance

Practice

1. What is the primary role of an API gateway in a system architecture?

easy

A. It replaces the need for backend services.

B. It stores all the data for the application.

C. It directly manages the database connections.

D. It acts as a single entry point for client requests to backend services.

Solution

1. Step 1: Understand the role of API gateway

   An API gateway serves as the main entry point for clients to access multiple backend services, simplifying communication.

2. Step 2: Eliminate incorrect roles

   Storing data, managing database connections, or replacing backend services are not responsibilities of an API gateway.

3. Final Answer:

   It acts as a single entry point for client requests to backend services. -> Option D

4. Quick Check:

   API gateway = single entry point [OK]

Hint: API gateway = one door for all client requests [OK]

Common Mistakes:

• Confusing API gateway with database or storage
• Thinking API gateway replaces backend services
• Assuming API gateway manages data directly

2. Which of the following is the correct function of an API gateway in handling client requests?

easy

A. Directly querying the database for each request.

B. Serving static files like images and videos.

C. Routing requests to appropriate backend services.

D. Replacing the client application.

Solution

1. Step 1: Identify API gateway functions

   An API gateway routes incoming client requests to the correct backend services based on the request type or URL.

2. Step 2: Remove unrelated functions

   Serving static files, querying databases directly, or replacing clients are not typical API gateway roles.

3. Final Answer:

   Routing requests to appropriate backend services. -> Option C

4. Quick Check:

   API gateway routes requests = C [OK]

Hint: API gateway directs traffic to backend services [OK]

Common Mistakes:

• Thinking API gateway queries databases directly
• Confusing API gateway with CDN or static server
• Assuming API gateway replaces client apps

3. Consider this simplified API gateway flow:

Client -> API Gateway -> Service A (auth) -> Service B (data)

If the API gateway receives a request and Service A rejects it due to failed authentication, what will the client receive?

medium

A. Authentication error response

B. Data from Service B

C. No response (timeout)

D. Success response with empty data

Solution

1. Step 1: Trace request through services

   The request first goes to Service A for authentication. If Service A rejects it, the request does not proceed further.

2. Step 2: Determine client response

   Since authentication failed, the API gateway returns an authentication error to the client instead of data or success.

3. Final Answer:

   Authentication error response -> Option A

4. Quick Check:

   Failed auth = error response [OK]

Hint: Failed auth stops flow, returns error [OK]

Common Mistakes:

• Assuming data is returned despite failed auth
• Thinking client gets no response (timeout)
• Believing success with empty data is sent

4. An API gateway is configured to route requests to multiple services. However, clients report 404 errors for valid endpoints. What is the most likely cause?

medium

A. Backend services are down.

B. API gateway routing rules are misconfigured.

C. Client requests are malformed.

D. API gateway is overloaded.

Solution

1. Step 1: Analyze 404 error meaning

   A 404 error means the requested resource was not found, often due to incorrect routing or URL mapping.

2. Step 2: Identify likely cause in API gateway

   If backend services are up and clients send correct requests, misconfigured routing rules in the API gateway cause 404 errors.

3. Final Answer:

   API gateway routing rules are misconfigured. -> Option B

4. Quick Check:

   404 = routing misconfiguration [OK]

Hint: 404 usually means wrong routing path [OK]

Common Mistakes:

• Assuming backend services are down without checking
• Blaming client requests without validation
• Thinking overload causes 404 errors

5. You want to design an API gateway that aggregates data from three backend services before responding to the client. Which approach best fits this requirement?

hard

A. API gateway calls all three services, combines their responses, then sends one response to client.

B. API gateway caches data and never calls backend services.

C. API gateway sends separate responses from each service to the client.

D. API gateway forwards client requests directly to one backend service only.

Solution

1. Step 1: Understand aggregation requirement

   The API gateway must collect data from multiple services and combine it into a single response for the client.

2. Step 2: Evaluate options for aggregation

   Forwarding to one service or sending separate responses does not aggregate data. Caching alone does not fulfill aggregation if fresh data is needed.

3. Final Answer:

   API gateway calls all three services, combines their responses, then sends one response to client. -> Option A

4. Quick Check:

   Aggregation = combine multiple service responses [OK]

Hint: Aggregate by combining multiple service responses [OK]

Common Mistakes:

• Thinking forwarding to one service aggregates data
• Sending multiple responses instead of one combined
• Assuming caching replaces aggregation