gRPC for internal services in HLD - System Design Exercise

Design: Internal Microservices Communication using gRPC

Design focuses on internal service-to-service communication using gRPC. Out of scope are external client communication, persistent storage design, and UI components.

Functional Requirements

FR1: Enable efficient communication between multiple internal microservices

FR2: Support synchronous request-response calls with low latency

FR3: Allow easy definition and evolution of service APIs

FR4: Ensure strong typing and contract enforcement between services

FR5: Support load balancing and service discovery within the internal network

FR6: Provide mechanisms for error handling and retries

FR7: Secure communication within the internal network

Non-Functional Requirements

NFR1: Handle up to 10,000 requests per second across services

NFR2: Maintain p99 latency under 50ms for RPC calls

NFR3: Ensure 99.9% availability for internal service communication

NFR4: Use protobuf for message serialization

NFR5: Operate within a trusted internal network environment

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

❓ Question 6

Key Components

gRPC client and server stubs generated from protobuf definitions

Service registry or discovery mechanism (e.g., DNS, Consul, or Kubernetes)

Load balancer for distributing requests

Authentication and authorization middleware

Monitoring and logging tools for RPC calls

Retry and timeout policies

Design Patterns

API contract-first design with protobuf

Unary RPC vs streaming RPC patterns

Client-side vs server-side load balancing

Circuit breaker and retry patterns

Versioning strategies for protobuf APIs

Reference Architecture

                    +---------------------+
                    |   Service Registry   |
                    +----------+----------+
                               |
          +--------------------+--------------------+
          |                                         |
  +-------v-------+                         +-------v-------+
  |  Microservice |                         |  Microservice |
  |     A (gRPC)  |                         |     B (gRPC)  |
  +-------+-------+                         +-------+-------+
          |                                         |
          +--------------------+--------------------+
                               |
                    +----------v----------+
                    |    Load Balancer    |
                    +---------------------+

Components

gRPC Server Stub

gRPC with Protobuf

Defines and implements service APIs to handle incoming RPC calls.

gRPC Client Stub

gRPC with Protobuf

Allows microservices to call other services using generated client code.

Service Registry

Consul / Kubernetes DNS

Keeps track of available service instances and their network locations.

Load Balancer

Envoy / gRPC built-in load balancing

Distributes requests evenly across healthy service instances.

Authentication Middleware

mTLS / JWT

Secures communication between services within the internal network.

Monitoring and Logging

Prometheus / Grafana / OpenTelemetry

Tracks RPC call metrics, latency, errors for observability.

Request Flow

1. 1. Microservice A wants to call Microservice B's API.

2. 2. Microservice A uses the gRPC client stub generated from protobuf to create a request.

3. 3. The client queries the Service Registry to find available instances of Microservice B.

4. 4. The Load Balancer selects a healthy instance of Microservice B.

5. 5. The request is sent over HTTP/2 with protobuf serialization using gRPC protocol.

6. 6. Authentication middleware verifies the identity and permissions of Microservice A.

7. 7. Microservice B's gRPC server stub receives the request and processes it.

8. 8. Microservice B sends back the response via gRPC over HTTP/2.

9. 9. Microservice A receives the response and continues processing.

Database Schema

Not applicable as this design focuses on communication between services, not data storage.

Scaling Discussion

Bottlenecks

Service Registry becoming a single point of failure or bottleneck.

Load Balancer overload under high request volume.

Network latency and bandwidth limits affecting RPC call performance.

Large message sizes causing serialization/deserialization delays.

Version incompatibility between services causing communication failures.

Solutions

Use highly available and distributed service registries like Consul or Kubernetes DNS.

Implement client-side load balancing to reduce dependency on central load balancers.

Optimize network infrastructure and use compression for messages.

Limit message sizes and use streaming RPC for large data transfers.

Adopt strict API versioning and backward compatibility strategies.

Interview Tips

Time: Spend 10 minutes understanding requirements and clarifying scope, 20 minutes designing the architecture and data flow, 10 minutes discussing scaling and trade-offs, and 5 minutes summarizing.

Explain why gRPC is suitable for internal microservices communication (low latency, strong typing).

Discuss protobuf for API contract and versioning benefits.

Highlight service discovery and load balancing importance.

Mention security considerations even within internal networks.

Address how to handle failures and retries gracefully.

Talk about monitoring for observability and debugging.

Practice

1. What is the main advantage of using gRPC for communication between internal services?

easy

A. It provides fast and efficient communication using HTTP/2 and Protocol Buffers.

B. It uses plain text JSON for easy debugging.

C. It requires no schema definition for messages.

D. It only works for external client-server communication.

Solution

1. Step 1: Understand gRPC communication basics

   gRPC uses HTTP/2 for transport and Protocol Buffers for message serialization, making it fast and efficient.

2. Step 2: Compare options with gRPC features

   Options A and B are incorrect because gRPC uses binary Protocol Buffers, not JSON, and requires schema definitions. It only works for external client-server communication. is wrong because gRPC is often used internally.

3. Final Answer:

   It provides fast and efficient communication using HTTP/2 and Protocol Buffers. -> Option A

4. Quick Check:

   gRPC speed and efficiency = It provides fast and efficient communication using HTTP/2 and Protocol Buffers. [OK]

Hint: Remember gRPC uses HTTP/2 + Protocol Buffers for speed [OK]

Common Mistakes:

• Confusing gRPC with REST using JSON
• Thinking gRPC doesn't need schemas
• Assuming gRPC is only for external APIs

2. Which of the following is the correct way to define a gRPC service method in a Protocol Buffers (.proto) file?

easy

A. service UserService { function GetUser (UserRequest) returns UserResponse; }

B. service UserService { call GetUser (UserRequest) returns UserResponse; }

C. service UserService { method GetUser (UserRequest) -> UserResponse; }

D. service UserService { rpc GetUser (UserRequest) returns (UserResponse); }

Solution

1. Step 1: Recall gRPC service syntax in .proto files

   The correct syntax uses rpc MethodName(RequestType) returns (ResponseType); inside a service block.

2. Step 2: Match options to correct syntax

   Only service UserService { rpc GetUser (UserRequest) returns (UserResponse); } matches the exact gRPC syntax. Others use incorrect keywords like function, method, or call.

3. Final Answer:

   service UserService { rpc GetUser (UserRequest) returns (UserResponse); } -> Option D

4. Quick Check:

   gRPC method syntax = service UserService { rpc GetUser (UserRequest) returns (UserResponse); } [OK]

Hint: Look for 'rpc' keyword inside service block [OK]

Common Mistakes:

• Using 'function' or 'method' instead of 'rpc'
• Omitting parentheses around request/response types
• Misplacing 'returns' keyword

3. Given this gRPC service definition snippet:

service OrderService {
  rpc CreateOrder (OrderRequest) returns (OrderResponse);
}

What happens when a client calls CreateOrder with invalid data that violates the schema?

medium

A. The server returns a validation error before processing the request.

B. The server accepts the request and processes it anyway.

C. The client call fails at compile time due to schema mismatch.

D. The server crashes due to invalid input.

Solution

1. Step 1: Understand gRPC schema validation

   gRPC uses Protocol Buffers which enforce message structure. Invalid data is rejected by the server with an error.

2. Step 2: Evaluate options based on gRPC behavior

   The server returns a validation error before processing the request. is correct because the server validates input and returns errors. The server accepts the request and processes it anyway. is unsafe and incorrect. The client call fails at compile time due to schema mismatch. is wrong because client code compiles but runtime validation happens. The server crashes due to invalid input. is incorrect as servers handle errors gracefully.

3. Final Answer:

   The server returns a validation error before processing the request. -> Option A

4. Quick Check:

   gRPC input validation = The server returns a validation error before processing the request. [OK]

Hint: Invalid data triggers server validation error, not crash [OK]

Common Mistakes:

• Assuming client compile-time failure for invalid data
• Thinking server processes invalid requests
• Believing server crashes on bad input

4. You notice that your internal gRPC service calls are slow and sometimes timeout. Which of the following is the best first step to debug this issue?

medium

A. Remove Protocol Buffers and switch to JSON messages.

B. Rewrite all services to use REST instead of gRPC.

C. Check network latency and HTTP/2 connection health between services.

D. Increase client timeout without investigating.

Solution

1. Step 1: Identify common causes of gRPC slowness

   Network issues and HTTP/2 connection problems often cause delays in gRPC calls.

2. Step 2: Evaluate debugging approaches

   Check network latency and HTTP/2 connection health between services. is the best first step to diagnose root cause. Options B and C are drastic changes without diagnosis. Increase client timeout without investigating. ignores the problem source.

3. Final Answer:

   Check network latency and HTTP/2 connection health between services. -> Option C

4. Quick Check:

   Debug gRPC slowness = Check network latency and HTTP/2 connection health between services. [OK]

Hint: Start debugging network and HTTP/2 before changing code [OK]

Common Mistakes:

• Jumping to rewrite services without diagnosis
• Ignoring network as a cause
• Blindly increasing timeouts

5. You are designing an internal microservices system using gRPC. Which design choice best improves scalability and fault isolation?

hard

A. Avoid using Protocol Buffers and send raw JSON over gRPC.

B. Define small, focused gRPC services with clear method boundaries and use load balancing.

C. Use gRPC only for external APIs and REST internally.

D. Combine all functionality into one large gRPC service to reduce network calls.

Solution

1. Step 1: Understand microservices design principles with gRPC

   Small, focused services with clear APIs improve scalability and fault isolation. Load balancing helps distribute load.

2. Step 2: Evaluate design options

   Define small, focused gRPC services with clear method boundaries and use load balancing. aligns with best practices. Combine all functionality into one large gRPC service to reduce network calls. creates a monolith, hurting scalability. Use gRPC only for external APIs and REST internally. wastes gRPC benefits internally. Avoid using Protocol Buffers and send raw JSON over gRPC. breaks gRPC efficiency by avoiding Protocol Buffers.

3. Final Answer:

   Define small, focused gRPC services with clear method boundaries and use load balancing. -> Option B

4. Quick Check:

   Microservices + gRPC best practice = Define small, focused gRPC services with clear method boundaries and use load balancing. [OK]

Hint: Small services + load balancing = scalable gRPC design [OK]

Common Mistakes:

• Building large monolithic gRPC services
• Using REST internally instead of gRPC
• Ignoring Protocol Buffers benefits