Bird
Raised Fist0

gRPC for Internal Service Communication - Protocol Buffers, Streaming

HLDmediumInterview freq: medium
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🏗️
gRPC for Internal Service Communication - Protocol Buffers, Streaming
Designing a scalable, low-latency internal RPC system using gRPC with Protocol Buffers and streaming
mediumAPI Design and GatewayFreq: medium
⚙️
Functional Requirements
  • Support synchronous unary RPC calls between internal microservices with low latency (<50ms P99).
  • Support server-side streaming RPC for event notifications and data feeds.
  • Use Protocol Buffers as the serialization format for compact, fast message encoding.
  • Provide built-in authentication and authorization for internal service calls.
  • Enable automatic load balancing and service discovery for RPC endpoints.
  • Support deadline propagation and cancellation for graceful request termination.
📊
Non-Functional Requirements
  • Handle 2 billion daily internal RPC calls across all services (~23K QPS average).
  • Achieve 99.99% availability for RPC calls within the internal network.
  • Ensure end-to-end RPC latency P99 under 100ms under peak load.
  • Support horizontal scaling to handle 10x traffic growth without redesign.
  • Minimize CPU and network overhead from serialization/deserialization.
❓
Clarifying Questions to Ask
  • What is the expected peak QPS and average request size for internal RPC calls?
  • Are all services deployed within the same data center or across multiple regions?
  • Is mutual TLS required for authentication or is a simpler token-based auth sufficient?
  • What are the failure handling expectations for streaming RPCs (retries, reconnections)?
  • Should the system support bi-directional streaming or only unary and server streaming?
  • Is backward compatibility for Protocol Buffers schema evolution a requirement?
Out of Scope
✗ Designing external API gateways or public-facing APIs.
✗ Implementing persistent storage or database design for service data.
✗ Cross-data-center replication or multi-region failover.
✗ Client SDK generation or language-specific client libraries.
📌
Assumptions
  • 2 billion internal RPC calls per day across all services.
  • Average message size per RPC request/response is 100 bytes.
  • Each user generates 40 RPC calls per day internally.
  • Peak traffic is 3x average QPS during business hours.
⚡
QPS Calculation
write qps2B calls/day × 1 request per call / 86400 seconds ≈ 23,148 QPS
peak write qps23,148 QPS × 3 (peak factor) ≈ 69,444 QPS
read qpsAssuming 1 response per request, read QPS ≈ write QPS
💾
Storage Estimate
daily storage2B calls × 100 bytes = 200 GB/day
retention days7
total storage200 GB/day × 7 days = 1.4 TB
with replication1.4 TB × 3 replicas = 4.2 TB total replicated storage
🖥️
Server Estimate
grpc serversAssuming each server handles 2000 QPS, need 35 servers (69,444/2000 ≈ 35)
service discovery3 instances of Consul for high availability and low latency service discovery
🏗️
Architecture Diagram
Architecture Diagram
          +----------------------+
          |  Internal Client Svc  |
          +----------+-----------+
                     | gRPC Unary/Streaming
                     v
          +----------------------+
          |  gRPC Load Balancer   |
          +----------+-----------+
                     |
          +----------+-----------+
          |  gRPC Server Pool     |
          | (Multiple Instances)  |
          +----------+-----------+
                     |
          +----------+-----------+
          |  Service Discovery    |
          |    (Consul)           |
          +----------------------+
🧩
Components
gRPC Server Pool
Hosts internal microservices exposing gRPC endpoints for RPC calls
Why thisgRPC provides efficient HTTP/2 multiplexing, built-in streaming, and native Protocol Buffers support for low-latency RPC
Why notREST over HTTP/1.1 lacks multiplexing and streaming efficiency, increasing latency and CPU overhead at scale
Protocol Buffers
Serialization format for compact, fast encoding of RPC messages
Why thisProtobufs generate efficient binary payloads with backward compatibility and schema evolution support
Why notJSON is verbose and slower to parse, increasing network and CPU usage under high QPS
gRPC Load Balancer
Distributes incoming RPC calls across healthy gRPC server instances
Why thisEnvoy proxy supports gRPC-aware load balancing with HTTP/2 connection multiplexing and health checks
Why notDNS round-robin lacks health awareness and connection multiplexing, causing uneven load and higher latency
Service Discovery (Consul)
Maintains dynamic registry of available gRPC server instances for load balancer
Why thisConsul provides low-latency, strongly consistent service registry with health checks and multi-data center support
Why notStatic config or DNS-based discovery cannot handle dynamic scaling and failover at 70K QPS
Authentication & Authorization Middleware
Enforces mutual TLS or token-based auth for secure internal RPC communication
Why thisgRPC interceptors enable transparent auth enforcement with minimal latency impact
Why notImplementing auth at application layer increases complexity and risk of inconsistent enforcement
🔄
Data Flow
1
Internal Client Service → gRPC Load Balancer
Client initiates unary or streaming RPC call over HTTP/2
Uses Protocol Buffers for message serialization
2
gRPC Load Balancer → gRPC Server Pool
Load balancer routes RPC call to healthy server instance
Uses service discovery info for routing and connection multiplexing
3
gRPC Server Pool → Authentication Middleware
Server verifies client identity and permissions
Supports mutual TLS or token validation
4
gRPC Server Pool → Internal Service Logic
Server processes RPC request and streams response if applicable
Supports unary and server streaming RPCs
5
gRPC Server Pool → gRPC Load Balancer
Server sends response back over HTTP/2 stream
Load balancer forwards response to client
6
gRPC Load Balancer → Internal Client Service
Client receives RPC response or stream data
Client handles deadline and cancellation propagation

Practice

(1/5)
1. A database needs to support range queries on a large numeric column with frequent inserts and updates. Which indexing method best supports efficient range scans while maintaining balanced tree structure?
easy
A. Composite index on multiple columns, to speed up multi-column equality queries
B. Hash index, because it provides constant-time lookups for any key
C. B-Tree index, because it maintains sorted order and supports efficient range queries
D. Bitmap index, because it compresses well for high-cardinality numeric data

Solution

  1. Step 1: Identify index type for range queries

    B-Tree indexes maintain sorted order of keys, enabling efficient range scans by traversing leaf nodes sequentially.

  2. Step 2: Eliminate other options

    Hash indexes do not preserve order, so range queries require scanning all buckets. Composite indexes optimize multi-column equality but don't inherently improve single-column range scans. Bitmap indexes are efficient for low-cardinality columns, not large numeric ranges.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    B-Tree supports ordered traversal for range queries [check]
Hint: B-Tree keeps keys sorted for range queries [check]
Common Mistakes:
2. In a distributed lock system using ZooKeeper, at step 4 the client attempts to acquire the lock by creating an ephemeral sequential node. What happens if the client crashes immediately after creating this node but before confirming lock ownership?
easy
A. ZooKeeper automatically deletes the ephemeral node, releasing the lock
B. The ephemeral node remains until manually deleted, causing a deadlock
C. The client retains the lock until a timeout expires
D. The lock is transferred to the next waiting client immediately

Solution

  1. Step 1: Understand ephemeral node semantics

    Ephemeral nodes in ZooKeeper are automatically deleted when the client session ends (e.g., crash), preventing deadlocks (A) and stale locks.

  2. Step 2: Trace lock ownership after crash

    Since the ephemeral node is deleted, the next client waiting for the lock will be notified, but the lock is not immediately transferred (D) until they acquire it explicitly.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Ephemeral node deletion on client crash -> lock released [check]
Hint: Ephemeral nodes vanish on client crash, freeing locks [check]
Common Mistakes:
3. If the Kafka cluster responsible for photo upload event streaming goes down, what exactly breaks in the system?
medium
A. Photo uploads fail immediately with HTTP 500 errors to users
B. Feed generation continues normally but stories fail to update
C. Photo metadata extraction and thumbnail generation stop; uploads succeed but processing is delayed indefinitely
D. User notifications about new photos are lost but photos appear in feed

Solution

  1. Step 1: Understand Kafka's role in event streaming

    Kafka streams photo upload events to downstream processors like metadata extraction and feed generation.

  2. Step 2: Trace failure impact

    If Kafka is down, photo uploads still succeed because upload service writes to storage directly. Feed generation depends on Kafka events for stories update; without Kafka, stories updates stop, but feed generation for photos continues from DB. Metadata extraction is delayed but not necessarily indefinite. Notifications are handled separately.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Kafka outage stops stories update but feed photo generation continues [check]
Hint: Kafka outage blocks stories update, feed photos still served [check]
Common Mistakes:
4. At 10x scale, which component in a content negotiation versioning system is most likely to become the bottleneck, and why?
hard
A. Backend services, because they must support multiple versions simultaneously increasing resource usage
B. API gateway, because parsing and routing based on Accept headers is CPU-intensive and cannot be cached easily
C. CDN caches, because they cannot cache responses differentiated only by headers effectively
D. Client SDKs, because they must handle multiple media types increasing complexity and errors

Solution

  1. Step 1: Identify component handling content negotiation

    The API gateway parses Accept headers to route requests to correct versioned backend.

  2. Step 2: Analyze scaling impact

    Parsing headers is CPU-intensive and cannot leverage caching well, causing gateway CPU to saturate first at high request volumes.

  3. Step 3: Compare other components

    Backends scale horizontally; CDNs struggle but can be configured; client SDKs are not server bottlenecks.

  4. Final Answer:

    Option B -> Option B

  5. Quick Check:

    Gateway CPU parsing Accept headers saturates first at scale [check]
Hint: Gateway CPU parsing Accept headers saturates first [check]
Common Mistakes:
5. At 10x scale, which component in the real-time ML inference system is most likely to fail first and why?
hard
A. Model storage backend, due to increased read latency from large model files
B. Load balancer, since it cannot efficiently distribute traffic to a rapidly scaling number of servers
C. Kafka event stream, as partition count limits throughput and causes lag under heavy update load
D. Prediction API servers, because CPU-intensive model inference cannot be parallelized easily

Solution

  1. Step 1: Analyze scaling bottlenecks

    Model storage (A) scales with caching; Kafka (C) can be scaled by adding partitions; load balancers (D) scale horizontally.

  2. Step 2: Identify CPU bottleneck in inference

    Prediction API servers (B) perform CPU/GPU-bound inference; scaling is limited by hardware and model complexity, causing first failure under 10x load.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Inference servers hit CPU/GPU limits before storage or Kafka scale [check]
Hint: Inference servers hit CPU/GPU bottleneck before other components [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard