Bird
Raised Fist0

gRPC for Internal Service Communication - Protocol Buffers, Streaming

HLDmediumInterview freq: medium
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
gRPC for Internal Service Communication - Protocol Buffers, Streaming
45 min · senior
senior

You are designing an internal service communication framework for a 50 million daily active user (DAU) startup. The system must support low-latency, high-throughput RPC calls between microservices with P99 latency under 200ms. All communication must be encrypted end-to-end. Services require bi-directional streaming for real-time updates and offline message delivery with guaranteed at-least-once semantics. The system should gracefully handle service restarts and network partitions without message loss or duplication.

Constraints
  • End-to-end encryption mandatory for all RPC calls
  • P99 latency under 200ms for RPC calls at peak load
  • Support bi-directional streaming for real-time updates
  • Offline message delivery with at-least-once guarantee
  • Graceful handling of service restarts and network partitions
📋
Guided Steps
1/5
Step 1
Choosing the RPC framework and serialization format
Which RPC framework and serialization format will you use for internal service communication and why?
💡 Hint
Consider performance, cross-language support, and schema evolution.
Key Decision
Use gRPC with Protocol Buffers v3 for efficient binary serialization, strong schema contracts, and native support for streaming and multiple languages.
Interviewer Expectation
Mid: Names gRPC and Protocol Buffers. Senior: Explains Protocol Buffers compactness, backward compatibility, and gRPC streaming support versus alternatives like Thrift or JSON/REST.
⚠️Common Wrong Answer
Using JSON over HTTP REST - JSON parsing overhead and lack of streaming support cause high latency and poor throughput at 50M DAU.
Step 2
Designing streaming and message delivery guarantees
How will you implement bi-directional streaming with at-least-once delivery and offline message support?
💡 Hint
Think about persistent queues and client reconnection strategies.
Key Decision
Use gRPC bi-directional streaming combined with Apache Kafka for durable message storage and replay on reconnect, ensuring at-least-once delivery and offline support.
Interviewer Expectation
Mid: Mentions Kafka for durability. Senior: Explains Kafka partitioning keyed by service or client ID to preserve order and how gRPC streams reconnect to resume from last committed offset.
⚠️Common Wrong Answer
Relying solely on in-memory buffers for streaming - leads to message loss on client or server restarts and no offline delivery.
Step 3
Ensuring end-to-end encryption and authentication
How will you secure the RPC communication to meet encryption and authentication requirements?
💡 Hint
Consider transport layer security and identity verification.
Key Decision
Use mTLS (mutual TLS) with gRPC to provide end-to-end encryption and mutual authentication between services, leveraging a centralized certificate authority like HashiCorp Vault.
Interviewer Expectation
Mid: Mentions TLS encryption. Senior: Explains mutual TLS for both client and server authentication, certificate rotation, and integration with service mesh identity management.
⚠️Common Wrong Answer
Using plain TLS without mutual authentication - vulnerable to man-in-the-middle attacks and unauthorized service access.
Step 4
Handling service restarts and network partitions
How will your design handle service restarts and network partitions without losing or duplicating messages?
💡 Hint
Think about idempotency, message offsets, and reconnection logic.
Key Decision
Implement client-side offset tracking with Kafka consumer groups and idempotent message processing in services; gRPC streams reconnect and resume from last committed offset to avoid duplicates or loss.
Interviewer Expectation
Mid: Describes reconnection and retry logic. Senior: Details Kafka offset commits, idempotent handlers, and how gRPC stream lifecycle integrates with Kafka consumer state to guarantee exactly-once processing semantics.
⚠️Common Wrong Answer
No offset tracking or idempotency - leads to message duplication or loss during failures.
Step 5
API design for core gRPC endpoints
Sketch the 3 core gRPC endpoints for sending requests, streaming updates, and acknowledging messages.
💡 Hint
Define unary RPC for requests, bi-directional streaming for updates, and unary or streaming ACKs for delivery confirmation.
Key Decision
Define: 1) Unary RPC SendRequest(Request) returns Response; 2) Bi-directional streaming StreamUpdates(stream Update) returns stream Update; 3) Unary RPC AcknowledgeMessage(Ack) returns AckResponse for delivery confirmation.
Interviewer Expectation
Mid: Defines basic unary and streaming RPCs. Senior: Explains how streaming endpoints support backpressure, flow control, and how ACKs enable at-least-once semantics with Kafka offset commits.
⚠️Common Wrong Answer
Using only unary RPCs for all communication - inefficient for real-time updates and offline delivery.
🔌
API Design Challenge
Sketch 3 core gRPC endpoints for internal service communication: sending a request, bi-directional streaming updates, and acknowledging message receipt.
Hints
  • How does the client initiate and send a unary RPC request?
  • How does the client and server maintain a bi-directional streaming connection?
  • How does the recipient acknowledge message processing to ensure delivery guarantees?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Use gRPC with Protocol Buffers for efficient serialization and streaming
  • Implement durable message storage with Apache Kafka keyed by client/service ID
  • Use mutual TLS (mTLS) for encryption and authentication
  • Design client reconnection with Kafka offset tracking and idempotent processing
Good to Have - 30%
  • Define bi-directional streaming endpoints with backpressure support
  • Centralized certificate management with Vault or similar
  • Unary RPC endpoints for request and acknowledgment patterns
Bonus - 10%
  • Explain integration with service mesh for identity and routing
  • Discuss metrics and tracing for observability of streaming RPCs
Anti-patterns to Avoid
Using JSON over HTTP REST for internal RPC - causes high latency and no streaming support, collapsing under 50M DAU load
Relying on in-memory buffers for message delivery - leads to message loss on restarts and no offline support
Using TLS without mutual authentication - exposes system to man-in-the-middle and unauthorized access
No offset tracking or idempotency - causes message duplication or loss during failures
Using only unary RPCs for all communication - inefficient for real-time streaming and offline delivery
Ignoring backpressure and flow control - leads to increased latency and potential message loss under peak load
✅
Model Answer
▼ Reveal

This design uses gRPC with Protocol Buffers v3 for efficient, cross-language RPC and streaming. Apache Kafka provides durable, ordered message storage keyed by client or service ID, enabling offline message delivery and at-least-once semantics. Mutual TLS (mTLS) secures all RPC calls with end-to-end encryption and mutual authentication, managed centrally via HashiCorp Vault. Client-side Kafka offset tracking combined with idempotent message processing ensures no message loss or duplication during service restarts or network partitions. Bi-directional gRPC streaming endpoints support real-time updates with backpressure and flow control to maintain P99 latency under 200ms. Unary RPC endpoints handle request-response and acknowledgments to complement streaming delivery guarantees.

Key Decisions
  • Use gRPC with Protocol Buffers v3 for efficient binary serialization and native streaming support
  • Leverage Apache Kafka keyed by client/service ID for ordered, durable message storage and replay on reconnect
  • Implement mutual TLS (mTLS) with certificate rotation managed by HashiCorp Vault for end-to-end encryption and authentication
  • Track Kafka consumer offsets client-side and design idempotent message handlers to handle restarts and network partitions without loss or duplication
  • Design bi-directional streaming gRPC endpoints with backpressure and flow control to maintain low latency under peak load
  • Define unary RPC endpoints for sending requests and acknowledging message receipt to complement streaming delivery guarantees
  • gRPC with Protocol Buffers v3 for efficient, schema-driven RPC and streaming
  • Apache Kafka as durable message broker keyed by client/service ID for ordered, replayable streams
  • Mutual TLS (mTLS) for end-to-end encryption and mutual authentication between services
  • Client-side Kafka offset tracking and idempotent message handlers to handle restarts and partitions
  • Bi-directional gRPC streaming endpoints to support real-time updates and offline message delivery
  • Centralized certificate management with HashiCorp Vault for secure certificate issuance and rotation
  • Backpressure and flow control via gRPC streaming to maintain P99 latency under 200ms at peak load
  • Unary RPC endpoints for request-response and acknowledgments complement streaming for delivery guarantees

Practice

(1/5)
1. In a message processing pipeline, at Step 4 the Retry Worker reads a message from the Dead Letter Queue. What happens if the message exceeds the maximum retry attempts configured?
easy
A. The Retry Worker re-enqueues the message back to the main queue indefinitely
B. The Retry Worker moves the message to a separate permanent failure store for manual inspection
C. The Retry Worker discards the message permanently without notification
D. The Retry Worker resets the retry count and retries immediately

Solution

  1. Step 1: Understand retry limit enforcement

    When max retries are exceeded, messages should not be retried endlessly to avoid infinite loops.

  2. Step 2: Identify correct failure handling

    Moving messages to a permanent failure store (C) enables manual inspection and prevents loss.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Exceeded retries -> move to permanent failure store, not discard or infinite retry [check]
Hint: Max retries exceeded -> move to failure store, not discard or loop [OK]
Common Mistakes:
2. In the request flow of the fraud detection system, at step 4 the Real-Time Scoring component queries the Feature Store. What happens if the Feature Store returns stale data due to eventual consistency?
easy
A. The scoring component uses the stale features, potentially causing inaccurate fraud scores
B. The scoring component retries the query immediately, causing increased latency
C. The scoring component blocks until fresh data arrives, delaying the transaction
D. The scoring component falls back to default feature values, ignoring the stale data

Solution

  1. Step 1: Trace scoring behavior on stale data

    The scoring component queries the Feature Store asynchronously; it does not block (A) or retry immediately (C). It uses whatever data is returned.

  2. Step 2: Understand fallback mechanisms

    Fallback to default values (D) is not automatic; stale data is used as-is, which may reduce accuracy but maintains low latency.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Stale features used -> possible inaccurate scores, no blocking [check]
Hint: Real-time scoring uses latest available features, even if stale [check]
Common Mistakes:
3. For a CI/CD system that must support both rapid iterative deployments and guaranteed rollback safety, which deployment strategy is most appropriate?
medium
A. Blue-Green Deployment, because it minimizes downtime by switching traffic instantly
B. Rolling Deployment, because it updates servers one by one without extra infrastructure
C. Recreate Deployment, because it fully replaces old versions ensuring clean state
D. Canary Deployment, because it allows gradual exposure and easy rollback on failure

Solution

  1. Step 1: Analyze rollback safety and rapid iteration needs

    Canary deployment gradually exposes new versions, enabling quick rollback if issues arise, balancing speed and safety.

  2. Step 2: Evaluate other strategies

    Blue-Green minimizes downtime but requires double infrastructure; Rolling updates lack gradual exposure and rollback granularity; Recreate causes downtime and no gradual rollback.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Canary balances rapid deployment with rollback safety [check]
Hint: Canary = gradual rollout + rollback safety [check]
Common Mistakes:
4. If the Kafka cluster responsible for fan-out message delivery crashes, what is the exact failure observed in the News Feed system?
medium
A. New posts are lost permanently and never delivered to followers
B. The system blocks new post writes until Kafka recovers, causing user-facing delays
C. Followers receive duplicate feed updates due to Kafka replaying messages
D. Delivery Workers cannot fetch fan-out messages, so followers see stale feeds until recovery

Solution

  1. Step 1: Understand Kafka's role in fan-out delivery

    Kafka acts as a durable queue for fan-out messages. If it crashes, Delivery Workers cannot fetch new messages, causing stale feeds.

  2. Step 2: Evaluate failure modes

    Posts are not lost immediately (A) due to Kafka's durability; writes are asynchronous so no blocking (B); duplicates (D) occur only on replay, not crash.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Kafka crash stalls delivery, causing stale feeds [check]
Hint: Kafka crash stalls delivery, causing stale feeds [check]
Common Mistakes:
5. For computing ETA in a city with highly variable traffic, which approach is better and why?
medium
A. Use a precomputed route cache updated daily with average travel times
B. Use a static shortest path algorithm with fixed average speeds per road segment
C. Use a heuristic-based routing ignoring traffic but optimizing for shortest distance
D. Use a dynamic routing algorithm that incorporates live traffic data and historical patterns

Solution

  1. Step 1: Compare static vs dynamic ETA computation

    Static shortest path (A) and precomputed caches (D) ignore live traffic variability, leading to inaccurate ETAs.

  2. Step 2: Evaluate heuristic and dynamic approaches

    Heuristic ignoring traffic (C) fails in variable conditions; dynamic routing (B) adapts to live and historical data for accurate ETA.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Dynamic routing with live traffic yields accurate ETAs in variable conditions [check]
Hint: Dynamic routing adapts ETA to live traffic, static can't [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard