Bird
Raised Fist0

Service Discovery - Client-Side vs Server-Side Patterns

HLDmediumInterview freq: common
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Service Discovery - Client-Side vs Server-Side Patterns
45 min · senior
senior

You are designing a service discovery system for a startup with 50 million daily active users (DAU) and a microservices architecture comprising thousands of services deployed across multiple data centers. The system must enable services to discover each other efficiently and reliably with minimal latency. The environment requires encrypted communication between components, supports intermittent network partitions, and demands a P99 latency under 200ms for service resolution. The system must also handle dynamic scaling of services and frequent instance churn.

Constraints
  • All service discovery communication must be encrypted end-to-end (TLS 1.3).
  • System must tolerate network partitions and allow offline service operation with eventual consistency.
  • P99 service resolution latency must be under 200ms at peak load.
  • Support for dynamic scaling with thousands of service instances registering/deregistering per minute.
  • Cross-data center replication with eventual consistency and conflict resolution.
📋
Guided Steps
1/5
Step 1
Choosing Client-Side vs Server-Side Discovery
Should the service discovery system use client-side or server-side discovery? What are the trade-offs at 50M DAU scale?
💡 Hint
Consider latency, scalability, and complexity of client logic versus centralized control.
Key Decision
Adopt a hybrid approach: client-side discovery using DNS + local cache for low latency, combined with a server-side discovery API backed by Consul for centralized state and health management.
Interviewer Expectation
Mid: Names client-side DNS caching and server-side Consul. Senior: Explains DNS TTL tuning for cache freshness, Consul's gossip protocol for health, and trade-offs in latency and consistency.
⚠️Common Wrong Answer
Choosing pure client-side discovery with static DNS records - leads to stale data and poor handling of dynamic scaling at 50M DAU.
Step 2
Data Store for Service Registry
What data store technology will you use to store service instance metadata and health status? Why?
💡 Hint
Think about write-heavy churn, read latency, and multi-datacenter replication.
Key Decision
Use HashiCorp Consul's Raft-based KV store for strong consistency within a data center and integrate with Apache Cassandra for cross-datacenter eventual consistency and high write throughput.
Interviewer Expectation
Mid: Mentions Consul KV store or etcd. Senior: Explains why Consul Raft is chosen for local consistency and Cassandra for scalable multi-DC replication with tunable consistency.
⚠️Common Wrong Answer
Using a single relational database for registry - causes write bottlenecks and poor horizontal scaling under high churn.
Step 3
Handling Health Checks and Stale Entries
How will the system detect unhealthy or offline service instances and remove stale entries efficiently?
💡 Hint
Consider heartbeat mechanisms, TTLs, and failure detection protocols.
Key Decision
Implement TTL-based heartbeats stored in Consul with gossip-based failure detection; clients subscribe to health events via Consul watches and fallback to local cache with TTL expiration.
Interviewer Expectation
Mid: Describes heartbeat and TTL. Senior: Details Consul's gossip failure detection, watch APIs for event-driven updates, and fallback cache expiration to avoid stale reads.
⚠️Common Wrong Answer
Relying solely on periodic polling for health checks - causes high load and delayed failure detection at scale.
Step 4
API Design for Service Registration and Lookup
Design the 3 core API endpoints for service registration, lookup, and health update. What request/response formats and security measures do you include?
💡 Hint
Think REST/gRPC, authentication, encryption, and payload efficiency.
Key Decision
Expose gRPC endpoints with mutual TLS authentication: RegisterService(RegisterRequest), LookupService(LookupRequest), and UpdateHealth(HealthUpdateRequest) using protobuf for compact payloads and streaming support.
Interviewer Expectation
Mid: Suggests REST endpoints with TLS. Senior: Explains gRPC benefits for low latency, mutual TLS for strong auth, protobuf for serialization efficiency, and streaming for health updates.
⚠️Common Wrong Answer
Using unauthenticated HTTP REST endpoints - exposes security risks and increases latency due to verbose JSON.
Step 5
Scaling and Multi-Data Center Replication
How will you ensure the service discovery system scales horizontally and replicates state across multiple data centers with eventual consistency?
💡 Hint
Consider replication protocols, conflict resolution, and latency trade-offs.
Key Decision
Use Consul's WAN federation for cross-DC replication with eventual consistency and Cassandra as a backing store for global state; resolve conflicts via last-write-wins with vector clocks for causal ordering.
Interviewer Expectation
Mid: Mentions Consul WAN federation or Cassandra replication. Senior: Explains trade-offs of eventual consistency, conflict resolution strategies, and impact on service resolution latency.
⚠️Common Wrong Answer
Attempting synchronous cross-DC replication - causes high latency and availability degradation under network partitions.
🔌
API Design Challenge
Sketch 3 core API endpoints for your service discovery system that enable service registration, service lookup, and health status updates.
Hints
  • How does a service securely register or deregister itself?
  • How does a client efficiently query for healthy service instances?
  • How does the system propagate health status changes and handle stale entries?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Clear trade-off analysis between client-side and server-side discovery at 50M DAU scale.
  • Use of Consul Raft KV store for local registry with strong consistency.
  • Use of Cassandra for multi-data center replication and high write throughput.
  • Secure API design with mutual TLS and efficient serialization (gRPC + protobuf).
  • Robust health check mechanism using TTL heartbeats and gossip failure detection.
Good to Have - 30%
  • DNS caching strategy with TTL tuning for client-side discovery latency optimization.
  • WAN federation for cross-data center replication with conflict resolution.
  • Streaming APIs for health updates to reduce polling overhead.
Bonus - 10%
  • Discussion of fallback mechanisms for offline operation and eventual consistency guarantees.
  • Detailed explanation of vector clocks or causal ordering for conflict resolution.
Anti-patterns to Avoid
Static DNS records cause stale service data and poor handling of dynamic scaling, leading to service discovery failures at scale.
Using a single relational database for service registry creates write bottlenecks and poor horizontal scaling under high churn.
Relying solely on periodic polling for health checks causes high load and delayed failure detection at 50M DAU scale.
Unauthenticated HTTP REST endpoints expose security risks and increase latency due to verbose JSON payloads.
Attempting synchronous cross-data center replication causes high latency and availability degradation during network partitions.
✅
Model Answer
▼ Reveal

The system uses a hybrid service discovery approach combining client-side DNS caching with server-side Consul APIs. Consul's Raft-based KV store ensures strong consistency locally, while Apache Cassandra provides scalable multi-data center eventual consistency. Health checks use TTL-based heartbeats and gossip failure detection with event-driven updates via Consul watches. APIs are gRPC-based with mutual TLS and protobuf serialization for secure, efficient communication. WAN federation in Consul enables asynchronous cross-DC replication with conflict resolution using vector clocks.

Key Decisions
  • Hybrid discovery approach: DNS caching plus Consul server-side API for balance of latency and consistency.
  • Consul Raft KV store chosen for strong local consistency and fast failure detection.
  • Apache Cassandra used for high write throughput and eventual consistency across data centers.
  • gRPC with mutual TLS and protobuf for secure, low-latency, and compact communication.
  • TTL-based heartbeats combined with gossip failure detection and watch APIs to efficiently detect unhealthy instances and remove stale entries.
  • WAN federation with last-write-wins conflict resolution using vector clocks to handle cross-DC replication.
  • Consul for local service registry with Raft consensus ensuring strong consistency within data centers.
  • Apache Cassandra for scalable, write-optimized, multi-data center eventual consistency storage of service metadata.
  • gRPC with mutual TLS for secure, low-latency communication between clients and discovery servers.
  • DNS caching with tuned TTLs for client-side discovery to reduce lookup latency and load on servers.
  • Gossip protocol-based failure detection in Consul for fast detection of unhealthy instances.
  • TTL-based heartbeats and Consul watches for efficient health status propagation and stale entry cleanup.
  • WAN federation in Consul for asynchronous cross-data center replication balancing consistency and latency.
  • Protobuf serialization for compact payloads and streaming APIs for real-time health updates.

Practice

(1/5)
1. In a system where a user requests access to a protected resource, at step 4 the Authorization Service checks the user's access token. What happens if the token is expired at this step?
easy
A. The Authorization Service forwards the request to the Resource Server without validation
B. The Authorization Service grants access but logs a warning for token expiry
C. The Authorization Service refreshes the token automatically and proceeds
D. The Authorization Service rejects the request and returns an HTTP 401 Unauthorized error

Solution

  1. Step 1: Understand Authorization Service behavior at token validation

    The service validates token expiry and rejects expired tokens.

  2. Step 2: Eliminate incorrect behaviors

    Granting access with expired token violates security; automatic refresh is done by Authentication Service, not Authorization; forwarding without validation breaks security.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Expired tokens cause 401 Unauthorized rejection [check]
Hint: Expired token -> Authorization rejects with 401 Unauthorized [check]
Common Mistakes:
2. In the SLO error budget tracking system, at step 4 the Error Budget Calculator receives aggregated error counts and total request counts. What state does it maintain immediately after processing these inputs?
easy
A. It resets the error budget to full capacity regardless of recent errors
B. It discards error counts if the total request count is below a threshold
C. It queues the error counts for asynchronous processing by the alerting system
D. It updates the current error budget remaining by subtracting recent error rate from total budget

Solution

  1. Step 1: Understand the role of the Error Budget Calculator at step 4

    It updates the remaining error budget by subtracting the recent error rate from the total allowed errors, reflecting current SLO compliance.

  2. Step 2: Exclude other states that contradict error budget tracking logic

    Resetting the budget ignores recent errors; queuing is done later by alerting; discarding counts would lose accuracy.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Error Budget Calculator updates remaining budget after aggregation [check]
Hint: Error budget updates subtract recent errors, not reset or discard [OK]
Common Mistakes:
3. A web API must limit incoming requests to 100 per second with bursts allowed up to 200 requests. Which component best enforces this rate limiting with burst handling?
easy
A. A fixed window counter that resets every second
B. A simple request queue that drops requests when full
C. A leaky bucket algorithm that processes requests at a fixed rate without bursts
D. A token bucket algorithm that refills tokens at a steady rate with a maximum bucket size

Solution

  1. Step 1: Identify rate limiting with burst support

    The token bucket algorithm allows tokens to accumulate up to a max bucket size, enabling bursts beyond the steady refill rate.

  2. Step 2: Eliminate other options

    Fixed window counters cause spikes at window boundaries and don't handle bursts well. Leaky bucket smooths traffic but disallows bursts. Simple queues drop excess requests without rate control.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Token bucket allows bursts by token accumulation [check]
Hint: Token bucket = steady refill + burst capacity [OK]
Common Mistakes:
4. For encrypting sensitive data at rest in a distributed database, which approach is best when the system requires frequent data reads with minimal latency and strong security guarantees?
medium
A. Use client-side encryption where clients encrypt data before sending to the database
B. Use Transparent Data Encryption (TDE) at the database layer with hardware security modules (HSMs)
C. Use application-layer encryption with keys stored in environment variables on app servers
D. Use file system-level encryption on database storage volumes

Solution

  1. Step 1: Evaluate encryption approaches for read latency and security

    TDE with HSMs provides strong security with minimal impact on read latency as encryption/decryption is transparent and hardware-accelerated.

  2. Step 2: Eliminate less suitable options

    Client-side encryption increases latency and complicates querying; app-layer encryption with env vars risks key exposure and adds latency; file system encryption protects at disk level but not granular data access.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    TDE + HSM balances security and low-latency reads [check]
Hint: TDE with HSMs -> strong security + low latency reads [check]
Common Mistakes:
5. At 10x scale, which component in a microservices architecture is most likely to become the first bottleneck, and why?
hard
A. The shared database, because it cannot be sharded easily without complex coordination
B. The individual microservices, because their stateful WebSocket connections limit horizontal scaling
C. The service discovery system, because it must track exponentially more services and instances
D. The API Gateway, because it handles all incoming requests and becomes a centralized choke point

Solution

  1. Step 1: Identify stateful components at risk

    Stateful microservices holding WebSocket connections cannot easily shard or move connections, limiting horizontal scaling.

  2. Step 2: Compare other components' scaling characteristics

    Shared DB and service discovery can be sharded or cached; API Gateway can be scaled horizontally with load balancers.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Stateful WebSocket connections limit microservices horizontal scaling [check]
Hint: Stateful WebSocket limits microservices scaling first [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard