Bird
Raised Fist0

Microservices vs Monolith - When to Split and the Hidden Costs

HLDmediumInterview freq: common
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Microservices vs Monolith - When to Split and the Hidden Costs
45 min · senior
senior

You are an engineering lead at a 50 million daily active user (DAU) startup currently running a monolithic backend. The product is growing rapidly, and the team is debating whether to split the monolith into microservices. The system must maintain P99 latency under 200ms, support end-to-end encryption, and allow offline delivery of notifications. The engineering team is small but growing, and operational complexity must be manageable.

Constraints
  • End-to-end encryption required for all user data in transit and at rest
  • Offline delivery of notifications with guaranteed eventual consistency
  • P99 latency under 200ms for core user-facing APIs
  • Limited engineering resources for operational overhead
  • Rapid feature iteration with minimal cross-team dependencies
📋
Guided Steps
1/5
Step 1
Assessing Scale and Latency Requirements
How do you evaluate if the current monolith can meet the 50M DAU scale and P99 < 200ms latency without splitting?
💡 Hint
Consider request volume, resource contention, and latency tail at scale.
Key Decision
Use distributed tracing with OpenTelemetry and load testing with Locust to measure current monolith latency and throughput under 50M DAU simulated load.
Interviewer Expectation
Mid: Names load testing and tracing tools. Senior: Explains how tail latency grows non-linearly with contention and why monolith resource saturation causes P99 spikes.
⚠️Common Wrong Answer
Assuming monolith scales linearly without testing - leads to unexpected latency spikes and outages at scale.
Step 2
Identifying Service Boundaries
What criteria do you use to decide which parts of the monolith to split into microservices?
💡 Hint
Think about bounded contexts, data ownership, and team boundaries.
Key Decision
Define service boundaries based on domain-driven design (DDD) principles and data ownership, using Apache Kafka topics per bounded context for event-driven decoupling.
Interviewer Expectation
Mid: Mentions splitting by functionality. Senior: Explains DDD bounded contexts, Kafka topic partitioning for ordering, and how this reduces coupling and improves scalability.
⚠️Common Wrong Answer
Splitting by technical layers (e.g., UI vs DB) - causes tight coupling and complex cross-service transactions.
Step 3
Handling Encryption and Data Privacy
How do you ensure end-to-end encryption across microservices without degrading performance beyond 200ms P99?
💡 Hint
Consider encryption at transport and storage layers, and key management.
Key Decision
Use TLS 1.3 for transport encryption, AWS KMS for centralized key management, and envelope encryption with client-side encryption libraries to minimize decryption overhead in services.
Interviewer Expectation
Mid: Mentions TLS and encryption. Senior: Details envelope encryption benefits, key rotation strategies, and how to avoid decrypting data unnecessarily in microservices to maintain latency SLAs.
⚠️Common Wrong Answer
Encrypting/decrypting data in every microservice synchronously - causes latency spikes and resource exhaustion.
Step 4
Managing Offline Delivery and Eventual Consistency
How do you design offline notification delivery with eventual consistency in a microservices architecture?
💡 Hint
Think about message queues, retry policies, and idempotency.
Key Decision
Use Apache Kafka with compacted topics for event sourcing, combined with Redis Streams for offline delivery queues and idempotent consumer logic to ensure exactly-once processing.
Interviewer Expectation
Mid: Names Kafka and retry queues. Senior: Explains Kafka compaction for state recovery, Redis Streams for offline buffering, and idempotency to avoid duplicate notifications.
⚠️Common Wrong Answer
Using simple HTTP retries without durable queues - leads to message loss and inconsistent offline state.
Step 5
Evaluating Operational Complexity and Team Readiness
How do you balance the operational overhead of microservices with the need for rapid iteration and small engineering teams?
💡 Hint
Consider automation, monitoring, and team ownership models.
Key Decision
Adopt Kubernetes with Helm charts for deployment automation, Prometheus + Grafana for monitoring, and organize teams around services with clear ownership to reduce cognitive load.
Interviewer Expectation
Mid: Mentions containerization and monitoring. Senior: Explains how Kubernetes operators automate lifecycle, how SLO-based alerting reduces toil, and how team ownership boundaries reduce coordination overhead.
⚠️Common Wrong Answer
Ignoring operational overhead and splitting prematurely - causes deployment failures and slow feature delivery.
🔌
API Design Challenge
Sketch 3 core API endpoints that illustrate your chosen architecture for either monolith or microservices approach.
Hints
  • How does the client send data or requests to the backend?
  • How does the client synchronize state or receive updates?
  • How does the system acknowledge or confirm processing to the client?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Clear criteria for when to split monolith based on scale and latency measurements
  • Use of domain-driven design to define service boundaries
  • Secure encryption strategy that balances security and latency
  • Reliable offline delivery mechanism with idempotency and eventual consistency
  • Consideration of operational complexity and team readiness
Good to Have - 30%
  • Use of distributed tracing and load testing tools
  • Event-driven architecture with Kafka topic partitioning
  • Automated deployment and monitoring with Kubernetes and Prometheus
  • Explicit team ownership and reduced cross-service coupling
Bonus - 10%
  • Discussion of key rotation and envelope encryption details
  • Explanation of Kafka compaction and Redis Streams for offline delivery
  • SLO-based alerting to reduce operational toil
Anti-patterns to Avoid
Splitting services by technical layers rather than business domains - causes tight coupling and complex transactions that are hard to maintain and scale
Encrypting/decrypting data synchronously in every microservice - leads to latency spikes and resource exhaustion, violating P99 latency SLAs
Using HTTP polling or simple retries for offline delivery - causes message loss and inconsistent offline state due to lack of durable queues
Ignoring operational overhead and splitting prematurely - results in deployment failures and slow feature delivery due to increased complexity
Assuming monolith scales linearly without testing - leads to unexpected outages and latency spikes under real load
Lack of team ownership boundaries - increases coordination overhead and slows feature delivery by causing confusion and duplicated efforts
✅
Model Answer
▼ Reveal

At 50M DAU with strict latency and security requirements, splitting the monolith into microservices is justified when load testing and tracing reveal resource saturation causing P99 latency spikes. Defining service boundaries using Domain-Driven Design aligned with business domains and Kafka topic partitioning reduces coupling and improves scalability. End-to-end encryption is ensured using TLS 1.3, AWS KMS for key management, and envelope encryption to minimize decryption overhead. Offline notification delivery is handled with Kafka compacted topics and Redis Streams to guarantee eventual consistency and exactly-once processing. Operational complexity is managed by adopting Kubernetes with Helm for deployment automation, Prometheus and Grafana for monitoring, and organizing teams around clear service ownership to reduce coordination overhead.

Key Decisions
  • Use OpenTelemetry and Locust to validate monolith scalability and latency under 50M DAU load
  • Define microservice boundaries using Domain-Driven Design and Kafka topic partitioning to reduce coupling
  • Implement TLS 1.3 for transport encryption and AWS KMS with envelope encryption to balance security and latency
  • Design offline notification delivery with Kafka compacted topics and Redis Streams for idempotent, exactly-once processing
  • Adopt Kubernetes with Helm for automated deployment and scaling, and Prometheus/Grafana for monitoring
  • Organize engineering teams around service ownership to minimize cross-team dependencies and operational overhead
  • OpenTelemetry distributed tracing to measure latency and identify bottlenecks in monolith
  • Domain-Driven Design (DDD) to define microservice boundaries aligned with business domains
  • Apache Kafka for event-driven communication with topic partitioning per bounded context
  • TLS 1.3 and AWS KMS for secure transport and centralized key management with envelope encryption
  • Redis Streams for offline delivery queues supporting idempotent and exactly-once processing
  • Kubernetes with Helm for automated deployment and scaling of microservices
  • Prometheus and Grafana for real-time monitoring and alerting on service health and latency
  • Team ownership model aligned with microservices to reduce cross-team dependencies and improve velocity

Practice

(1/5)
1. In a system where Step 4 is the Delivery Worker reading from a Kafka topic partitioned by user ID, what is the Delivery Worker's state if it reads a message but the downstream notification service is temporarily unavailable?
easy
A. The message is marked as delivered and removed from Kafka
B. The Delivery Worker skips the message and moves to the next
C. The message is lost because Kafka does not guarantee delivery
D. The message remains uncommitted in Kafka and will be retried

Solution

  1. Step 1: Understand Kafka consumer commit semantics

    Kafka commits offsets only after successful processing. If downstream is unavailable, the message is not committed.

  2. Step 2: Delivery Worker retries uncommitted messages

    Uncommitted messages remain in the topic and will be redelivered on retry, ensuring no message loss.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Kafka offset commit after processing -> retries on failure [check]
Hint: Kafka commits offsets only after successful processing [OK]
Common Mistakes:
2. For a high-traffic e-commerce site needing to invalidate product pages immediately after price updates, which cache invalidation strategy is best and why?
medium
A. Use surrogate keys to invalidate all related product pages atomically
B. Use client-side cache control headers to force revalidation
C. Use short TTL to ensure freshness but accept higher cache misses
D. Use long TTL with manual purge to minimize cache misses

Solution

  1. Step 1: Identify need for immediate, atomic invalidation

    Price updates require instant invalidation of all related pages to avoid stale data.

  2. Step 2: Evaluate strategies

    Long TTL with manual purge delays invalidation; short TTL causes high cache misses; client-side headers don't guarantee server cache invalidation; surrogate keys enable atomic invalidation of related content.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Surrogate keys enable atomic, immediate invalidation of related cache [check]
Hint: Surrogate keys invalidate related content atomically [check]
Common Mistakes:
3. You must design a distributed key-value store that tolerates network partitions and prioritizes availability over consistency. Which consensus approach fits best?
medium
A. Use Raft to enforce strong consistency with leader election
B. Use Paxos to guarantee consensus before any write is committed
C. Use eventual consistency with conflict resolution, avoiding strict consensus
D. Use a centralized lock service to serialize all writes

Solution

  1. Step 1: Analyze CAP trade-offs

    Prioritizing availability over consistency during partitions means relaxing strict consensus.

  2. Step 2: Evaluate options

    Raft and Paxos enforce strong consistency, sacrificing availability; centralized locks create single points of failure; eventual consistency allows availability with conflict resolution.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Eventual consistency favors availability during partitions [check]
Hint: Availability over consistency -> eventual consistency with conflict resolution [check]
Common Mistakes:
4. For storing aggregated logs with fast query response and cost efficiency, which storage architecture is best when logs are mostly write-once and rarely updated?
medium
A. Use a distributed relational database with strong consistency
B. Use a NoSQL key-value store with frequent compactions
C. Use a columnar data warehouse optimized for append-only writes
D. Use a distributed file system with manual indexing

Solution

  1. Step 1: Analyze log storage requirements

    Logs are write-once, large volume, and require fast analytical queries.

  2. Step 2: Evaluate columnar data warehouse

    Columnar stores optimize for append-only writes and fast scans, reducing cost and improving query speed.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Columnar warehouse -> optimized for append-only and analytics [check]
Hint: Columnar storage excels at append-only analytics [check]
Common Mistakes:
5. At 10x transaction volume, which component of a relational database system is most likely to become the bottleneck first, and why?
hard
A. The storage engine, because disk I/O bandwidth saturates quickly
B. The query optimizer, because it cannot generate execution plans fast enough
C. The lock manager, due to increased contention causing transaction delays
D. The network layer, as increased client connections overwhelm bandwidth

Solution

  1. Step 1: Identify bottleneck under high concurrency

    Lock manager handles concurrency control; at 10x load, lock contention increases exponentially.

  2. Step 2: Compare other components

    Query optimizer scales well with caching; storage and network scale with hardware upgrades; lock manager contention is harder to mitigate.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Lock manager contention -> first bottleneck at high transaction scale [check]
Hint: Lock manager contention grows fastest at scale [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard