Bird
Raised Fist0

Document Databases - MongoDB Data Modeling Patterns at Scale

HLDmediumInterview freq: medium
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🏗️
Document Databases - MongoDB Data Modeling Patterns at Scale
Designing scalable MongoDB data models for high throughput and low latency
mediumDatabases - NoSQLFreq: medium
⚙️
Functional Requirements
  • Support flexible schema for semi-structured documents with nested fields
  • Efficiently handle 2 billion daily active users generating ~40 writes per user per day
  • Provide low-latency reads with 100ms P99 response time for common queries
  • Support complex queries including filtering on nested fields and aggregation
  • Ensure strong consistency for user-specific data updates
  • Enable horizontal scaling and sharding to handle peak write QPS
📊
Non-Functional Requirements
  • System must sustain 925,000 writes per second at peak (2B users * 40 writes/day / 86400 sec)
  • Read latency P99 under 100ms for 95% of queries
  • Availability SLA of 99.99% with automatic failover
  • Data retention of 30 days with efficient archival
  • Storage growth capped to under 10PB with replication
❓
Clarifying Questions to Ask
  • What are the primary query patterns and access frequencies for documents?
  • Are documents mostly read-heavy, write-heavy, or balanced?
  • What is the average document size and maximum expected document size?
  • Is strong consistency required for all operations or eventual consistency acceptable for some?
  • What is the expected growth rate of data and user base over the next 1-2 years?
  • Are there any constraints on schema evolution or versioning?
Out of Scope
✗ Designing the full application layer or business logic
✗ Handling multi-region global replication and geo-distribution
✗ Designing backup and disaster recovery beyond replication
✗ Real-time analytics or OLAP workloads
📌
Assumptions
  • 2 billion daily active users (DAU)
  • Each user generates 40 writes per day on average
  • Average document size is 100 bytes per write
  • Retention period is 30 days
⚡
QPS Calculation
write qps2,000,000,000 users * 40 writes/user/day / 86,400 sec/day ≈ 925,926 writes/sec
peak write qpsAssuming 2× peak factor: 925,926 * 2 = 1,851,852 writes/sec
read qpsAssuming 3 reads per write: 925,926 * 40 * 3 / 86,400 ≈ 2,777,778 reads/sec
💾
Storage Estimate
daily storage2,000,000,000 * 40 * 100 bytes = 8,000,000,000,000 bytes ≈ 7.28 TB/day
retention storage7.28 TB/day * 30 days = 218.4 TB
with replicationAssuming 3 replicas: 218.4 TB * 3 = 655.2 TB total storage
🖥️
Server Estimate
mongod shardsAssuming each shard handles 10,000 writes/sec, need ~185 shards (1,851,852 / 10,000)
mongos proxiesAssuming each mongos handles 10,000 QPS, need ~278 mongos routers (2,777,778 / 10,000)
🏗️
Architecture Diagram
Architecture Diagram
         +----------------+
         |   Client Apps  |
         +-------+--------+
                 |
         +-------v--------+
         |   Mongos Proxy |
         +-------+--------+
                 |
   +-------------+--------------+
   |             |              |
 +--v--+       +--v--+        +--v--+
 |Shard|       |Shard|        |Shard|
 |  1  |       |  2  |  ...   |  N  |
 +--+---+       +--+---+        +--+--+
    |             |              |
 +--v--+       +--v--+        +--v--+
 |ReplicaSet|  |ReplicaSet|   |ReplicaSet|
 +----------+ +----------+   +----------+
🧩
Components
MongoDB Sharded Cluster
Stores user documents distributed across shards for horizontal scaling
Why thisMongoDB native sharding supports automatic data partitioning by shard key, enabling linear write scalability to 1.8M writes/sec
Why notCassandra offers high write throughput but lacks rich document querying and flexible schema needed for nested documents
Mongos Query Routers
Route client queries to appropriate shards based on shard key
Why thisMongos provides transparent routing and query aggregation across shards, simplifying client logic
Why notDirect shard access complicates client logic and breaks abstraction, increasing error risk
Replica Sets
Provide high availability and read scaling with primary-secondary replication
Why thisMongoDB replica sets enable automatic failover and strong consistency on primary writes
Why notMaster-slave replication lacks automatic failover and can cause data loss on failover
Document Embedding Pattern
Embed related data within a single document to optimize read performance
Why thisEmbedding reduces the need for expensive joins and multiple queries, improving 100ms P99 read latency
Why notReferencing requires multiple queries and client-side joins, increasing latency and complexity
Reference Pattern with $lookup Aggregation
Reference related documents when embedding causes document size to exceed 16MB limit
Why thisMongoDB $lookup supports server-side joins for referenced documents, balancing flexibility and size constraints
Why notManual client-side joins increase latency and complexity, especially at 2B DAU scale
TTL Indexes
Automatically expire documents after retention period to control storage growth
Why thisMongoDB TTL indexes natively delete expired documents without manual cleanup jobs
Why notManual cron jobs for cleanup add operational complexity and risk of stale data
🔄
Data Flow
1
Client App → Mongos Proxy
Send write/read request with shard key
Shard key enables Mongos to route request to correct shard
2
Mongos Proxy → Shard Primary
Route write to primary replica of shard
Primary handles strong consistency writes
3
Shard Primary → Replica Set Secondaries
Replicate write asynchronously
Secondaries serve read queries for load balancing
4
Client App → Mongos Proxy
Send read request with shard key
Mongos routes to secondary or primary based on read preference
5
Shard Replica → Mongos Proxy
Return query results
Aggregation and $lookup performed on shard for referenced data
6
Mongos Proxy → Client App
Return aggregated response
Client receives fully resolved document

Practice

(1/5)
1. A system must handle millions of short video uploads per day, ensuring fast ingestion and metadata extraction without blocking the user. Which component is best suited to decouple upload ingestion from processing?
easy
A. Use a distributed message queue like Kafka to buffer uploads for asynchronous processing
B. Use a relational database to store uploads synchronously during user upload
C. Use a CDN edge cache to store uploads temporarily before processing
D. Use a REST API gateway that directly triggers video transcoding synchronously

Solution

  1. Step 1: Identify ingestion decoupling need

    Relational DB (A) blocks user upload due to synchronous writes; CDN (C) caches for delivery, not ingestion; REST API (D) triggers blocking transcoding.

  2. Step 2: Choose buffering component

    Kafka (B) buffers uploads asynchronously, enabling fast user response and reliable processing.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Kafka buffers uploads asynchronously for scalable ingestion [check]
Hint: Buffer uploads asynchronously with Kafka queue [OK]
Common Mistakes:
2. In a distributed cache using consistent hashing, at Step 4 the Cache Node receives a key lookup request. What happens if the node does not have the key locally?
easy
A. The node returns a cache miss immediately without forwarding
B. The node uses the consistent hash ring to forward the request to the correct node
C. The node queries a centralized metadata service for key location
D. The node broadcasts the request to all other cache nodes

Solution

  1. Step 1: Trace key lookup at Cache Node

    If the node lacks the key, it must find the correct node responsible for that key using the consistent hash ring.

  2. Step 2: Forwarding via consistent hash ring

    The node consults the hash ring to identify the node owning the key and forwards the request there; centralized metadata (The node queries a centralized metadata service for key location) adds latency and single point of failure; broadcasting (The node broadcasts the request to all other cache nodes) is inefficient.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Consistent hash ring lookup enables correct node forwarding [check]
Hint: Cache miss triggers consistent hash ring forwarding [check]
Common Mistakes:
3. For a high-throughput payment processing system, which approach best balances failure isolation and resource utilization?
medium
A. Implement bulkheads with separate thread pools per payment type to isolate failures
B. Use a single large thread pool shared by all payment types to maximize utilization
C. Use circuit breakers on the shared thread pool to isolate failures dynamically
D. Rely on autoscaling the shared thread pool based on load metrics

Solution

  1. Step 1: Evaluate shared thread pool risks

    A single shared pool (A) risks cascading failures; circuit breakers (C) detect failures but don't isolate resource exhaustion. Autoscaling (D) reacts late and may not prevent blast radius.

  2. Step 2: Assess bulkhead benefits

    Bulkheads (B) isolate failures per payment type, preventing one type's overload from affecting others, balancing isolation and utilization.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Bulkheads isolate failures per resource partition [check]
Hint: Bulkheads isolate by partitioning resources per failure domain [check]
Common Mistakes:
4. For a mobile client calling a flaky backend service, which approach best balances responsiveness and failure isolation?
medium
A. Use synchronous calls with retries and exponential backoff
B. Rely on client-side caching to avoid backend calls during failures
C. Implement a circuit breaker with a short timeout and fallback response
D. Use a message queue to buffer requests and process asynchronously

Solution

  1. Step 1: Evaluate synchronous retries (A)

    Retries increase latency and risk cascading failures, bad for mobile responsiveness.

  2. Step 2: Compare asynchronous queue (C) and caching (D)

    Queues add delay and complexity; caching avoids calls but may serve stale data, not ideal for dynamic data.

  3. Step 3: Circuit breaker with short timeout and fallback (B)

    Provides quick failure detection, prevents cascading failures, and returns fallback to maintain responsiveness.

  4. Final Answer:

    Option C -> Option C

  5. Quick Check:

    Circuit breaker with fallback balances isolation and responsiveness [check]
Hint: Circuit breaker with fallback keeps mobile responsive and isolated [OK]
Common Mistakes:
5. If the machine ID component is misconfigured and two machines generate Snowflake IDs with the same machine ID simultaneously, what is the most likely failure mode?
medium
A. The sequence number will automatically adjust to prevent collisions across machines.
B. Duplicate IDs will be generated within the same millisecond causing data corruption or overwrite.
C. IDs generated on both machines will be unique due to different timestamps, no collision occurs.
D. The system will detect the conflict and block one machine from generating IDs.

Solution

  1. Step 1: Understand machine ID role in uniqueness

    Machine ID differentiates ID generators on different machines; same machine ID risks collisions.

  2. Step 2: Analyze collision handling

    Different timestamps alone (A) do not guarantee uniqueness if clocks are synchronized; sequence number does not auto-adjust across machines (C is incorrect). System does not detect conflicts automatically (D).

  3. Step 3: Correct failure mode

    Duplicate IDs are generated if machine IDs collide, causing data corruption or overwrite.

  4. Final Answer:

    Option A -> Option A

  5. Quick Check:

    Same machine ID + same timestamp -> duplicate IDs and data corruption [check]
Hint: Same machine ID causes duplicate IDs, no auto correction [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard