Bird
Raised Fist0

Blob Store Design - S3-Like Object Storage Architecture End to End

HLDmediumInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🏗️
Blob Store Design - S3-Like Object Storage Architecture End to End
Design a scalable, durable, and highly available object storage system similar to Amazon S3
mediumStorage SystemsFreq: high
Prerequisites:System Design FundamentalsCAP TheoremDistributed Storage SystemsNetworking Basics
⚙️
Functional Requirements
  • Store and retrieve objects (blobs) of arbitrary size with a unique key (bucket + object key).
  • Support object metadata (e.g., content-type, creation time, versioning).
  • Provide strong read-after-write consistency for PUTs and DELETEs.
  • Support multipart uploads for large objects.
  • Allow bucket-level namespace isolation and access control.
  • Support object lifecycle policies (e.g., automatic deletion or archival).
📊
Non-Functional Requirements
  • Handle 2 billion daily active users (DAU) globally.
  • Provide 99.99% availability SLA for object GET and PUT operations.
  • Achieve 100ms P99 latency for small object GETs and PUTs.
  • Ensure 11 nines (99.999999999%) durability annually for stored objects.
  • Scale to store exabytes of data with linear scalability.
❓
Clarifying Questions to Ask
  • What is the expected average and max object size?
  • Is strong consistency required globally or per bucket?
  • What is the expected read-to-write ratio?
  • Are versioning and object immutability required?
  • What is the retention period for objects?
  • Are there any specific compliance or encryption requirements?
Out of Scope
✗ Designing client SDKs or authentication protocols (assume existing IAM).
✗ Detailed billing or quota management systems.
✗ Cross-region replication and disaster recovery (only single region design).
✗ Advanced analytics or search on object contents.
📌
Assumptions
  • 2 billion DAU globally
  • Each user uploads 40 objects per day on average
  • Average object size is 100 KB
  • Read-to-write ratio is 5:1
⚡
QPS Calculation
write qps2B users × 40 objects/day ÷ 86400 sec/day ≈ 925,926 writes/sec
peak write qpsAssuming 3× peak factor: 925,926 × 3 ≈ 2.78M writes/sec
read qpsWrite QPS × 5 (read-to-write ratio) = 925,926 × 5 ≈ 4.63M reads/sec
peak read qps4.63M × 3 peak factor ≈ 13.9M reads/sec
💾
Storage Estimate
daily storage2B × 40 × 100 KB = 8 PB/day
retention period365 days
total storage8 PB/day × 365 days = 2,920 PB ≈ 2.9 EB
with replicationAssuming 3x replication: 2.9 EB × 3 = 8.7 EB
🖥️
Server Estimate
metadata serviceAssuming 10K metadata ops/sec per server, need ~278 metadata servers for peak writes
storage nodesAssuming 500 TB usable per storage node, need ~17,400 storage nodes for 8.7 EB
🏗️
Architecture Diagram
Architecture Diagram
          +----------------+
          |   Client SDK   |
          +-------+--------+
                  |
          +-------v--------+
          |  API Gateway   |
          +-------+--------+
                  |
          +-------v--------+
          | Metadata Store |
          +-------+--------+
                  |
          +-------v--------+
          |  Storage Nodes |
          +-------+--------+
                  |
          +-------v--------+
          | Replication   |
          +----------------+
🧩
Components
API Gateway
Handles client requests, authentication, authorization, and routes to metadata and storage layers.
Why thisNginx + Envoy proxy for scalable, low-latency routing with TLS termination and rate limiting.
Why notDirect client-to-storage bypasses auth and metadata validation, risking security and consistency.
Metadata Store
Stores object metadata, bucket info, and manages namespace consistency.
Why thisApache Cassandra chosen for linear scalability, tunable consistency, and high write throughput at 2.78M writes/sec peak.
Why notPostgreSQL rejected due to complex manual sharding and inability to handle 2.78M writes/sec with low latency.
Storage Nodes
Persist object data blobs on distributed storage with erasure coding for durability.
Why thisCeph Object Storage with erasure coding provides petabyte scale, 11 nines durability, and efficient storage utilization.
Why notHDFS rejected due to poor small file performance and lack of native object semantics.
Replication Service
Ensures data durability and availability by replicating data across storage nodes.
Why thisCustom replication layer integrated with Ceph to asynchronously replicate data with configurable replication factor.
Why notKafka rejected as replication backbone due to lack of direct integration with object storage and high storage overhead.
Multipart Upload Coordinator
Manages multipart upload sessions, assembling parts into final objects.
Why thisImplemented as a stateless microservice backed by metadata store to track upload parts and commit.
Why notStoring multipart state in storage nodes rejected due to complexity and risk of partial data loss.
🔄
Data Flow
1
Client SDK → API Gateway
Client sends PUT/GET/DELETE request with bucket and object key
API Gateway authenticates and authorizes request
2
API Gateway → Metadata Store
Query or update object metadata (existence, version, ACLs)
Strong consistency ensured via Cassandra lightweight transactions
3
API Gateway → Storage Nodes
Upload or download object data blobs
Data stored with erasure coding for durability
4
Storage Nodes → Replication Service
Replicate data asynchronously to other storage nodes
Ensures durability and availability
5
Multipart Upload Coordinator → Metadata Store
Track multipart upload parts and commit final object
Ensures atomic visibility of multipart uploads

Practice

(1/5)
1. A distributed database requires that every write is confirmed by a majority of replicas before acknowledging success to the client. Which replication strategy best fits this requirement?
easy
A. Asynchronous replication with eventual consistency
B. Single-leader replication without acknowledgments
C. Synchronous replication with quorum writes
D. Master-slave replication with delayed slave updates

Solution

  1. Step 1: Identify replication strategy matching majority confirmation

    Synchronous replication with quorum writes requires majority replicas to acknowledge before success.

  2. Step 2: Eliminate asynchronous and single-leader without ack

    Asynchronous replication and single-leader without acknowledgments do not guarantee majority confirmation before success.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Quorum writes ensure majority ack before success [check]
Hint: Quorum writes wait for majority acks before success [OK]
Common Mistakes:
2. In the task scheduler's execution flow, at Step 4 the Worker Node polls the distributed priority queue for the next task. What happens if the task's scheduled time has not yet arrived?
easy
A. The Worker Node marks the task as failed due to premature execution
B. The Worker Node re-queues the task with the same priority and waits
C. The Worker Node skips the task and polls for the next available task
D. The Worker Node immediately executes the task ignoring the schedule

Solution

  1. Step 1: Understand Worker Node behavior at Step 4

    The Worker Node must respect the scheduled execution time; executing early (A) violates correctness; re-queuing (B) risks duplicate tasks; skipping (C) ignores the task indefinitely.

  2. Step 2: Identify failure handling

    Marking the task as failed (D) triggers retry or alert mechanisms, ensuring no premature execution and visibility into scheduling issues.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Premature tasks are failed to prevent early execution [check]
Hint: Fail early if task scheduled time not reached [check]
Common Mistakes:
3. A system must handle 1 million writes per second with flexible schema and eventual consistency. Which database technology best fits this requirement?
easy
A. Use a NoSQL document store with horizontal sharding
B. Use a traditional relational database with vertical scaling
C. Use an in-memory cache like Redis without persistence
D. Use a graph database optimized for complex joins

Solution

  1. Step 1: Identify write throughput and schema flexibility needs

    Relational DBs struggle with 1M writes/sec and rigid schemas; graph DBs optimize for relationships, not write scale.

  2. Step 2: Evaluate NoSQL document store capabilities

    NoSQL document stores support horizontal sharding and flexible schemas, enabling high write throughput with eventual consistency.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    NoSQL document stores scale horizontally for high writes [check]
Hint: High writes + flexible schema -> NoSQL document store [check]
Common Mistakes:
4. To handle personalized ranking at scale, should the system use a centralized ranking service with a global user profile store or a distributed ranking service co-located with Trie shards holding local user profiles? Consider latency and data freshness constraints.
medium
A. Centralized ranking with global profiles for consistent personalization
B. Distributed ranking co-located with Trie shards to reduce latency despite eventual consistency
C. Centralized ranking but cache user profiles locally to reduce latency
D. Distributed ranking with no user profiles, relying only on global popularity

Solution

  1. Step 1: Analyze latency and freshness trade-offs

    Centralized ranking with global profiles (A) adds network latency; caching (C) adds complexity and stale data risk.

  2. Step 2: Evaluate distributed ranking with local profiles

    Distributed ranking co-located with Trie shards (B) reduces latency by local data access, tolerating eventual consistency for freshness.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Distributed ranking + local profiles -> low latency, eventual consistency [check]
Hint: Local profiles with distributed ranking reduce latency despite stale data [check]
Common Mistakes:
5. In an asynchronous messaging system, what exactly breaks if the message queue component is removed?
medium
A. Senders block until recipients acknowledge, causing high latency
B. Messages are duplicated due to lack of deduplication in the queue
C. Messages are delivered immediately but with no ordering guarantees
D. The system silently drops messages without notifying senders

Solution

  1. Step 1: Understand queue's role in message deduplication and buffering

    Queues often provide deduplication and buffering to ensure exactly-once or at-least-once delivery.

  2. Step 2: Consequence of removing the queue

    Without the queue, messages may be duplicated due to lack of deduplication and buffering mechanisms.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Removing queue causes message duplication due to missing deduplication [check]
Hint: No queue means no deduplication, causing duplicates [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard