Bird
Raised Fist0

Blob Store Design - S3-Like Object Storage Architecture End to End

HLDmediumInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Blob Store Design - S3-Like Object Storage Architecture End to End
45 min · senior
senior

Design a highly available, durable, and scalable object storage system similar to Amazon S3 for a startup with 50 million daily active users. The system must support storing and retrieving large blobs (up to 5TB), provide strong encryption at rest and in transit, and guarantee P99 latency under 200ms for GET and PUT operations. The system should support offline delivery scenarios where clients can sync objects asynchronously. Data durability must be ensured with multi-region replication. The system should also support versioning and lifecycle policies for objects.

Constraints
  • Support up to 50 million daily active users with peak 100K PUTs and 1M GETs per second
  • P99 latency for GET and PUT operations must be under 200ms
  • End-to-end encryption: data encrypted at rest (AES-256) and in transit (TLS 1.3)
  • Multi-region replication for durability and disaster recovery
  • Support offline delivery with client sync and conflict resolution
  • Object versioning and lifecycle management (automatic archival and deletion)
📋
Guided Steps
1/6
Step 1
Storage Backend Choice and Data Partitioning
Which storage technology and data partitioning scheme will you use to store blobs reliably and scale to 50M DAU?
💡 Hint
Consider object size variability, write/read throughput, and durability requirements.
Key Decision
Use Amazon S3-compatible object storage backed by distributed erasure-coded storage like Ceph or MinIO with object keys partitioned by a hash of bucket+object name to distribute load evenly.
Interviewer Expectation
Mid: Names S3-compatible storage or Ceph. Senior: Explains erasure coding for durability, hash partitioning for even load, and why block storage or traditional RDBMS fail at this scale.
⚠️Common Wrong Answer
Using a relational database like MySQL to store blobs leads to poor write throughput and storage inefficiency at petabyte scale.
Step 2
Metadata Management and Indexing
How will you manage and store metadata such as object versions, ACLs, and lifecycle policies to support fast lookups and updates?
💡 Hint
Think about a scalable key-value store with strong consistency and efficient range queries.
Key Decision
Use Apache Cassandra with partition key as bucket name and clustering key as object name and version to enable efficient range scans for versioning and lifecycle operations.
Interviewer Expectation
Mid: Suggests a NoSQL store like Cassandra. Senior: Explains partition and clustering keys for efficient version range queries and why DynamoDB or MongoDB may not meet latency or consistency needs.
⚠️Common Wrong Answer
Using a relational database for metadata causes hotspots and poor scalability under high concurrency.
Step 3
Encryption and Security Model
How will you implement encryption at rest and in transit while maintaining performance and key management?
💡 Hint
Consider envelope encryption and integration with a centralized key management system.
Key Decision
Implement envelope encryption with AES-256 for data at rest, TLS 1.3 for data in transit, and use HashiCorp Vault or AWS KMS for centralized key management and rotation.
Interviewer Expectation
Mid: Mentions TLS and encryption at rest. Senior: Details envelope encryption benefits, key rotation policies, and how to minimize performance impact by encrypting only data keys.
⚠️Common Wrong Answer
Encrypting entire blobs on the client side without centralized key management complicates key rotation and auditing.
Step 4
Replication and Durability Strategy
How will you ensure multi-region replication for durability and disaster recovery without compromising latency SLAs?
💡 Hint
Think about asynchronous replication with conflict resolution and quorum reads/writes.
Key Decision
Use asynchronous multi-region replication with conflict-free version vectors and quorum reads/writes in the primary region to meet P99 latency under 200ms while ensuring eventual consistency globally.
Interviewer Expectation
Mid: Suggests multi-region replication. Senior: Explains asynchronous replication trade-offs, conflict resolution via version vectors, and why synchronous cross-region replication breaks latency SLAs.
⚠️Common Wrong Answer
Synchronous cross-region replication increases PUT latency beyond 200ms P99, violating SLA.
Step 5
Offline Delivery and Client Sync
How will clients synchronize objects when offline and resolve conflicts upon reconnection?
💡 Hint
Consider versioning, conflict detection, and eventual consistency models.
Key Decision
Support object versioning with vector clocks stored in metadata; clients sync by exchanging version vectors and resolving conflicts via last-writer-wins or application-defined merge logic.
Interviewer Expectation
Mid: Mentions versioning and sync. Senior: Explains vector clocks for conflict detection, client-side merge strategies, and how this supports offline-first use cases.
⚠️Common Wrong Answer
No versioning or conflict detection leads to data loss or inconsistent client states after offline edits.
Step 6
API Design for Core Operations
Sketch 3 core RESTful endpoints for uploading, downloading, and syncing objects, including authentication and versioning support.
💡 Hint
Consider multipart upload for large objects, conditional GETs for caching, and sync endpoints returning version metadata.
Key Decision
Define PUT /bucket/object for uploads supporting multipart upload with pre-signed URLs; GET /bucket/object with If-None-Match header for caching; and GET /bucket/sync?since=version for incremental sync with version metadata.
Interviewer Expectation
Mid: Lists basic upload/download endpoints. Senior: Details multipart upload, pre-signed URLs for secure direct upload, conditional GETs for latency optimization, and sync endpoints with version vectors.
⚠️Common Wrong Answer
Single monolithic upload endpoint without multipart support causes failures on large objects and poor client experience.
🔌
API Design Challenge
Sketch 3 core RESTful endpoints for the blob store API that cover uploading, downloading, and syncing objects.
Hints
  • How does the client upload large objects efficiently and securely?
  • How does the client retrieve objects with low latency and consistency?
  • How does the client synchronize changes and handle version conflicts?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Use of erasure-coded distributed object storage like Ceph or MinIO for scalable blob storage
  • Metadata stored in a scalable NoSQL store like Cassandra with appropriate partition and clustering keys
Good to Have - 30%
    Bonus - 10%
      Anti-patterns to Avoid
      ✅
      Model Answer
      ▼ Reveal

      Design an S3-like object storage system supporting 50M DAU with strong durability, low latency, and security. Use erasure-coded distributed storage for blobs, Cassandra for metadata, envelope encryption with centralized KMS, asynchronous multi-region replication, and vector clocks for offline sync. API supports multipart uploads, conditional GETs, and sync endpoints.

      Key Decisions
      • Use Ceph or MinIO for scalable, durable blob storage with erasure coding to reduce storage overhead and improve fault tolerance.
      • Store metadata in Apache Cassandra using bucket as partition key and object name plus version as clustering keys for efficient range queries and strong consistency.
      • Implement envelope encryption with AES-256 for data at rest and TLS 1.3 for data in transit; manage keys centrally with Vault or AWS KMS to enable rotation and auditing.
      • Adopt asynchronous multi-region replication with conflict-free version vectors to ensure durability and disaster recovery without violating 200ms P99 latency SLA.
      • Support offline client sync using vector clocks stored in metadata; clients exchange version vectors and resolve conflicts via last-writer-wins or custom merge logic.
      • Design RESTful API with multipart upload using pre-signed URLs for secure, efficient large object uploads; conditional GETs with ETag and If-None-Match headers to optimize bandwidth and latency; and sync endpoints returning version metadata for incremental updates.
      • Ceph or MinIO for distributed erasure-coded object storage to ensure durability and scalability
      • Apache Cassandra with bucket+object+version keys for metadata storage enabling efficient version range queries
      • Envelope encryption with AES-256 and TLS 1.3 combined with HashiCorp Vault or AWS KMS for secure key management
      • Asynchronous multi-region replication with conflict-free version vectors to balance durability and latency
      • Vector clocks for object versioning enabling offline client sync and conflict resolution
      • Multipart upload with pre-signed URLs for efficient and secure large object uploads
      • Conditional GETs with ETag and If-None-Match headers to optimize read latency and bandwidth
      • Lifecycle policies implemented via metadata triggers in Cassandra to automate archival and deletion

      Practice

      (1/5)
      1. In the cache warming system, at step 3 the Cache Loader fetches data from the database. What happens if the database response is delayed beyond the configured timeout?
      easy
      A. Cache Loader retries indefinitely until data is fetched, blocking subsequent steps
      B. Cache Loader falls back to a secondary cache layer to fetch the data synchronously
      C. Cache Loader marks the key as stale and skips warming that key for this cycle
      D. Cache Loader writes a null value to cache to prevent cache miss storms

      Solution

      1. Step 1: Trace Cache Loader timeout behavior

        On timeout, Cache Loader skips warming that key to avoid blocking the entire warming process.

      2. Step 2: Eliminate retry and fallback options

        Indefinite retries block warming; fallback cache is not part of warming design; null writes risk stale data.

      3. Final Answer:

        Option C -> Option C

      4. Quick Check:

        Timeout causes skipping key warming to maintain overall progress [check]
      Hint: Timeout skips key warming to avoid blocking cache load [check]
      Common Mistakes:
      2. A system needs to serve AI code completions with sub-100ms latency while ensuring up-to-date code context. Which component is best suited to provide fast, low-latency retrieval of relevant code snippets?
      easy
      A. A relational database with complex SQL queries for code indexing
      B. An in-memory vector search engine with approximate nearest neighbor (ANN) indexing
      C. A distributed file system storing raw code files for batch processing
      D. A message queue to buffer code snippet requests before processing

      Solution

      1. Step 1: Identify low-latency retrieval need

        Relational DBs (A) have high query latency for vector similarity; distributed file systems (C) are for storage, not fast retrieval; message queues (D) buffer requests but don't serve retrieval.

      2. Step 2: Recognize vector search role

        ANN-based in-memory vector search engines (B) provide fast approximate nearest neighbor lookups, ideal for sub-100ms latency code snippet retrieval.

      3. Final Answer:

        Option B -> Option B

      4. Quick Check:

        ANN vector search enables fast similarity queries [check]
      Hint: Vector search engines excel at fast similarity retrieval [check]
      Common Mistakes:
      3. In the Spotify offline download flow, at step 4 the Download Manager checks the local cache before fetching audio segments. What happens if the segment is already cached?
      easy
      A. The Download Manager deletes the cached segment and fetches anew
      B. The Download Manager re-downloads the segment to ensure freshness
      C. The Download Manager skips network fetch and marks segment as ready
      D. The Download Manager queues the segment for background validation

      Solution

      1. Step 1: Trace Download Manager behavior at Step 4

        It checks local cache to avoid redundant downloads and save bandwidth.

      2. Step 2: Determine action on cache hit

        On cache hit, it skips network fetch and marks segment ready for playback. Re-downloading (A) wastes resources; deleting (C) is unnecessary; background validation (D) is not immediate playback behavior.

      3. Final Answer:

        Option C -> Option C

      4. Quick Check:

        Cache hit -> skip fetch, mark ready [check]
      Hint: Cache hit means skip fetch and mark ready [check]
      Common Mistakes:
      4. In a pub-sub system, at step 4 the Delivery Worker reads a message from the topic partition and checks the subscriber's offset. What happens if the subscriber's offset is behind the latest committed offset?
      easy
      A. The Delivery Worker deletes the message from the topic partition
      B. The Delivery Worker skips the message and moves to the next one
      C. The Delivery Worker delivers the message to the subscriber and updates the offset
      D. The Delivery Worker blocks until the subscriber catches up

      Solution

      1. Step 1: Understand subscriber offset semantics

        The subscriber offset tracks the last consumed message. If behind, the Delivery Worker must deliver pending messages to catch up.

      2. Step 2: Delivery Worker behavior

        It delivers the message at the subscriber's offset and then updates the offset to the next message. Skipping or deleting messages (Options A and C) breaks delivery guarantees. Blocking (The Delivery Worker blocks until the subscriber catches up) is unnecessary and stalls the system.

      3. Final Answer:

        Option C -> Option C

      4. Quick Check:

        Delivery Worker delivers and advances subscriber offset [check]
      Hint: Subscriber offset behind -> deliver message and update offset [check]
      Common Mistakes:
      5. If the gRPC server's message queue is removed, what exactly breaks in the message delivery flow?
      medium
      A. Clients receive duplicate messages due to lack of deduplication
      B. The server blocks sending until the client acknowledges, causing backpressure
      C. Messages are lost if the client is slow or disconnected, with no retry mechanism
      D. The server switches to synchronous unary calls, increasing latency

      Solution

      1. Step 1: Role of message queue in gRPC server

        The queue buffers messages for slow or disconnected clients, enabling retries and flow control.

      2. Step 2: Effect of removing the queue

        Without the queue, messages sent to slow/disconnected clients are dropped immediately, causing message loss without retries.

      3. Final Answer:

        Option C -> Option C

      4. Quick Check:

        No queue -> message loss on slow/disconnected clients [check]
      Hint: No queue -> messages lost if client slow or disconnected [check]
      Common Mistakes:
      Recommended

      Ready to prove it?

      Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

      10 questions~5 mineasy hard