Bird
Raised Fist0

Block Storage vs Object Storage vs File Storage - When to Use Which

HLDmediumInterview freq: medium
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Block Storage vs Object Storage vs File Storage - When to Use Which
45 min · senior
senior

You are designing the storage backend for a 50M DAU cloud platform that supports diverse workloads including VM disks, media archives, and collaborative document editing. The system must handle encrypted data at rest, provide offline access for some clients, and maintain P99 latency under 200ms for metadata operations.

Constraints
  • All data must be encrypted at rest with key rotation support
  • Offline access required for file storage use cases
  • P99 metadata operation latency < 200ms
  • Support for large unstructured data (multi-GB files)
  • High durability and availability with geo-replication
📋
Guided Steps
1/5
Step 1
Identify Storage Types and Their Core Characteristics
What are the fundamental differences between block, object, and file storage in terms of data access patterns and metadata management?
💡 Hint
Consider how data is addressed and accessed at the lowest level and how metadata is stored and queried.
Key Decision
Block storage (e.g., Amazon EBS) exposes raw byte-addressable volumes optimized for low-latency random I/O; object storage (e.g., Amazon S3) stores immutable objects with rich metadata and eventual consistency; file storage (e.g., NFS, Amazon EFS) provides hierarchical namespace with POSIX semantics and strong consistency for metadata.
Interviewer Expectation
Mid: Names block, object, and file storage with basic definitions. Senior: Explains access granularity, metadata handling, and consistency models distinguishing each storage type.
⚠️Common Wrong Answer
Treating all storage types as interchangeable without recognizing metadata and access pattern differences, leading to poor performance or incorrect consistency guarantees.
Step 2
Map Use Cases to Storage Types Based on Scale and Constraints
Given the scenario constraints, which storage type fits best for VM disks, media archives, and collaborative documents, and why?
💡 Hint
Match workload characteristics like IOPS, latency, metadata operations, and offline access to storage capabilities.
Key Decision
Use block storage (Amazon EBS) for VM disks requiring low-latency random I/O; object storage (Amazon S3) for media archives needing high durability and scalability; file storage (Amazon EFS) for collaborative documents requiring POSIX semantics and offline access.
Interviewer Expectation
Mid: Correctly matches workloads to storage types. Senior: Justifies choices with scale numbers (e.g., 50M DAU × metadata ops/sec), encryption impact, and offline access requirements.
⚠️Common Wrong Answer
Using object storage for VM disks causing unacceptable latency or block storage for media archives causing scalability bottlenecks.
Step 3
Design Encryption and Key Management Across Storage Types
How will you implement encryption at rest with key rotation support across block, object, and file storage?
💡 Hint
Consider native encryption features and integration with centralized key management services like AWS KMS.
Key Decision
Leverage native encryption: Amazon EBS with AWS KMS-managed keys for block storage; Amazon S3 server-side encryption with KMS for object storage; Amazon EFS encryption at rest integrated with KMS for file storage, enabling seamless key rotation and auditability.
Interviewer Expectation
Mid: Mentions encryption at rest and key rotation. Senior: Explains how each storage type’s native encryption integrates with KMS, impact on performance, and how key rotation is handled without downtime.
⚠️Common Wrong Answer
Implementing encryption at the application layer only, causing complexity and inconsistent security posture.
Step 4
Ensure Low Latency Metadata Operations and Offline Access
How will you architect metadata handling to meet P99 < 200ms latency and support offline access for file storage?
💡 Hint
Think about caching, metadata service design, and client-side synchronization mechanisms.
Key Decision
Use a distributed metadata service backed by a low-latency NoSQL store like Amazon DynamoDB with DAX caching for metadata operations; implement client-side metadata caching and conflict resolution protocols (e.g., CRDTs) for offline file access and sync.
Interviewer Expectation
Mid: Proposes caching metadata to reduce latency. Senior: Details DynamoDB with DAX for sub-200ms P99, explains offline sync challenges and CRDT-based conflict resolution for file storage.
⚠️Common Wrong Answer
Relying solely on synchronous metadata calls to backend storage causing high latency and poor offline experience.
Step 5
Design Geo-Replication and Durability Strategies
How will you ensure high durability and availability with geo-replication across storage types?
💡 Hint
Consider replication models, consistency trade-offs, and failure recovery mechanisms.
Key Decision
Use asynchronous cross-region replication for Amazon S3 with versioning for object durability; multi-AZ replication for Amazon EBS volumes with snapshot backups; Amazon EFS replication with strong consistency guarantees for file storage metadata and data.
Interviewer Expectation
Mid: Names replication strategies per storage type. Senior: Explains asynchronous vs synchronous replication trade-offs, impact on consistency and recovery time objectives (RTO).
⚠️Common Wrong Answer
Using synchronous replication for all storage types causing high write latency and reduced throughput.
🔌
API Design Challenge
Sketch 3 core API endpoints for your storage system that illustrate how clients interact with block, object, and file storage layers.
Hints
  • How does a client write and read data in each storage type?
  • How does the system handle metadata operations differently for file storage?
  • How does the client manage consistency and locking for collaborative file editing?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Correctly distinguishes block, object, and file storage with real technologies
  • Maps workloads to appropriate storage types with scale and latency justification
  • Designs encryption at rest with key rotation using AWS KMS integration
  • Proposes metadata architecture meeting P99 < 200ms latency
  • Includes geo-replication strategy balancing consistency and availability
Good to Have - 30%
  • Details client-side offline sync mechanisms for file storage
  • Explains trade-offs between synchronous and asynchronous replication
  • Includes API design endpoints illustrating client interactions
Bonus - 10%
  • Discusses failure modes and recovery sequences for each storage type
  • Considers cost implications of storage choices at 50M DAU scale
Anti-patterns to Avoid
pattern: Treating all storage types as interchangeable
consequence: Leads to poor performance or incorrect consistency guarantees due to ignoring metadata and access pattern differences.
pattern: Using object storage for VM disks
consequence: Causes unacceptable latency for low-latency random I/O workloads.
pattern: Using block storage for media archives
consequence: Results in scalability bottlenecks and inefficient handling of large unstructured data.
pattern: Implementing encryption only at the application layer
consequence: Causes complexity and inconsistent security posture across storage types.
pattern: Relying solely on synchronous metadata calls
consequence: Leads to high latency and poor offline user experience.
pattern: Using synchronous replication for all storage types
consequence: Causes high write latency and reduced throughput, impacting availability.
pattern: HTTP polling at 50M DAU
consequence: 50M reconnects/min just to check for messages - collapses under load.
✅
Model Answer
▼ Reveal

This design evaluates the appropriate use of block, object, and file storage for a large-scale cloud platform with 50M DAU, focusing on workload fit, encryption, metadata latency, offline access, and geo-replication.

Key Decisions
  • Choosing Amazon EBS for VM disks due to its byte-addressable volumes and low-latency random I/O support
  • Selecting Amazon S3 for media archives because of its immutable object storage, rich metadata, and eventual consistency model
  • Using Amazon EFS for collaborative documents to provide hierarchical namespace, POSIX compliance, and offline access capabilities
  • Implementing encryption at rest using native AWS KMS integration for each storage type to enable key rotation without downtime
  • Designing metadata handling with DynamoDB and DAX caching to meet P99 latency under 200ms
  • Employing client-side caching and CRDTs to handle offline access and conflict resolution in file storage
  • Applying asynchronous cross-region replication for S3 to balance durability and write latency, multi-AZ replication and snapshots for EBS to ensure availability, and strong consistency replication for EFS to maintain data integrity
  • Amazon EBS block storage for low-latency random I/O workloads like VM disks
  • Amazon S3 object storage for scalable, durable media archives with rich metadata
  • Amazon EFS file storage for POSIX-compliant collaborative documents with offline access
  • AWS KMS integration for encryption at rest with seamless key rotation across all storage types
  • Amazon DynamoDB with DAX caching for low-latency metadata operations under 200ms P99
  • Client-side metadata caching and CRDT-based conflict resolution for offline file sync
  • Asynchronous cross-region replication for S3 to balance durability and write latency
  • Multi-AZ replication and snapshot backups for EBS to ensure availability and durability
  • EFS replication with strong consistency for file metadata and data across regions

Practice

(1/5)
1. For storing user and item embeddings in a recommendation system, which storage choice best balances low-latency reads and frequent updates at scale?
medium
A. Use a document store like MongoDB with embedding documents
B. Use a distributed key-value store like Cassandra with tunable consistency
C. Use a file system with periodic batch loading into memory
D. Use a relational database with normalized tables for embeddings

Solution

  1. Step 1: Identify storage requirements

    Embedding storage requires low-latency reads and frequent writes at scale with high availability.

  2. Step 2: Evaluate storage options

    Relational DB (A) struggles with write throughput and scaling; Cassandra (B) offers tunable consistency but may add complexity; file system batch loading (D) introduces stale data and high latency. Document store (C) supports flexible schema and fast reads with embedded documents, balancing latency and update frequency.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Document store balances flexible schema and read/write latency [check]
Hint: Document store balances schema flexibility and latency [check]
Common Mistakes:
2. If the machine ID component is misconfigured and two machines generate Snowflake IDs with the same machine ID simultaneously, what is the most likely failure mode?
medium
A. The sequence number will automatically adjust to prevent collisions across machines.
B. Duplicate IDs will be generated within the same millisecond causing data corruption or overwrite.
C. IDs generated on both machines will be unique due to different timestamps, no collision occurs.
D. The system will detect the conflict and block one machine from generating IDs.

Solution

  1. Step 1: Understand machine ID role in uniqueness

    Machine ID differentiates ID generators on different machines; same machine ID risks collisions.

  2. Step 2: Analyze collision handling

    Different timestamps alone (A) do not guarantee uniqueness if clocks are synchronized; sequence number does not auto-adjust across machines (C is incorrect). System does not detect conflicts automatically (D).

  3. Step 3: Correct failure mode

    Duplicate IDs are generated if machine IDs collide, causing data corruption or overwrite.

  4. Final Answer:

    Option A -> Option A

  5. Quick Check:

    Same machine ID + same timestamp -> duplicate IDs and data corruption [check]
Hint: Same machine ID causes duplicate IDs, no auto correction [OK]
Common Mistakes:
3. Without a centralized coordination service (like ZooKeeper) in a horizontally scaled cluster, what exact failure occurs during leader election?
medium
A. All nodes elect themselves leader simultaneously, causing split-brain and inconsistent writes
B. The cluster halts because no node can become leader, blocking all writes
C. Leader election proceeds normally but with increased latency due to retries
D. Nodes elect multiple leaders but automatically resolve conflicts without data loss

Solution

  1. Step 1: Understand role of centralized coordination in leader election

    Coordination service ensures single leader election and consensus.

  2. Step 2: Analyze failure without coordination

    Without it, leader election retries occur causing latency but eventual leader election succeeds via timeouts.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Leader election delays but does not halt cluster without coordination [check]
Hint: No coordination causes leader election delays, not total failure [check]
Common Mistakes:
4. At 10x scale, which component in a distributed tracing system is most likely to become the bottleneck first, and why?
hard
A. Trace Collector, because it must synchronously process all incoming spans causing CPU saturation
B. Centralized Trace Storage, because indexing and querying massive trace volumes overwhelms storage I/O
C. Trace Context Propagation headers, because large headers increase network overhead and cause packet loss
D. Instrumentation Library, because injecting trace context adds significant latency per request

Solution

  1. Step 1: Analyze component scaling challenges

    Centralized Trace Storage must index and store exponentially more spans at 10x scale, stressing storage I/O and query performance.

  2. Step 2: Evaluate other components

    Trace Collector is often horizontally scalable and asynchronous, reducing CPU bottleneck risk. Instrumentation overhead is minimal per request. Trace headers increase size but not enough to cause packet loss at scale.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Storage I/O overwhelmed first at 10x scale [check]
Hint: Storage I/O chokes first at massive trace volume scale [OK]
Common Mistakes:
5. At 10x scale, which component in an async messaging chat system fails first and why?
hard
A. The database, because increased writes cause lock contention
B. The message queue brokers, due to partition leader overload
C. The chat servers, because WebSocket connections are stateful and cannot be resharded easily
D. The delivery workers, because retry queues grow exponentially

Solution

  1. Step 1: Analyze stateful components and scaling challenges

    Databases (The database, because increased writes cause lock contention) can be scaled with sharding and replicas. Message queue brokers (The message queue brokers, due to partition leader overload) scale horizontally with partitions. Delivery workers (The delivery workers, because retry queues grow exponentially) can be scaled with backpressure.

  2. Step 2: Identify chat servers as stateful WebSocket connection holders

    Chat servers maintain persistent WebSocket connections bound to server memory, making resharding or migration difficult at scale.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Stateful WebSocket servers -> first bottleneck at 10x scale [check]
Hint: Stateful WebSocket servers limit horizontal scaling first [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard