Bird
Raised Fist0

Storage Tiering - Hot, Warm, Cold, Archive and Cost Optimization

HLDmediumInterview freq: medium
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Storage Tiering - Hot, Warm, Cold, Archive and Cost Optimization
45 min · senior
senior

You are designing a storage tiering system for a startup with 50 million daily active users (DAU). The system must store user-generated content and metadata with different access patterns: hot data requires low latency access, warm data is accessed less frequently, cold data is rarely accessed but must be retrievable, and archive data is for compliance and long-term retention. The system must optimize storage costs by automatically migrating data between tiers based on access patterns and age. Data must be encrypted at rest and in transit. Offline delivery of data to clients is required, and the system must maintain a P99 read latency under 200ms for hot and warm tiers. Design the architecture, data flow, and APIs to support these requirements.

Constraints
  • Data encryption at rest and in transit is mandatory
  • Offline delivery support for clients with intermittent connectivity
  • P99 read latency under 200ms for hot and warm tiers
  • Automatic data migration between tiers based on access patterns and age
  • Support for 50 million DAU with peak QPS of 500K reads/sec and 100K writes/sec
  • Cost optimization by leveraging different storage technologies per tier
📋
Guided Steps
1/5
Step 1
Choosing Storage Technologies for Each Tier
Which storage technologies would you select for hot, warm, cold, and archive tiers, and why?
💡 Hint
Consider latency, throughput, cost per GB, and data access patterns for each tier.
Key Decision
Use Amazon DynamoDB with DAX for hot tier (sub-10ms latency, high throughput), Amazon S3 Standard-IA for warm tier (lower cost, ~100ms latency), Amazon S3 Glacier Instant Retrieval for cold tier (cost-effective, minutes retrieval), and Amazon S3 Glacier Deep Archive for archive tier (lowest cost, hours retrieval).
Interviewer Expectation
Mid: Names storage types like DynamoDB and S3 tiers. Senior: Explains DynamoDB DAX caching for hot tier latency, S3 IA cost-latency tradeoff, Glacier retrieval times, and why EBS or RDS are unsuitable at scale.
⚠️Common Wrong Answer
Using a single database like MySQL for all tiers - leads to high cost and poor latency for cold/archive data, and scalability bottlenecks.
Step 2
Data Migration and Tiering Logic
How would you design the automatic migration of data between tiers based on access patterns and data age?
💡 Hint
Think about event-driven workflows and metadata tracking for data lifecycle management.
Key Decision
Implement a Lambda-based lifecycle manager triggered by CloudWatch Events that evaluates access logs stored in Amazon Athena and moves data using S3 Lifecycle policies and DynamoDB TTL for hot tier eviction.
Interviewer Expectation
Mid: Mentions scheduled jobs or cron for migration. Senior: Details event-driven Lambda with Athena queries for access pattern analysis, DynamoDB TTL for hot tier eviction, and S3 Lifecycle policies for cost-effective tier transitions.
⚠️Common Wrong Answer
Manual migration or fixed time-based migration without access pattern analysis - causes unnecessary cost or latency by moving hot data prematurely.
Step 3
Ensuring Encryption and Data Security
How will you ensure encryption at rest and in transit across all tiers and during migration?
💡 Hint
Consider native cloud encryption features and key management services.
Key Decision
Use AWS KMS-managed keys for server-side encryption (SSE-KMS) on S3 and DynamoDB, enforce TLS 1.2+ for all client-server and inter-service communication, and encrypt data client-side before upload for end-to-end security.
Interviewer Expectation
Mid: Mentions TLS and server-side encryption. Senior: Explains SSE-KMS key rotation, client-side envelope encryption for zero-trust, and how encryption integrates with migration workflows without latency impact.
⚠️Common Wrong Answer
Relying only on server-side encryption without TLS or client-side encryption - exposes data during transit and increases risk.
Step 4
Designing APIs for Client Interaction
What are the 3 core API endpoints you would design to support client upload, retrieval, and acknowledgment with tier awareness and offline support?
💡 Hint
Consider how clients upload encrypted data, fetch data with tier fallback, and notify the system about data receipt or migration needs.
Key Decision
Design UploadData (POST /data/upload) with client-side encryption and metadata tagging, FetchData (GET /data/{id}) with tier-aware routing and offline cache validation, and AcknowledgeData (POST /data/{id}/ack) to confirm receipt and trigger migration if needed.
Interviewer Expectation
Mid: Lists basic CRUD endpoints. Senior: Details encryption metadata in UploadData, tier-aware routing logic in FetchData with cache validation headers, and AcknowledgeData triggering lifecycle events.
⚠️Common Wrong Answer
Simple CRUD without tier awareness or offline support - leads to inefficient data access and poor user experience.
Step 5
Meeting P99 Latency and Scalability Requirements
How will you architect the system to maintain P99 read latency under 200ms for hot and warm tiers at 50M DAU with peak 500K reads/sec?
💡 Hint
Consider caching, partitioning, and load balancing strategies.
Key Decision
Use DynamoDB with DAX caching for hot tier to achieve sub-10ms reads, S3 Select for warm tier to reduce data transfer latency, and API Gateway with autoscaling Lambda functions for load balancing and throttling.
Interviewer Expectation
Mid: Mentions caching and autoscaling. Senior: Explains DAX's in-memory caching reducing DynamoDB read latency, S3 Select to avoid full object retrieval, and API Gateway throttling to protect backend.
⚠️Common Wrong Answer
Direct S3 GET for warm tier without Select or caching - causes high latency and throttling under load.
🔌
API Design Challenge
Sketch 3 core API endpoints to support client interaction with the storage tiering system.
Hints
  • How does the client upload and encrypt data to the system?
  • How does the client retrieve data with tier awareness and offline support?
  • How does the client acknowledge data receipt or request data migration?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Correct selection of storage technologies per tier with scale and latency justification
  • Event-driven data migration using AWS Lambda and lifecycle policies
  • Encryption at rest and in transit with KMS and TLS
  • Tier-aware API design supporting offline delivery and acknowledgments
  • Caching and query optimization to meet P99 latency
Good to Have - 30%
    Bonus - 10%
      Anti-patterns to Avoid
      ✅
      Model Answer
      ▼ Reveal

      This design uses a multi-tier storage architecture optimized for cost and latency to handle 50M DAU with 500K reads/sec peak. Hot data is stored in DynamoDB with DAX caching for sub-10ms latency. Warm data uses Amazon S3 Standard-IA with S3 Select to reduce retrieval latency. Cold and archive data use Glacier Instant Retrieval and Deep Archive respectively for cost savings. Data migration is automated via AWS Lambda triggered by CloudWatch Events analyzing access logs in Athena. Encryption is enforced at rest with SSE-KMS and in transit with TLS 1.2+. Client-side envelope encryption ensures end-to-end security. APIs support encrypted uploads, tier-aware retrieval with offline cache validation, and acknowledgments triggering lifecycle events. The system uses API Gateway and autoscaling Lambda for load balancing and throttling to maintain P99 latency under 200ms.

      Key Decisions
      • Use DynamoDB with DAX caching to achieve sub-10ms P99 read latency for hot tier
      • Leverage S3 Standard-IA and Glacier tiers to balance cost and retrieval latency for warm, cold, and archive data
      • Implement Lambda-based lifecycle manager triggered by CloudWatch Events to automate data migration based on access patterns and age
      • Encrypt all data at rest using SSE-KMS and enforce TLS 1.2+ for all data in transit
      • Apply client-side envelope encryption to protect data end-to-end
      • Design APIs with tier-aware routing and offline cache validation to support intermittent client connectivity
      • Use API Gateway throttling and autoscaling Lambda to handle peak loads and maintain latency SLAs
      • Utilize S3 Select to reduce data transfer and improve warm tier read performance
      • Employ DynamoDB TTL to automatically evict stale hot data to warm tier, reducing manual overhead
      • Amazon DynamoDB with DAX for hot tier: sub-10ms latency, high throughput, and seamless scaling
      • Amazon S3 Standard-IA and Glacier tiers for warm, cold, and archive storage balancing cost and retrieval latency
      • AWS Lambda with CloudWatch Events and Athena for event-driven data lifecycle management and migration
      • AWS KMS for SSE-KMS encryption and TLS 1.2+ enforced for all data in transit
      • Client-side envelope encryption to ensure end-to-end data security beyond server-side encryption
      • API Gateway and autoscaling AWS Lambda for tier-aware API endpoints supporting offline delivery and acknowledgments
      • S3 Select to optimize warm tier data retrieval by querying partial data without full object download
      • DynamoDB TTL for automatic eviction of hot data to warm tier, reducing manual intervention and cost

      Practice

      (1/5)
      1. A high-throughput messaging system must guarantee no data loss even if consumers are slow or temporarily offline. Which component best ensures durability without blocking producers?
      easy
      A. Use a distributed log like Kafka with replication and disk persistence
      B. Use a cache like Redis with asynchronous persistence
      C. Use a relational database with immediate commit on every message
      D. Use an in-memory queue with periodic snapshots to disk

      Solution

      1. Step 1: Identify durability guarantees needed

        In-memory queues and caches (A, D) risk data loss on crashes; relational DB (C) can block producers due to synchronous commits.

      2. Step 2: Evaluate Kafka's role

        Kafka provides durable, replicated logs that persist to disk and decouple producers from slow consumers, ensuring no data loss.

      3. Final Answer:

        Option A -> Option A

      4. Quick Check:

        Kafka replication and disk persistence ensure durability without blocking [check]
      Hint: Durability needs disk + replication, not just memory or sync DB [check]
      Common Mistakes:
      2. Without the Expiration Worker component, what exactly breaks in the Pastebin system?
      medium
      A. Expired snippets remain stored indefinitely, causing storage bloat and stale data served to users
      B. Users cannot create new snippets because the storage is locked waiting for expiration cleanup
      C. Snippet retrieval latency increases drastically due to missing expiration indexes
      D. Analytics data becomes inaccurate because expired snippets are still counted as active

      Solution

      1. Step 1: Identify Expiration Worker's role

        It removes expired snippets to free storage and prevent stale data serving.

      2. Step 2: Trace failure impact without Expiration Worker

        Expired snippets accumulate indefinitely, increasing storage and possibly serving stale content.

      3. Final Answer:

        Option A -> Option A

      4. Quick Check:

        Without expiration cleanup, storage bloat and stale data occur [check]
      Hint: No expiration worker -> storage bloat and stale snippets served [check]
      Common Mistakes:
      3. Without a clear action item tracking system after a post-mortem, what specific failure occurs in the incident resolution process?
      medium
      A. The root cause analysis becomes inaccurate due to missing data
      B. Action items are forgotten or delayed, causing repeated incidents
      C. Incident detection slows down because alerts are not configured
      D. The blame culture intensifies as no one is held accountable

      Solution

      1. Step 1: Identify role of action item tracking

        Tracking ensures follow-through on improvements, preventing recurrence and reducing blame.

      2. Step 2: Trace failure without tracking

        Without tracking, accountability gaps appear, blame culture can worsen as no clear ownership exists.

      3. Final Answer:

        Option D -> Option D

      4. Quick Check:

        No tracking -> blame culture intensifies due to lack of ownership [check]
      Hint: No tracking -> blame culture grows, fixes stall [check]
      Common Mistakes:
      4. At 10x scale, which component in the Google Search architecture is most likely to become the first bottleneck, and why?
      hard
      A. The distributed file system storing raw crawled pages, because storage bandwidth saturates before compute
      B. The crawler's URL Frontier, because it must maintain global URL ordering and politeness state, which is hard to shard
      C. The query serving layer, because caching effectiveness drops and latency-sensitive lookups increase exponentially
      D. The PageRank computation cluster, because the graph size grows superlinearly, increasing memory and CPU demands drastically

      Solution

      1. Step 1: Analyze scaling challenges per component

        The PageRank cluster faces superlinear growth in graph edges, causing memory and CPU bottlenecks.

      2. Step 2: Compare other components

        URL Frontier can shard with heuristics; query serving scales with caching; storage bandwidth scales better than PageRank compute.

      3. Final Answer:

        Option D -> Option D

      4. Quick Check:

        PageRank graph growth causes first bottleneck at 10x scale [check]
      Hint: PageRank graph grows superlinearly -> first bottleneck [check]
      Common Mistakes:
      5. At 10x traffic scale, which component is most likely to fail first in a system without rate limiting, and why?
      hard
      A. The backend cache, because its memory fills up causing evictions and cache misses
      B. The database, due to increased write load causing slow queries and deadlocks
      C. The network load balancer, as it cannot handle the increased number of TCP connections
      D. The API servers, because they maintain stateful connections that cannot be easily sharded

      Solution

      1. Step 1: Analyze component scalability at 10x traffic

        Databases (A) and caches (D) can be scaled horizontally or sharded; load balancers (C) can be scaled or replaced with L4/L7 solutions.

      2. Step 2: Identify API servers as bottleneck

        API servers maintaining stateful connections (e.g., WebSockets) are hard to shard or move; connection limits cause saturation and failure first.

      3. Final Answer:

        Option D -> Option D

      4. Quick Check:

        Stateful API servers hit connection limits first at scale [check]
      Hint: Stateful API servers fail first due to connection limits [check]
      Common Mistakes:
      Recommended

      Ready to prove it?

      Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

      10 questions~5 mineasy hard