Bird
Raised Fist0

Key-Value Stores - Redis vs DynamoDB, Use Cases, Trade-offs

HLDmediumInterview freq: common
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Key-Value Stores - Redis vs DynamoDB, Use Cases, Trade-offs
45 min · senior
senior

You are building the backend for a 50M DAU social media startup that requires a key-value store for user session management, user profile caching, and global configuration storage. The system must support P99 latency under 200ms, ensure data encryption at rest and in transit, and handle offline client data sync with eventual consistency. The system must also support high write throughput during peak hours and provide strong availability guarantees.

Constraints
  • P99 latency under 200ms for read and write operations
  • Encryption at rest and in transit is mandatory
  • Support offline client data sync with eventual consistency
  • Handle 50M DAU with peak write throughput of 1M writes/sec
  • Strong availability with minimal downtime (99.99%)
📋
Guided Steps
1/5
Step 1
Choosing the Key-Value Store Technology
Given the scale and constraints, which key-value store technology would you choose for session management and why?
💡 Hint
Consider in-memory speed, persistence, and latency guarantees for session data.
Key Decision
Use Redis Enterprise with AOF persistence and TLS encryption for session management - Redis provides sub-millisecond latency, supports encryption in transit and at rest, and persistence via Append Only File (AOF) for durability.
Interviewer Expectation
Mid: Names Redis for caching. Senior: Explains Redis Enterprise AOF persistence, TLS encryption, and why DynamoDB's higher latency is unsuitable for session data.
⚠️Common Wrong Answer
Choosing DynamoDB for session management - DynamoDB's typical P99 latency (~10-20ms) is too high for session reads/writes requiring sub-5ms latency, causing poor user experience.
Step 2
Selecting the Store for User Profile Caching
Which key-value store would you use for user profile caching that requires durability and global availability?
💡 Hint
Think about multi-region replication, durability, and eventual consistency trade-offs.
Key Decision
Use Amazon DynamoDB Global Tables for user profile caching - DynamoDB offers multi-region replication with strong availability, encryption at rest, and handles 1M+ writes/sec with P99 latency under 20ms.
Interviewer Expectation
Mid: Names DynamoDB for durability. Senior: Explains Global Tables for multi-region availability, encryption at rest, and why Redis persistence is insufficient for global durable cache.
⚠️Common Wrong Answer
Using Redis alone for profile caching - Redis persistence modes (RDB/AOF) do not guarantee multi-region durability and availability, risking data loss during failover.
Step 3
Designing Offline Client Sync with Eventual Consistency
How would you design offline client sync using these key-value stores to ensure eventual consistency and conflict resolution?
💡 Hint
Consider write conflict detection, vector clocks or timestamps, and idempotent writes.
Key Decision
Use DynamoDB with conditional writes and version attributes for conflict detection, combined with client-side vector clocks to resolve conflicts during offline sync; Redis is used only for ephemeral session data, not offline sync state.
Interviewer Expectation
Mid: Mentions DynamoDB conditional writes. Senior: Details version attributes, vector clocks for conflict resolution, and why Redis lacks built-in conflict resolution for offline sync.
⚠️Common Wrong Answer
Using Redis for offline sync state - Redis lacks native conflict resolution and durability guarantees needed for offline sync, leading to data loss or stale data.
Step 4
Ensuring Encryption and Security Compliance
How do you ensure encryption at rest and in transit for both Redis and DynamoDB in this system?
💡 Hint
Look into TLS, encryption keys management, and compliance certifications.
Key Decision
Enable TLS for all Redis Enterprise client connections and use Redis Enterprise's encryption at rest with customer-managed keys; for DynamoDB, enable AWS KMS encryption at rest and enforce TLS for all API calls.
Interviewer Expectation
Mid: Names TLS and encryption at rest. Senior: Explains Redis Enterprise encryption with CMKs, DynamoDB KMS integration, and how these meet compliance requirements.
⚠️Common Wrong Answer
Ignoring encryption at rest or using unencrypted Redis - exposes sensitive data to risk and violates compliance, risking data breaches.
Step 5
API Design for Core Operations
Sketch three core API endpoints to support session creation/update, profile caching, and offline sync acknowledgement.
💡 Hint
Consider RESTful design, idempotency, and conflict resolution parameters.
Key Decision
Define POST /session to create/update sessions with TTL, GET /profile/{userId} to fetch cached profiles with DynamoDB read, and POST /sync to accept offline changes with versioning and conflict resolution metadata.
Interviewer Expectation
Mid: Provides basic CRUD endpoints. Senior: Designs idempotent APIs with versioning, conflict metadata, and clear separation of ephemeral vs durable data paths.
⚠️Common Wrong Answer
Designing synchronous blocking APIs without versioning - leads to poor offline sync support and inconsistent data.
🔌
API Design Challenge
Sketch 3 core API endpoints for the key-value store system that support session management, profile caching, and offline sync.
Hints
  • How does the client create or update a session or profile?
  • How does the client sync offline changes and resolve conflicts?
  • How does the system acknowledge successful writes and handle retries?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Correct choice of Redis Enterprise for session management with latency justification
  • Use of DynamoDB Global Tables for durable profile caching with multi-region support
  • Design for offline sync using DynamoDB conditional writes and conflict resolution
  • Encryption at rest and in transit implementation details for both stores
  • API design covering session, profile, and offline sync with idempotency and versioning
Good to Have - 30%
  • Explanation of Redis persistence modes and why AOF is chosen
  • Discussion of DynamoDB throughput scaling and partition keys
  • Details on vector clocks or version attributes for conflict resolution
  • Consideration of failure modes and fallback strategies
  • Trade-offs between latency and durability for each store
Bonus - 10%
  • Discussion of Redis Enterprise clustering and failover mechanisms
  • Use of AWS KMS customer-managed keys for compliance
  • Handling of offline sync retries and exponential backoff
  • Monitoring and alerting strategies for latency and availability
Anti-patterns to Avoid
Using Redis without persistence for critical data - leads to data loss on failover and impacts availability during node failures
Using DynamoDB for ultra-low latency session data - DynamoDB's higher P99 latency (~10-20ms) causes poor user experience for session reads/writes requiring sub-5ms latency
Ignoring encryption at rest or using unencrypted Redis - exposes sensitive data to risk and violates compliance, risking data breaches
Using Redis for offline sync state - Redis lacks native conflict resolution and durability guarantees needed for offline sync, leading to data loss or stale data
HTTP polling at 50M DAU: 50M reconnects/min just to check for messages - collapses under load and increases latency
Designing synchronous blocking APIs without versioning - leads to poor offline sync support and inconsistent data
✅
Model Answer
▼ Reveal

This design uses Redis Enterprise for ultra-low latency session management with AOF persistence and TLS encryption, and DynamoDB Global Tables for durable, multi-region user profile caching and offline sync with conditional writes and versioning. The system ensures encryption at rest and in transit using Redis Enterprise CMKs and AWS KMS, supports 1M writes/sec peak throughput, and provides strong availability with 99.99% uptime. The API design includes idempotent endpoints with versioning and conflict metadata to support offline sync and consistency.

Key Decisions
  • Use Redis Enterprise with AOF persistence and TLS encryption for sub-millisecond latency session data
  • Use DynamoDB Global Tables for durable, multi-region user profile caching with P99 latency under 20ms
  • Implement offline sync using DynamoDB conditional writes and version attributes with client-side conflict resolution
  • Enforce TLS for all client-server communication and enable encryption at rest via Redis CMKs and DynamoDB KMS
  • Design idempotent RESTful APIs with versioning and conflict metadata to handle offline sync and retries
  • Redis Enterprise with AOF persistence and TLS for low-latency session management
  • Amazon DynamoDB Global Tables for durable, multi-region user profile caching
  • DynamoDB conditional writes and version attributes for offline sync conflict detection
  • Redis used only for ephemeral data, avoiding durability and conflict resolution issues
  • TLS enforced for all client-server communication for encryption in transit
  • Encryption at rest via Redis Enterprise CMKs and DynamoDB KMS integration
  • API design with idempotent POST /session, GET /profile/{userId}, and POST /sync endpoints
  • Use of vector clocks or versioning metadata to resolve offline sync conflicts

Practice

(1/5)
1. In the blob store request flow, at Step 4 the Metadata Service validates the object's metadata and checks for existing versions. What happens if the metadata validation fails at this step?
easy
A. The request is rejected and the client receives an error before any data is stored
B. The system retries metadata validation asynchronously and proceeds with write
C. The object is immediately written to the storage nodes without metadata update
D. The object is stored but marked as invalid for later cleanup

Solution

  1. Step 1: Trace Metadata Service validation at Step 4

    The Metadata Service performs synchronous validation; failure causes immediate rejection to prevent invalid data.

  2. Step 2: Eliminate other options

    Writing without metadata update breaks consistency, async retry delays error feedback, and storing invalid data risks corruption.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Metadata validation failure causes immediate client error [check]
Hint: Metadata validation failure rejects request immediately [check]
Common Mistakes:
2. For a social network with users having highly variable follower counts (from tens to millions), which fan-out architecture best balances write latency and read latency?
medium
A. Pure pull fan-out: followers fetch posts on demand from a global store
B. Pure push fan-out: push every post to all followers' feeds synchronously
C. Hybrid fan-out: push to high-follower-count users' feeds, pull for low-follower-count users
D. Push fan-out only for active users, pull for inactive users

Solution

  1. Step 1: Analyze pure push fan-out (A)

    Push to all followers synchronously causes huge write amplification for celebrities, increasing write latency and storage.

  2. Step 2: Analyze pure pull fan-out (C)

    Pull reduces write cost but increases read latency and complexity for users with many followers.

  3. Step 3: Evaluate hybrid fan-out (B)

    Hybrid pushes to users with many followers (to reduce read latency) and pulls for users with few followers (to reduce write amplification), balancing both latencies.

  4. Step 4: Evaluate push only for active users (D)

    Activity-based push doesn't address follower count skew effectively.

  5. Final Answer:

    Option C -> Option C

  6. Quick Check:

    Hybrid fan-out balances write and read latency by follower count [check]
Hint: Hybrid fan-out balances write/read latency by follower count [check]
Common Mistakes:
3. If the message queue responsible for decoupling job application submissions from downstream processing crashes, what is the immediate failure observed in the system?
medium
A. Applications are lost silently with no retry or alert
B. The system automatically switches to a backup queue with no impact
C. Application submission API blocks until the queue recovers, causing client timeouts
D. Applications are accepted but downstream components never receive them, causing processing delays

Solution

  1. Step 1: Understand queue role in decoupling

    The queue buffers submissions so downstream processing is asynchronous.

  2. Step 2: Trace failure impact

    If queue crashes, submissions accepted by API but not delivered downstream, causing processing delays (C). Silent loss (A) is prevented by persistence; blocking API (B) contradicts decoupling; automatic failover (D) is ideal but not guaranteed.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Queue crash -> accepted but undelivered applications -> delayed processing [check]
Hint: Queue crash delays downstream, submissions accepted but unprocessed [check]
Common Mistakes:
4. For the Explore page, you must recommend trending photos to users with low latency and high freshness. Which architecture best fits this constraint?
medium
A. Use a batch job that runs every 6 hours to compute trending photos and store results in a database
B. Use a CDN to cache Explore page content globally and invalidate cache every hour
C. Use client-side polling every minute to fetch trending photos from a static cache
D. Use a streaming pipeline that updates trending scores in real-time and stores results in a fast key-value store

Solution

  1. Step 1: Analyze freshness and latency requirements

    Trending photos must be updated with low latency and high freshness, ruling out long batch intervals.

  2. Step 2: Evaluate architectures

    Batch jobs every 6 hours (A) are too stale. Client polling (C) adds load and latency. CDN cache invalidation (D) is coarse-grained and slow. Streaming pipeline (B) updates trending scores in real-time and stores in fast KV store, meeting freshness and latency.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Streaming pipeline enables real-time trending updates [check]
Hint: Real-time streaming pipelines enable fresh trending data [check]
Common Mistakes:
5. At 10x transaction volume, which component in the fraud detection system is most likely to become the first bottleneck, and why?
hard
A. Rule Engine, because complex rules increase CPU usage linearly with transactions and cannot be parallelized
B. Feature Store, because it must serve low-latency queries for all transactions and cannot scale horizontally easily
C. Kafka ingestion cluster, because partition count limits parallelism and increasing partitions impacts ordering guarantees
D. Real-Time Scoring service, because it maintains stateful caches per user that cannot be sharded easily

Solution

  1. Step 1: Analyze Feature Store scaling (A)

    Feature Stores often scale horizontally with caching and sharding; not usually first bottleneck.

  2. Step 2: Analyze Kafka ingestion cluster (B)

    Kafka partitions limit parallelism; increasing partitions can break ordering guarantees per user, making scaling challenging and causing ingestion bottlenecks.

  3. Step 3: Analyze Rule Engine and Scoring (C and D)

    Rule Engine can be scaled horizontally with stateless evaluation; scoring caches can be sharded by user ID, so less likely first bottleneck.

  4. Final Answer:

    Option C -> Option C

  5. Quick Check:

    Kafka partition limits -> ingestion bottleneck at 10x scale [check]
Hint: Kafka partitions limit ingestion scale before others [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard