Bird
Raised Fist0

Design Pastebin - Text Storage, Expiration, Analytics

HLDmediumInterview freq: medium
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Design Pastebin - Text Storage, Expiration, Analytics
45 min · senior
senior

You are tasked with designing a Pastebin-like service that allows users to store arbitrary text snippets with optional expiration times. The system must support 50 million daily active users (DAU), handle high write and read throughput, ensure data encryption at rest and in transit, provide near real-time analytics on paste usage, and maintain a P99 latency under 200ms for read and write operations.

Constraints
  • All stored text must be encrypted at rest using AES-256 with customer key management.
  • P99 latency for read and write APIs must be under 200ms globally.
  • Pastes can have optional expiration times; expired pastes must be deleted or inaccessible within 1 minute of expiration.
  • Analytics must provide near real-time metrics on paste creation rates, popular languages, and expiration patterns.
  • The system must handle 50M DAU with an estimated 10 million new pastes created daily.
📋
Guided Steps
1/5
Step 1
Data Storage Choice and Schema Design
Which database technology and schema design would you choose to store encrypted text snippets with optional expiration, and why?
💡 Hint
Consider write-heavy workloads, TTL support, and range queries for expiration.
Key Decision
Use Apache Cassandra with partition key = paste_id and clustering on creation_time; leverage Cassandra's native TTL feature for automatic expiration.
Interviewer Expectation
Mid-level: Names Cassandra for scalability. Senior: Explains TTL support for expiration and why relational DBs like PostgreSQL fail at 50M DAU due to write amplification and scaling limits.
⚠️Common Wrong Answer
Using a relational database like MySQL with frequent deletes causes write amplification and poor horizontal scalability, collapsing under 10M writes/day.
Step 2
Encryption Strategy
How would you implement encryption for stored pastes to meet AES-256 encryption at rest and in transit requirements?
💡 Hint
Think about key management, encryption location, and performance impact.
Key Decision
Encrypt text client-side using AES-256 with keys managed by AWS KMS; store ciphertext in Cassandra; use TLS 1.3 for in-transit encryption.
Interviewer Expectation
Mid-level: Mentions TLS and encryption at rest. Senior: Details client-side encryption to reduce server trust surface and explains AWS KMS integration for key rotation and audit.
⚠️Common Wrong Answer
Encrypting only at the server side increases risk if server is compromised and adds latency due to synchronous encryption on every request.
Step 3
Expiration Handling and Deletion
How will you ensure expired pastes are deleted or inaccessible within 1 minute of expiration?
💡 Hint
Consider TTL features, background cleanup jobs, and cache invalidation.
Key Decision
Leverage Cassandra's TTL for automatic data expiry; complement with a background cleanup job using Apache Spark to remove tombstones and update analytics; invalidate Redis cache entries on expiration.
Interviewer Expectation
Mid-level: Uses TTL for expiration. Senior: Explains tombstone compaction impact on Cassandra and uses Spark batch jobs to maintain storage health and analytics accuracy.
⚠️Common Wrong Answer
Relying solely on background jobs without TTL causes stale data exposure and inconsistent expiration timing.
Step 4
Analytics Pipeline Design
How would you design the analytics pipeline to provide near real-time metrics on paste creation and expiration patterns?
💡 Hint
Think about event streaming, aggregation, and storage for dashboards.
Key Decision
Use Apache Kafka to stream paste creation and expiration events; process with Apache Flink for real-time aggregation; store results in ClickHouse for low-latency analytics queries.
Interviewer Expectation
Mid-level: Names Kafka for event streaming. Senior: Explains Flink's windowing for near real-time aggregation and why ClickHouse outperforms Elasticsearch for analytical queries at scale.
⚠️Common Wrong Answer
Batch processing with Hadoop causes high latency and stale analytics, failing near real-time requirements.
Step 5
Caching and Latency Optimization
What caching strategy would you implement to maintain P99 latency under 200ms for reads?
💡 Hint
Consider cache invalidation on expiration and cache hit ratio optimization.
Key Decision
Use Redis as a read-through cache with TTL matching Cassandra TTL; on cache miss, fetch from Cassandra; invalidate Redis keys immediately upon paste expiration events from Kafka.
Interviewer Expectation
Mid-level: Suggests Redis caching. Senior: Explains read-through cache pattern with TTL sync and event-driven invalidation to avoid stale reads and reduce Cassandra load.
⚠️Common Wrong Answer
Using write-through cache without invalidation leads to stale data and inconsistent reads post-expiration.
🔌
API Design Challenge
Sketch 3 core RESTful API endpoints for the Pastebin system that cover paste creation, retrieval, and expiration handling.
Hints
  • How does the client send and store encrypted text securely?
  • How does the client retrieve a paste efficiently with low latency?
  • How does the system notify or handle paste expiration and deletion?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Use of Cassandra with TTL for scalable storage and expiration.
  • Client-side encryption with AWS KMS key management.
  • Kafka + Flink + ClickHouse pipeline for near real-time analytics.
  • Redis caching with TTL and event-driven invalidation.
Good to Have - 30%
  • Explanation of tombstone compaction impact and cleanup strategy.
  • Detailed API design covering encryption and expiration handling.
  • Latency estimation and justification for P99 < 200ms.
Bonus - 10%
  • Discussion of multi-region replication and disaster recovery.
  • Use of CDN or edge caching for global latency optimization.
Anti-patterns to Avoid
Using relational databases like MySQL for write-heavy paste storage leads to poor horizontal scalability and high write amplification, causing system collapse under high write loads.
Encrypting only at the server side increases risk of data exposure if server is compromised and adds latency due to synchronous encryption on every request.
Relying solely on batch jobs for expiration causes stale data exposure and inconsistent user experience because expired pastes remain accessible until cleanup runs.
Polling-based cache invalidation or expiration detection causes high load and latency spikes due to massive numbers of requests at scale (e.g., 50M DAU polling causes 50M reconnects/min).
Batch analytics processing causes stale metrics, failing near real-time requirements and delaying actionable insights.
Write-through caching without invalidation leads to stale reads after paste expiration, causing users to see expired content.
✅
Model Answer
▼ Reveal

The system uses Apache Cassandra with TTL for encrypted paste storage and expiration, client-side AES-256 encryption with AWS KMS for secure key management, and TLS 1.3 for in-transit encryption. Expiration is handled via Cassandra TTL combined with Apache Spark batch jobs to compact tombstones and maintain storage health. The analytics pipeline streams paste creation and expiration events through Apache Kafka, processes them in real-time with Apache Flink using windowing, and stores aggregated results in ClickHouse for low-latency queries. Redis is used as a read-through cache with TTL aligned to Cassandra and event-driven invalidation triggered by expiration events from Kafka to maintain P99 latency under 200ms globally.

Key Decisions
  • Use Cassandra with partition key = paste_id and clustering on creation_time to support high write throughput and TTL-based expiration.
  • Encrypt text client-side using AES-256 with AWS KMS key management to reduce server trust surface and enable key rotation.
  • Leverage Cassandra TTL for automatic expiration and complement with Spark batch jobs to handle tombstone compaction and maintain analytics accuracy.
  • Stream paste creation and expiration events via Kafka, process with Flink for near real-time aggregation, and store results in ClickHouse for low-latency analytics.
  • Implement Redis read-through cache with TTL matching Cassandra and invalidate cache entries immediately upon expiration events from Kafka to ensure low latency and data freshness.
  • Apache Cassandra with paste_id partition key and TTL for scalable encrypted text storage and expiration.
  • Client-side AES-256 encryption with AWS KMS for secure key management and TLS 1.3 for in-transit encryption.
  • Cassandra TTL combined with Apache Spark batch jobs for tombstone compaction and cleanup.
  • Apache Kafka event streaming for paste creation and expiration events.
  • Apache Flink for real-time analytics aggregation with windowing.
  • ClickHouse as OLAP store for low-latency analytics queries.
  • Redis read-through cache with TTL aligned to Cassandra and event-driven invalidation.
  • RESTful API design supporting encrypted paste creation, retrieval, and expiration notification.

Practice

(1/5)
1. In a message processing pipeline, at Step 4 the Retry Worker reads a message from the Dead Letter Queue. What happens if the message exceeds the maximum retry attempts configured?
easy
A. The Retry Worker re-enqueues the message back to the main queue indefinitely
B. The Retry Worker moves the message to a separate permanent failure store for manual inspection
C. The Retry Worker discards the message permanently without notification
D. The Retry Worker resets the retry count and retries immediately

Solution

  1. Step 1: Understand retry limit enforcement

    When max retries are exceeded, messages should not be retried endlessly to avoid infinite loops.

  2. Step 2: Identify correct failure handling

    Moving messages to a permanent failure store (C) enables manual inspection and prevents loss.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Exceeded retries -> move to permanent failure store, not discard or infinite retry [check]
Hint: Max retries exceeded -> move to failure store, not discard or loop [OK]
Common Mistakes:
2. In a Cassandra-based messaging system, at step 4 the coordinator node receives a write request for a message with partition key user123. What happens if the coordinator node's local replica is down?
easy
A. The coordinator immediately fails the write request with an error
B. The coordinator writes to a backup relational database instead
C. The coordinator routes the write to other replicas responsible for user123's partition
D. The coordinator retries the write on the same node until it recovers

Solution

  1. Step 1: Understand coordinator role in Cassandra write path

    The coordinator routes writes to all replicas responsible for the partition key; if one replica is down, it routes to the others.

  2. Step 2: Evaluate failure handling

    Writes are not failed immediately if a replica is down; coordinator does not retry indefinitely on the same node; no fallback to relational DB.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Coordinator routes writes to available replicas -> ensures write availability [check]
Hint: Coordinator routes writes to live replicas, not fail immediately [check]
Common Mistakes:
3. Without the automated rollback system, what exactly breaks when a misconfiguration causes a route leak?
medium
A. The system silently drops all BGP updates, causing network partition
B. Route leak detection alerts flood the monitoring system but traffic continues normally
C. The system automatically quarantines all routes, causing a complete service outage
D. Alice's traffic is routed incorrectly; operators must manually detect and fix the leak, causing prolonged outages

Solution

  1. Step 1: Trace failure without rollback

    Misconfigurations propagate incorrect routes; without rollback, manual detection delays fix.

  2. Step 2: Eliminate other failure modes

    Silent drops (A) don't occur; automatic quarantine (C) would cause total outage, not typical; alert floods (D) don't prevent traffic misrouting.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Manual fix delays cause prolonged incorrect routing [check]
Hint: No rollback means manual fix delays outages [check]
Common Mistakes:
4. If the distributed lock service is unavailable, what exactly breaks in the hotel booking system's reservation flow?
medium
A. Reservations proceed normally but double bookings increase due to lack of coordination
B. Reservation requests block indefinitely waiting for locks, causing user timeouts and failed bookings
C. Inventory service marks rooms as available incorrectly, causing UI to show wrong availability
D. Payment processing fails because it depends on the lock service for transaction ordering

Solution

  1. Step 1: Identify lock service role in reservation flow

    The lock service ensures exclusive access to room inventory during reservation to prevent double booking.

  2. Step 2: Trace failure impact

    Without the lock service, Inventory cannot reliably mark rooms as unavailable, leading to stale or incorrect availability states shown to users (Inventory service marks rooms as available incorrectly, causing UI to show wrong availability). Reservations may still proceed but with inconsistent inventory state. Blocking indefinitely (Reservation requests block indefinitely waiting for locks, causing user timeouts and failed bookings) is unlikely as fallback mechanisms exist. Payment processing (Payment processing fails because it depends on the lock service for transaction ordering) is independent of locking. Double bookings (Reservations proceed normally but double bookings increase due to lack of coordination) may increase but the immediate failure is incorrect inventory state.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Lock service failure causes stale inventory availability state [check]
Hint: Lock service failure -> stale inventory state, not immediate booking block [check]
Common Mistakes:
5. If the Kafka cluster responsible for fan-out message delivery crashes, what is the exact failure observed in the News Feed system?
medium
A. New posts are lost permanently and never delivered to followers
B. The system blocks new post writes until Kafka recovers, causing user-facing delays
C. Followers receive duplicate feed updates due to Kafka replaying messages
D. Delivery Workers cannot fetch fan-out messages, so followers see stale feeds until recovery

Solution

  1. Step 1: Understand Kafka's role in fan-out delivery

    Kafka acts as a durable queue for fan-out messages. If it crashes, Delivery Workers cannot fetch new messages, causing stale feeds.

  2. Step 2: Evaluate failure modes

    Posts are not lost immediately (A) due to Kafka's durability; writes are asynchronous so no blocking (B); duplicates (D) occur only on replay, not crash.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Kafka crash stalls delivery, causing stale feeds [check]
Hint: Kafka crash stalls delivery, causing stale feeds [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard