Bird
Raised Fist0

Design Pastebin - Text Storage, Expiration, Analytics

HLDmediumInterview freq: medium
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🏗️
Design Pastebin - Text Storage, Expiration, Analytics
Design a scalable Pastebin service supporting text storage, expiration, and usage analytics
mediumStorage and CollaborationFreq: medium
Prerequisites:System Design FundamentalsCAP Theorem
⚙️
Functional Requirements
  • Store arbitrary text snippets submitted by users with unique URLs
  • Support configurable expiration times per snippet (e.g., 1 hour to 1 year)
  • Retrieve snippets by URL with low latency
  • Provide analytics on snippet usage (views, creation trends) aggregated daily
  • Allow snippet deletion before expiration
  • Support bulk expiration cleanup without impacting read performance
📊
Non-Functional Requirements
  • Handle 2 billion daily active users (DAU)
  • Support 925K write requests per second at peak (derived from assumptions)
  • Ensure 99.99% availability for read and write operations
  • Achieve 100ms P99 read latency for snippet retrieval
  • Data durability with replication factor of 3 and eventual consistency for writes
❓
Clarifying Questions to Ask
  • What is the maximum size allowed per text snippet?
  • Is expiration time fixed or user-configurable per snippet?
  • What analytics metrics are required and their freshness requirements?
  • Are snippets immutable after creation or can they be updated?
  • What is the expected read-to-write ratio?
  • Is global distribution required or single-region deployment acceptable?
Out of Scope
✗ User authentication and authorization
✗ Rich text formatting or multimedia content storage
✗ Real-time collaborative editing
✗ Advanced search over snippet contents
📌
Assumptions
  • 2 billion DAU
  • Each user creates 40 snippets per day on average
  • Average snippet size is 100 bytes
  • Read-to-write ratio is 10:1
⚡
QPS Calculation
write qps2B users × 40 snippets/day ÷ 86400 sec ≈ 925,926 writes/sec
peak write qpsAssuming 2× peak factor → 1,851,852 writes/sec
read qpsWrite QPS × 10 (read-to-write ratio) = 9,259,260 reads/sec
💾
Storage Estimate
daily storage925,926 writes/sec × 86400 sec × 100 bytes ≈ 8 TB/day
retention days30
total storage8 TB/day × 30 days = 240 TB
with replication240 TB × 3 (replication factor) = 720 TB
🖥️
Server Estimate
write serversAssuming each write server handles 10K writes/sec → 1,851,852 ÷ 10,000 ≈ 186 servers
read serversAssuming each read server handles 50K reads/sec → 9,259,260 ÷ 50,000 ≈ 186 read servers
🏗️
Architecture Diagram
Architecture Diagram
          +-------------+
          |  Client UI  |
          +------+------+
                 |
                 v
          +------+-------+
          | API Gateway  |
          +------+-------+
                 |
        +--------+---------+
        |                  |
        v                  v
 +------+-------+   +------+-------+
 | Write Service|   | Read Service |
 +------+-------+   +------+-------+
        |                  |
        v                  v
 +------+-------+   +------+-------+
 | Kafka Topic  |   | Cache Layer  |
 +------+-------+   +------+-------+
        |                  |
        v                  v
 +------+-------+   +------+-------+
 | Cassandra DB |   | Analytics DB |
 +--------------+   +--------------+
        |
        v
 +------+-------+
 | Expiration   |
 | Worker       |
 +--------------+
🧩
Components
API Gateway
Entry point for all client requests, routes to read/write services
Why thisNginx with Layer 7 routing supports high QPS and SSL termination efficiently
Why notAWS ALB lacks fine-grained routing control and higher latency at 10M QPS scale
Write Service
Handles snippet creation, validates input, writes to Kafka
Why thisStateless microservice in Go for low latency and easy horizontal scaling
Why notMonolithic app would bottleneck and complicate scaling at 1.8M writes/sec
Kafka Topic
Durable, ordered write-ahead log for snippet writes and expiration events
Why thisKafka supports high throughput, partitioning, and retention policies for 2B DAU scale
Why notRabbitMQ lacks partitioning and throughput for 1.8M writes/sec sustained
Cassandra DB
Primary storage for snippets with TTL-based expiration
Why thisCassandra's wide-column model supports high write throughput and TTL natively for expiration
Why notPostgreSQL requires manual sharding and does not scale linearly at 1.8M writes/sec
Cache Layer (Redis Cluster)
Cache hot snippets for sub-100ms P99 read latency
Why thisRedis TTL natively expires stale snippets without cron jobs, supports clustering for scale
Why notMemcached lacks native TTL and persistence, complicating expiration handling
Read Service
Handles snippet retrieval, queries cache first then Cassandra
Why thisStateless microservice optimized for high read QPS with cache fallback
Why notDirect DB queries would increase latency and load on Cassandra
Expiration Worker
Processes expired snippet events from Kafka to clean up storage and cache
Why thisKafka consumer group ensures reliable, scalable expiration processing asynchronously
Why notCron jobs scanning DB cause high load and inconsistent expiration timing
Analytics DB (ClickHouse)
Stores aggregated snippet usage metrics for analytics queries
Why thisClickHouse supports fast OLAP queries on large datasets with daily aggregation
Why notElasticsearch is less efficient for large-scale time-series aggregation
🔄
Data Flow
1
Client UI → API Gateway
Submit snippet creation request with text and expiration
API Gateway routes to Write Service
2
Write Service → Kafka Topic
Publish snippet creation event with metadata
Ensures durable, ordered write log
3
Expiration Worker → Cassandra DB and Redis Cache
Consume expiration events, delete expired snippets
Keeps storage and cache clean without impacting reads
4
Read Service → Redis Cache
Check cache for snippet by URL
Fast path for snippet retrieval
5
Read Service → Cassandra DB
Fallback read if cache miss
Ensures data availability with eventual consistency
6
Write Service → Analytics DB
Send snippet creation and view events asynchronously
Supports daily usage analytics

Practice

(1/5)
1. A distributed task scheduler must support millions of scheduled jobs with varying priorities and ensure tasks are executed exactly once. Which component is best suited to manage task ordering and priority under these constraints?
easy
A. A distributed priority queue with partitioned shards
B. A relational database with indexed priority columns
C. A simple FIFO message queue like RabbitMQ without priority support
D. A cron job scheduler running independently on each node

Solution

  1. Step 1: Identify priority and scale requirements

    A relational DB (A) struggles with high write throughput and priority ordering at scale; simple FIFO queues (C) lack priority support; cron jobs (D) do not coordinate distributed execution or priority.

  2. Step 2: Evaluate distributed priority queue

    A distributed priority queue with partitioned shards (B) can scale horizontally, maintain priority ordering per shard, and support exactly-once execution semantics.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Distributed priority queue shards scale and preserve ordering [check]
Hint: Distributed priority queues scale and order tasks by priority [check]
Common Mistakes:
2. Without the seat lock expiration mechanism, what exact failure occurs during concurrent bookings?
medium
A. Users receive immediate errors due to deadlock detection in the database
B. Seats remain locked indefinitely, causing availability to drop and bookings to stall
C. Payment processing fails because seat status is inconsistent across services
D. The system automatically cancels bookings after a fixed timeout regardless of payment

Solution

  1. Step 1: Understand role of seat lock expiration

    Expiration releases seats if payment or booking is abandoned, preventing indefinite holds.

  2. Step 2: Trace failure without expiration

    Locks never expire, so seats remain reserved even if users abandon payment, reducing available seats and causing booking stalls.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    No TTL -> stale locks -> seats unavailable indefinitely [check]
Hint: No expiration -> stale locks -> seats blocked forever [OK]
Common Mistakes:
3. For a payment API that must guarantee exactly-once processing despite client retries, which architecture best balances consistency and performance?
medium
A. Use a strongly consistent centralized idempotency key store with synchronous writes before processing
B. Use a distributed cache with eventual consistency for idempotency keys to maximize throughput
C. Rely on client-generated unique request IDs and process all requests without server-side deduplication
D. Use asynchronous background reconciliation to detect and fix duplicates after processing

Solution

  1. Step 1: Evaluate consistency needs

    Exactly-once processing requires strong consistency on idempotency keys to prevent duplicates during retries.

  2. Step 2: Compare architectures

    Distributed cache with eventual consistency risks duplicates; client-only IDs lack server enforcement; async reconciliation delays correctness.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Strongly consistent idempotency store ensures exactly-once processing [check]
Hint: Strong consistency needed for exactly-once idempotency [check]
Common Mistakes:
4. At 10x scale, which component in a cache stampede prevention system is most likely to fail first, and why?
hard
A. Cache servers, because they run out of memory storing more keys
B. Backend database, because it cannot handle the increased read load from cache misses
C. Distributed lock service, because lock contention and coordination overhead grow exponentially
D. Load balancers, because they cannot distribute requests evenly at high concurrency

Solution

  1. Step 1: Identify bottlenecks at 10x scale

    Backend and cache scale horizontally; load balancers scale well.

  2. Step 2: Analyze distributed lock service

    Lock service faces increased contention and coordination overhead, causing latency and failures.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Distributed lock service bottlenecks first due to coordination overhead [check]
Hint: Lock service bottlenecks first due to exponential coordination overhead [OK]
Common Mistakes:
5. At 10x scale, which component in the proximity service is most likely to fail first and why?
hard
A. The Quadtree service because its recursive traversal becomes a CPU bottleneck with many concurrent queries
B. The database because it cannot handle the increased write throughput for location updates
C. The Geohash prefix index because its fixed-length buckets cause uneven load and hotspotting
D. The API gateway because it cannot maintain the increased number of open WebSocket connections

Solution

  1. Step 1: Analyze component scalability

    Databases (The database because it cannot handle the increased write throughput for location updates) can be sharded and scaled horizontally; API gateways (The API gateway because it cannot maintain the increased number of open WebSocket connections) typically scale with load balancers.

  2. Step 2: Identify Quadtree bottleneck

    Quadtree's recursive traversal is CPU-intensive and stateful, making it hard to parallelize and shard, causing CPU bottlenecks under high query concurrency.

  3. Step 3: Evaluate Geohash index load

    Geohash buckets (The Geohash prefix index because its fixed-length buckets cause uneven load and hotspotting) can cause hotspots but are easier to shard and cache than Quadtree traversal.

  4. Final Answer:

    Option A -> Option A

  5. Quick Check:

    Quadtree traversal CPU bottleneck limits scale first [check]
Hint: Quadtree traversal CPU bottleneck limits scale first [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard