Bird
Raised Fist0

Stateless vs Stateful Services - Design Implications at Scale

HLDmediumInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Stateless vs Stateful Services - Design Implications at Scale
45 min · senior
senior

You are designing backend services for a 50 million daily active user (DAU) social media startup. The system must handle 50M concurrent users with P99 latency under 200ms for core user interactions. Data encryption at rest and in transit is mandatory due to privacy regulations. The system must support offline message delivery and seamless user session failover. You need to decide when to use stateless vs stateful services and design the architecture accordingly.

Constraints
  • P99 latency under 200ms for user requests
  • End-to-end encryption at rest and in transit
  • Support offline message delivery with guaranteed eventual consistency
  • Seamless session failover without user disruption
  • Scale to 50 million DAU with peak concurrency of 10 million users
📋
Guided Steps
1/5
Step 1
Choosing Stateless vs Stateful Service Boundaries
How would you decide which components should be stateless and which should maintain state? What criteria influence this decision at 50M DAU scale?
💡 Hint
Consider scalability, fault tolerance, and session affinity requirements.
Key Decision
Use stateless services (e.g., Kubernetes pods behind an ALB) for front-end API gateways to enable horizontal scaling and failover; use stateful services (e.g., Redis Cluster with persistence) for session management and offline message queues to maintain user state and guarantee delivery.
Interviewer Expectation
Mid: Identifies stateless frontends for scalability. Senior: Explains stateful session stores for failover and offline delivery, and why embedding state in frontends breaks at scale due to sticky sessions and recovery complexity.
⚠️Common Wrong Answer
Making all services stateful leads to poor horizontal scalability and complex failover, causing increased latency and downtime.
Step 2
Session State Management and Failover
How would you design session state management to support seamless failover and offline delivery with P99 < 200ms?
💡 Hint
Think about distributed in-memory stores with persistence and replication.
Key Decision
Use Redis Enterprise with active-active geo-replication and AOF persistence to store session state and offline messages, enabling fast reads/writes and seamless failover with minimal data loss.
Interviewer Expectation
Mid: Names Redis for session store. Senior: Details Redis Enterprise active-active replication and AOF persistence to meet latency and durability constraints under failover.
⚠️Common Wrong Answer
Using local in-memory session state in API pods causes session loss on pod failure and requires sticky sessions, which degrade scalability and availability.
Step 3
Data Encryption and Compliance
How do you ensure end-to-end encryption at rest and in transit without compromising latency?
💡 Hint
Consider TLS termination points and encryption at storage layers.
Key Decision
Terminate TLS at the API gateway (Envoy proxy) for in-transit encryption; use AWS KMS-integrated encryption for data at rest in Redis and Cassandra to meet compliance without adding latency to hot path.
Interviewer Expectation
Mid: Mentions TLS and encryption at rest. Senior: Explains TLS termination at Envoy for low latency, and KMS integration for transparent encryption in Redis and Cassandra with minimal performance impact.
⚠️Common Wrong Answer
Encrypting data in application code adds CPU overhead and increases latency beyond 200ms P99.
Step 4
Offline Message Delivery and Synchronization
How would you design offline message delivery and client synchronization to guarantee eventual consistency and low latency on reconnect?
💡 Hint
Think about message queues with partitioning and cursor-based sync.
Key Decision
Use Apache Kafka with conversation_id as partition key to ensure ordered message delivery; clients sync using cursor offsets stored in Cassandra for durable, scalable offline message retrieval.
Interviewer Expectation
Mid: Names Kafka for messaging. Senior: Explains partitioning by conversation_id to guarantee order and locality, and Cassandra for durable offset storage enabling efficient cursor-based sync.
⚠️Common Wrong Answer
Using HTTP polling for offline sync causes massive load spikes and latency spikes at 50M DAU.
Step 5
API Design for Stateless and Stateful Interactions
Sketch 3 core API endpoints that illustrate client interactions with stateless and stateful services, including message send, sync, and acknowledgment.
💡 Hint
Consider REST or gRPC endpoints for sending messages, syncing offline messages, and acknowledging receipt.
Key Decision
Design POST /messages to stateless API gateway forwarding to Kafka producer; GET /messages/sync to query Cassandra for offline messages; POST /messages/ack to update Redis session state and Kafka consumer offsets.
Interviewer Expectation
Mid: Provides basic REST endpoints. Senior: Explains separation of concerns-stateless API gateway for ingestion, Kafka for messaging, Cassandra for durable sync, Redis for session state and ACK tracking.
⚠️Common Wrong Answer
Combining all logic in a single stateful service leads to poor scalability and complex failure recovery.
🔌
API Design Challenge
Sketch 3 core API endpoints illustrating how clients interact with stateless and stateful services in this system.
Hints
  • How does the client send and receive messages with session state?
  • How does the client synchronize offline messages upon reconnect?
  • How does the client acknowledge message receipt to ensure delivery guarantees?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Clear distinction between stateless and stateful components with scale rationale
  • Use of Redis Enterprise with persistence and geo-replication for session state
  • Kafka partitioning strategy for ordered message delivery
  • Cassandra for durable offset storage enabling cursor-based sync
  • Encryption strategy that balances compliance and latency
Good to Have - 30%
  • API design that cleanly separates message send, sync, and ack
  • Discussion of failover and recovery scenarios for stateful services
  • Consideration of offline delivery guarantees and eventual consistency
Bonus - 10%
  • Detailed latency budget analysis for each component
  • Discussion of multi-region deployment for disaster recovery
Anti-patterns to Avoid
Embedding session state in stateless API pods causes sticky sessions and poor failover, leading to increased latency and downtime.
Using HTTP polling for offline sync at 50M DAU causes 50 million reconnects per minute just to check for messages, collapsing the system under load and causing latency spikes.
Encrypting data in application code adds CPU overhead and increases latency beyond the 200ms P99 SLA, degrading user experience.
✅
Model Answer
▼ Reveal

This design balances stateless and stateful services to meet strict latency, scalability, and security requirements for a 50M DAU social media platform. Stateless API gateways enable horizontal scaling and TLS termination, while Redis Enterprise manages session state with geo-replication and persistence for failover. Kafka ensures ordered message delivery partitioned by conversation_id, and Cassandra stores durable offsets for efficient offline sync. AWS KMS integration provides compliant encryption at rest without impacting latency. The API design cleanly separates message ingestion, synchronization, and acknowledgment to maintain scalability and reliability.

Key Decisions
  • Use Kubernetes pods behind an ALB for stateless API gateways to enable horizontal scaling and TLS termination.
  • Employ Redis Enterprise with active-active geo-replication and AOF persistence to store session state and offline messages, ensuring fast access and seamless failover.
  • Partition Kafka topics by conversation_id to guarantee ordered message delivery and locality.
  • Store client sync cursor offsets in Cassandra partitioned by conversation_id for durable, scalable offline message retrieval.
  • Terminate TLS at Envoy proxy to minimize latency and use AWS KMS-integrated encryption for data at rest in Redis and Cassandra to meet compliance without impacting hot path performance.
  • Design REST endpoints: POST /messages for sending messages via stateless gateway to Kafka, GET /messages/sync for offline message retrieval from Cassandra, and POST /messages/ack to update Redis session state and Kafka offsets.
  • Stateless API gateways using Envoy on Kubernetes for horizontal scaling and TLS termination
  • Redis Enterprise with active-active geo-replication and AOF persistence for session state and offline message caching
  • Apache Kafka with conversation_id partition key for ordered, scalable message delivery
  • Cassandra with partition_key=conversation_id for durable offset storage and efficient cursor-based client sync
  • AWS KMS-integrated encryption for data at rest in Redis and Cassandra to meet compliance without latency impact
  • Cursor-based synchronization protocol to enable efficient offline message retrieval and eventual consistency
  • Separation of concerns: stateless frontends for request handling, stateful backends for session and message state
  • API design with POST /messages, GET /messages/sync, and POST /messages/ack endpoints to cleanly separate message ingestion, sync, and acknowledgment

Practice

(1/5)
1. A professional networking platform must handle 1 million connection requests per second and ensure eventual consistency without blocking user actions. Which component is best suited to decouple the high write throughput from downstream processing?
easy
A. Use a distributed message queue like Kafka to buffer and asynchronously process connection events
B. Use an in-memory cache like Redis to store connection requests temporarily without persistence
C. Use a relational database with immediate synchronous writes for all connection requests
D. Use a REST API gateway that directly writes to the graph database synchronously

Solution

  1. Step 1: Identify the need for decoupling high write throughput

    Immediate synchronous writes (Use a relational database with immediate synchronous writes for all connection requests and D) block user actions and don't scale well at 1M writes/sec. Redis (Use an in-memory cache like Redis to store connection requests temporarily without persistence) lacks durability and persistence guarantees for critical connection data.

  2. Step 2: Evaluate Kafka's role

    Kafka buffers high write volume asynchronously, ensuring durability and eventual consistency without blocking user requests.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Kafka buffers writes asynchronously -> decouples load [check]
Hint: Use durable async queues to decouple high write load [OK]
Common Mistakes:
2. If the Metadata Server crashes before acknowledging a file rename operation, what is the most likely failure observed by the client?
medium
A. Client's write operations to the file silently fail without notification
B. Client receives an error and the file remains under the old name with no partial rename
C. Client sees the file renamed but data blocks lost
D. Client sees the file under both old and new names due to partial metadata update

Solution

  1. Step 1: Understand rename operation atomicity

    Rename involves updating metadata entries; if crash occurs mid-update, partial state may persist.

  2. Step 2: Trace failure effect

    Partial metadata update can cause both old and new filenames to appear, confusing clients.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Partial metadata updates cause duplicate filenames visible to client [check]
Hint: Partial metadata update causes duplicate filenames visible [check]
Common Mistakes:
3. You must choose between a monolithic architecture and microservices for a system with rapidly evolving features and a small team. Which choice best justifies your design decision to the interviewer?
medium
A. Choose microservices because they always scale better regardless of team size
B. Choose monolith to reduce operational complexity and speed up feature delivery for a small team
C. Choose microservices to isolate failures even if it increases deployment overhead
D. Choose monolith because it requires no testing or CI/CD pipelines

Solution

  1. Step 1: Analyze team size and feature velocity constraints

    Small teams benefit from simpler architectures to reduce overhead and speed delivery.

  2. Step 2: Evaluate operational complexity of microservices

    Microservices add deployment and operational complexity, which can slow small teams.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Monolith reduces complexity, speeds delivery for small teams [check]
Hint: Small teams prefer monoliths for simplicity and speed [check]
Common Mistakes:
4. At 10x scale, which component in the notification system is most likely to fail first and why?
hard
A. The Delivery Workers, because they cannot horizontally scale beyond a fixed number of threads per machine
B. The push notification gateway, because mobile platform rate limits throttle outgoing messages
C. The relational database storing user preferences, due to increased read/write contention
D. The presence cache, because it is stateful and cannot be sharded easily across many servers

Solution

  1. Step 1: Analyze Delivery Worker scalability

    Delivery Workers are CPU and thread-bound; increasing scale requires more workers, but thread limits and resource contention cause bottlenecks.

  2. Step 2: Compare other components

    Relational DB (A) can be sharded; push gateways (C) are rate-limited but can be scaled with multiple credentials; presence cache (D) can be sharded with consistent hashing.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Delivery Workers hit thread and CPU limits first at 10x scale [check]
Hint: Delivery Workers hit thread limits before other components [check]
Common Mistakes:
5. At 10x scale, which component in an async messaging chat system fails first and why?
hard
A. The database, because increased writes cause lock contention
B. The message queue brokers, due to partition leader overload
C. The chat servers, because WebSocket connections are stateful and cannot be resharded easily
D. The delivery workers, because retry queues grow exponentially

Solution

  1. Step 1: Analyze stateful components and scaling challenges

    Databases (The database, because increased writes cause lock contention) can be scaled with sharding and replicas. Message queue brokers (The message queue brokers, due to partition leader overload) scale horizontally with partitions. Delivery workers (The delivery workers, because retry queues grow exponentially) can be scaled with backpressure.

  2. Step 2: Identify chat servers as stateful WebSocket connection holders

    Chat servers maintain persistent WebSocket connections bound to server memory, making resharding or migration difficult at scale.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Stateful WebSocket servers -> first bottleneck at 10x scale [check]
Hint: Stateful WebSocket servers limit horizontal scaling first [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard