Bird
Raised Fist0

Design a Multi-Tenant SaaS Platform - Isolation, Billing, Feature Flags

HLDhardInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Design a Multi-Tenant SaaS Platform - Isolation, Billing, Feature Flags
45 min · senior
senior

You are designing a multi-tenant SaaS platform serving 50 million daily active users (DAU) across thousands of tenants. Each tenant requires strong data isolation, secure billing integration, and dynamic feature flag management. The system must handle tenant-specific customizations without impacting others, ensure encrypted data at rest and in transit, and maintain P99 latency under 200ms for all user-facing APIs.

Constraints
  • Data must be encrypted at rest and in transit to comply with enterprise security standards.
  • Tenant data isolation must prevent any cross-tenant data leakage even under failure scenarios.
  • Billing system must support real-time usage tracking and invoicing with eventual consistency.
  • Feature flags must be dynamically configurable per tenant with near real-time propagation (<5s).
  • System must maintain P99 API latency under 200ms at 50M DAU scale.
📋
Guided Steps
1/5
Step 1
Tenant Data Isolation Strategy
How would you design tenant data isolation to prevent cross-tenant data leakage at scale?
💡 Hint
Consider database schema design and network-level isolation techniques.
Key Decision
Use separate logical schemas per tenant in PostgreSQL with Row-Level Security (RLS) enabled, combined with network segmentation via AWS VPCs per tenant group to enforce isolation.
Interviewer Expectation
Mid: Names RLS for data isolation. Senior: Explains why RLS combined with network segmentation prevents both accidental and malicious cross-tenant access and how it scales with thousands of tenants.
⚠️Common Wrong Answer
Single shared schema without RLS - leads to risk of accidental data leakage and complex query filters that degrade performance at scale.
Step 2
Billing System Design
How would you design the billing system to support real-time usage tracking and eventual consistency for invoicing?
💡 Hint
Think about event-driven architectures and data stores optimized for time-series usage data.
Key Decision
Use Apache Kafka for ingesting usage events with tenant_id as partition key, and Apache Druid for real-time OLAP queries to generate billing reports with eventual consistency.
Interviewer Expectation
Mid: Mentions Kafka for event ingestion. Senior: Explains tenant_id partitioning ensures ordering and scalability, and why Druid is chosen over traditional RDBMS for fast aggregation on large time-series data.
⚠️Common Wrong Answer
Direct writes to relational DB for usage events - causes write bottlenecks and poor query performance at 50M DAU scale.
Step 3
Feature Flag Management
How would you implement dynamic, tenant-specific feature flags with near real-time propagation and low latency?
💡 Hint
Consider distributed configuration stores and cache invalidation strategies.
Key Decision
Use LaunchDarkly or an open-source alternative like Unleash backed by Redis for caching feature flags per tenant, with Kafka-based event bus to propagate flag changes within 5 seconds.
Interviewer Expectation
Mid: Names feature flag service with caching. Senior: Explains how Redis caching reduces latency under 200ms P99 and Kafka event bus ensures eventual consistency and near real-time updates per tenant.
⚠️Common Wrong Answer
Polling feature flag service every request - causes high latency and load, violating P99 SLA.
Step 4
API Design for Multi-Tenant Platform
Design 3 core REST API endpoints: tenant onboarding, feature flag evaluation, and billing usage reporting. What are the key request/response elements and security considerations?
💡 Hint
Focus on authentication, tenant context propagation, and idempotency.
Key Decision
Tenant onboarding API uses OAuth2 client credentials flow with tenant metadata stored in DynamoDB; feature flag evaluation API includes tenant_id in JWT claims and caches flags in Redis; billing usage API accepts idempotent event IDs and authenticates via mTLS.
Interviewer Expectation
Mid: Defines endpoints and basic auth. Senior: Details OAuth2 flows, JWT tenant scoping, idempotency tokens to prevent double billing, and mTLS for secure usage reporting.
⚠️Common Wrong Answer
No tenant context in API calls - leads to security risks and inability to isolate tenant data.
Step 5
Handling Scale and Latency Constraints
How would you ensure the system maintains P99 latency under 200ms at 50M DAU scale?
💡 Hint
Consider caching, load balancing, and database partitioning strategies.
Key Decision
Use Redis for caching tenant metadata and feature flags, AWS Application Load Balancer with sticky sessions per tenant group, and partition PostgreSQL schemas by tenant to reduce query latency.
Interviewer Expectation
Mid: Mentions caching and load balancing. Senior: Explains how sticky sessions reduce cross-tenant cache misses, and schema partitioning reduces query complexity to meet 200ms P99 SLA.
⚠️Common Wrong Answer
Relying solely on database vertical scaling - leads to bottlenecks and SLA violations at scale.
🔌
API Design Challenge
Sketch 3 core RESTful API endpoints for the platform: tenant onboarding, feature flag evaluation, and billing usage reporting.
Hints
  • How does the client securely onboard a new tenant and provision resources?
  • How does the client query feature flags ensuring low latency and tenant isolation?
  • How does the client report usage events for billing with fault tolerance?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Strong tenant data isolation using PostgreSQL RLS and network segmentation to prevent data leakage.
  • Event-driven billing ingestion with Kafka partitioned by tenant_id for scalability and ordering.
  • Feature flag system with Redis caching and event bus for near real-time updates and low latency.
  • Secure API design embedding tenant context with OAuth2 and JWT to enforce authorization.
Good to Have - 30%
  • Use of Apache Druid for fast OLAP queries on billing data enabling real-time invoicing.
  • Sticky sessions on load balancers to improve cache hit ratio and reduce latency spikes.
  • Idempotency tokens on billing usage API to prevent double counting under retries.
Bonus - 10%
  • Multi-region deployment with geo-partitioning of tenants to reduce latency and improve availability.
  • Automated tenant onboarding pipeline integrating resource provisioning and security policies.
Anti-patterns to Avoid
Single shared database schema without RLS - causes risk of cross-tenant data leakage and complex query filters that degrade performance.
Direct writes of usage events to relational databases - causes write bottlenecks and poor query performance at 50M DAU scale.
Polling feature flag service every request - causes high latency and load, violating P99 SLA.
No tenant context in API calls - leads to security risks and inability to isolate tenant data.
Relying solely on vertical scaling of databases - leads to bottlenecks and SLA violations at scale.
✅
Model Answer
▼ Reveal

This design uses PostgreSQL with Row-Level Security and AWS VPC segmentation to ensure strong tenant isolation. Billing ingestion is handled via Apache Kafka partitioned by tenant_id for scalability and ordering, with Apache Druid for fast OLAP queries enabling near real-time invoicing. Feature flags are managed using LaunchDarkly or Unleash backed by Redis caching and Kafka event bus for low latency and near real-time updates. APIs are secured with OAuth2 client credentials flow and JWT tokens embedding tenant context, with idempotency tokens and mTLS for billing usage reporting. Redis caching and AWS Application Load Balancer with sticky sessions ensure P99 latency under 200ms at 50M DAU scale.

Key Decisions
  • Use PostgreSQL RLS combined with AWS VPC segmentation to prevent cross-tenant data leakage.
  • Partition Kafka topics by tenant_id to ensure ordered, scalable ingestion of billing events.
  • Use Apache Druid for fast aggregation on large time-series billing data enabling eventual consistency.
  • Implement feature flags with Redis caching and Kafka event bus for near real-time propagation under 5 seconds.
  • Secure APIs with OAuth2 client credentials flow and embed tenant_id in JWT claims for authorization.
  • Use idempotency tokens on billing usage API to prevent double counting under retries.
  • Employ Redis caching and sticky sessions on load balancers to meet P99 latency SLA at 50M DAU.
  • PostgreSQL with Row-Level Security (RLS) per tenant schema for strong data isolation and compliance.
  • AWS VPC network segmentation to isolate tenant groups at the network layer preventing lateral movement.
  • Apache Kafka with tenant_id partition key for ordered, scalable ingestion of billing usage events.
  • Apache Druid for real-time OLAP queries enabling fast billing report generation on large time-series data.
  • LaunchDarkly or Unleash feature flag service backed by Redis caching for low-latency tenant-specific flag evaluation.
  • OAuth2 client credentials flow and JWT tokens embedding tenant context for secure API authentication and authorization.
  • Redis caching layer for tenant metadata and feature flags to meet P99 latency under 200ms at 50M DAU.
  • AWS Application Load Balancer with sticky sessions to optimize cache hit rates and reduce latency variability.

Practice

(1/5)
1. In a system trace, at Step 3 the Storage Estimator component calculates daily storage needs. If the average object size is 2MB and the system expects 500K new objects daily, what is the storage estimate at Step 3?
easy
A. Cannot estimate without knowing retention period
B. 1TB per day, calculated as 500K objects x 2MB each
C. 2TB per day, assuming 4MB average object size instead
D. 500GB per day, assuming 1MB average object size instead

Solution

  1. Step 1: Calculate daily storage

    Multiply 500,000 objects by 2MB each = 1,000,000 MB = 1TB per day.

  2. Step 2: Eliminate other options

    500GB per day, assuming 1MB average object size instead uses wrong object size; 2TB per day, assuming 4MB average object size instead doubles size incorrectly; Cannot estimate without knowing retention period ignores that retention affects total storage, not daily estimate.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    500K x 2MB = 1TB daily storage [check]
Hint: Daily storage = object count x average size [check]
Common Mistakes:
2. In the unique ID generation system, at step 3 the ID generator node receives a timestamp and node ID. What state does the generator maintain to ensure uniqueness when the timestamp is the same as the previous request?
easy
A. It resets the sequence number to zero for each new timestamp
B. It increments a sequence number to differentiate IDs within the same timestamp
C. It waits for the timestamp to advance before generating the next ID
D. It generates a random suffix to append to the ID

Solution

  1. Step 1: Understand sequence number reset behavior

    For each new timestamp, the sequence number resets to zero to start fresh counting.

  2. Step 2: Ensure uniqueness within timestamp by incrementing sequence

    Within the same timestamp, the sequence number increments; resetting happens only when timestamp changes.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Sequence number resets on new timestamp -> uniqueness maintained [check]
Hint: Sequence number resets to zero on new timestamp [check]
Common Mistakes:
3. For a distributed counter requiring strong eventual consistency and minimal coordination, which architecture is best suited?
medium
A. Use sharded CRDT counters replicated asynchronously with conflict-free merges
B. Use a centralized counter service with distributed locks to serialize increments
C. Use a distributed database with linearizable transactions for each increment
D. Use a message queue with exactly-once delivery and synchronous aggregation

Solution

  1. Step 1: Evaluate coordination and consistency trade-offs

    Centralized locks (A) and linearizable transactions (C) impose coordination and reduce availability. Message queues with synchronous aggregation (D) add latency and bottlenecks.

  2. Step 2: Recognize sharded CRDT counters provide eventual consistency with no coordination

    They allow concurrent increments and asynchronous merges, fitting minimal coordination and strong eventual consistency.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Sharded CRDTs enable coordination-free eventual consistency [check]
Hint: CRDTs avoid coordination, unlike locks or linearizable DBs [check]
Common Mistakes:
4. At 10x traffic scale during a flash sale, which component is most likely to fail first and why?
hard
A. The relational database because it cannot handle 10x write throughput without sharding
B. The API Gateway because it maintains stateful user sessions and cannot horizontally scale easily
C. The cache layer because TTL expiration floods the database with cache misses
D. The inventory service because distributed locking causes contention and latency spikes

Solution

  1. Step 1: Identify stateful components under load

    API Gateway maintains user session state (e.g., WebSocket connections), which are sticky and stateful, limiting horizontal scaling.

  2. Step 2: Compare other components

    Relational DB (A) can be sharded; cache (C) TTL floods are mitigated by pre-warming; inventory locking (D) is optimized with partitioning. API Gateway statefulness causes connection limits and bottlenecks first.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    API Gateway stateful sessions limit horizontal scaling [check]
Hint: Stateful API Gateway limits scaling under 10x load [check]
Common Mistakes:
5. At 10x scale, which component in a Cassandra cluster is most likely to fail first and why?
hard
A. The coordinator nodes due to increased load managing large partitions and routing requests
B. The storage nodes due to disk space exhaustion
C. The client application due to increased request latency
D. The network switches due to bandwidth saturation

Solution

  1. Step 1: Identify bottlenecks at 10x scale

    Coordinator nodes handle query routing and aggregation; increased scale causes them to manage larger partitions and more requests.

  2. Step 2: Compare other components

    Storage nodes scale horizontally and add disk; network bandwidth scales with infrastructure upgrades; client latency is secondary to cluster bottlenecks.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Coordinator overload -> first failure point at 10x scale [check]
Hint: Coordinator nodes bottleneck first at scale due to routing load [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard