Bird
Raised Fist0

Distributed File System Design - Metadata Servers and Data Nodes

HLDmediumInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Distributed File System Design - Metadata Servers and Data Nodes
45 min · senior
senior

You are designing a distributed file system for a startup with 50 million daily active users. The system must support encrypted file storage, provide offline file access, and guarantee P99 metadata operation latency under 200ms. The system should scale to billions of files and petabytes of data, ensuring high availability and consistency of metadata.

Constraints
  • All file metadata and data must be encrypted at rest and in transit.
  • Offline file access requires local caching with eventual sync.
  • Metadata operations (e.g., file lookup, permission checks) must have P99 latency < 200ms.
  • System must handle 50M DAU with billions of files and petabytes of data.
  • High availability with no single metadata server bottleneck.
📋
Guided Steps
1/5
Step 1
Metadata Storage Architecture
How would you design the metadata storage layer to scale for billions of files and ensure low-latency lookups?
💡 Hint
Consider distributed key-value stores optimized for range queries and high write throughput.
Key Decision
Use Apache Cassandra with file_path as partition key and file_version as clustering key - LSM-tree storage enables linear write scalability and efficient range scans for directory listings.
Interviewer Expectation
Mid: Names Cassandra for scalability. Senior: Explains why file_path as partition key enables co-located metadata and why relational DBs like MySQL fail at this scale due to write bottlenecks.
⚠️Common Wrong Answer
Using a single relational database for metadata - leads to write bottlenecks and high latency at billions of files.
Step 2
Metadata Consistency and Availability
How will you ensure metadata consistency and availability across multiple metadata servers?
💡 Hint
Think about consensus protocols and replication strategies that balance latency and fault tolerance.
Key Decision
Implement metadata replication using Apache Cassandra’s tunable consistency with QUORUM reads/writes and lightweight transactions (Paxos-based) for metadata updates to ensure strong consistency and availability.
Interviewer Expectation
Mid: Mentions replication and quorum. Senior: Details Cassandra’s tunable consistency levels and why Paxos-based lightweight transactions are needed for metadata atomicity.
⚠️Common Wrong Answer
Using eventual consistency without quorum - causes stale metadata reads and conflicts under concurrent updates.
Step 3
Data Node Design and Encryption
How do you design data nodes to store encrypted file chunks and support offline access?
💡 Hint
Consider chunking, encryption at rest, and local caching mechanisms.
Key Decision
Use Ceph OSDs for distributed object storage with client-side AES-256 encryption before upload; local client caches encrypted chunks with RocksDB for offline access and sync.
Interviewer Expectation
Mid: Names Ceph for storage. Senior: Explains client-side encryption to prevent server-side key exposure and RocksDB local cache for offline with sync conflict resolution.
⚠️Common Wrong Answer
Encrypting data only on server side - risks key exposure and complicates offline access.
Step 4
Metadata Server Scalability and Load Balancing
How do you scale metadata servers and balance load to maintain P99 < 200ms latency?
💡 Hint
Think about sharding, consistent hashing, and caching layers.
Key Decision
Shard metadata by file_path prefix using consistent hashing across Cassandra nodes; front with Redis caching layer for hot metadata entries to reduce read latency.
Interviewer Expectation
Mid: Mentions sharding metadata. Senior: Explains consistent hashing to avoid hotspots and Redis TTL caching to maintain low latency under burst traffic.
⚠️Common Wrong Answer
Using a single metadata server or naive hashing - causes hotspots and latency spikes.
Step 5
API Design for Metadata and Data Operations
Design 3 core API endpoints for clients interacting with metadata servers and data nodes.
💡 Hint
How does the client securely create or update file metadata?How does the client fetch metadata and file chunks efficiently?How does the client acknowledge successful writes or syncs?
Key Decision
Define endpoints: (1) PUT /files/{file_path} with encrypted metadata payload and version for atomic updates; (2) GET /files/{file_path} returns metadata and chunk locations; (3) POST /files/{file_path}/ack to confirm successful chunk upload and trigger metadata update.
Interviewer Expectation
Mid: Lists basic CRUD endpoints. Senior: Details versioning for concurrency control, encrypted payloads, and acknowledgement to ensure metadata-data consistency.
⚠️Common Wrong Answer
Designing stateless endpoints without versioning or acknowledgement - leads to race conditions and inconsistent metadata.
🔌
API Design Challenge
Sketch 3 core API endpoints for the distributed file system focusing on metadata and data node interactions.
Hints
  • How does the client create or update file metadata securely?
  • How does the client fetch file metadata and data efficiently?
  • How does the client acknowledge successful file writes or syncs?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Use of a distributed scalable metadata store like Apache Cassandra with appropriate partitioning.
  • Strong consistency mechanisms for metadata updates using Cassandra lightweight transactions.
  • Client-side encryption for data security and offline access support.
  • Sharding and caching strategies to meet P99 latency < 200ms.
Good to Have - 30%
  • Use of Ceph OSDs for distributed data node storage.
  • Local RocksDB caching on clients for offline file access.
  • Versioned API design with acknowledgements for metadata-data consistency.
Bonus - 10%
  • Detailed explanation of failure modes in metadata replication and recovery.
  • Discussion of conflict resolution strategies during offline sync.
Anti-patterns to Avoid
Using a single relational database for metadata causes write bottlenecks and high latency spikes at scale, making the system unscalable.
Eventual consistency without quorum leads to stale metadata reads and conflicts under concurrent updates, causing data inconsistency.
Encrypting data only on the server side exposes encryption keys to the server, risking data confidentiality and complicating offline access.
Naive hashing or no sharding causes metadata hotspots, resulting in degraded latency and potential server overload.
Stateless API design without versioning or acknowledgements causes race conditions and inconsistent metadata updates.
✅
Model Answer
▼ Reveal

The system uses Apache Cassandra for scalable, strongly consistent metadata storage with file_path-based sharding and Paxos-based lightweight transactions. Ceph OSDs store encrypted file chunks with client-side AES-256 encryption. Clients cache encrypted chunks locally using RocksDB for offline access and eventual sync. Metadata is sharded using consistent hashing to avoid hotspots, and Redis caching reduces read latency to meet P99 < 200ms SLA. API endpoints support versioned encrypted metadata payloads and acknowledgements to ensure atomic updates and consistency.

Key Decisions
  • Use Apache Cassandra with tunable consistency (QUORUM reads/writes) and Paxos-based lightweight transactions for strong metadata consistency and availability.
  • Shard metadata by file_path prefix using consistent hashing to distribute load evenly and avoid hotspots.
  • Encrypt file chunks client-side with AES-256 before upload to Ceph OSDs to protect encryption keys from server exposure.
  • Implement local RocksDB cache on clients to enable offline access and eventual sync with conflict resolution.
  • Use Redis caching with TTL for hot metadata entries to maintain P99 latency under 200ms during traffic bursts.
  • Design API endpoints with encrypted, versioned metadata payloads and acknowledgement POST calls to ensure atomic updates and metadata-data consistency.
  • Apache Cassandra for metadata storage with file_path partition key for linear scalability and efficient range scans.
  • Cassandra tunable consistency with QUORUM reads/writes and lightweight transactions for strong metadata consistency.
  • Ceph Object Storage Daemons (OSDs) for distributed encrypted file chunk storage with client-side AES-256 encryption.
  • Local RocksDB cache on client for offline encrypted chunk access and eventual sync.
  • Consistent hashing for metadata sharding across Cassandra nodes to avoid hotspots and maintain latency SLAs.
  • Redis caching layer with TTL for hot metadata entries to reduce P99 latency below 200ms.
  • API endpoints with versioned encrypted metadata payloads and acknowledgements to ensure atomic updates and consistency.
  • Client-side encryption to protect data confidentiality and simplify offline access security.

Practice

(1/5)
1. A system must match riders with nearby drivers in under 2 seconds while handling millions of concurrent requests. Which component is best suited to efficiently index and query driver locations in real-time?
easy
A. An in-memory spatial index like an R-tree or geohash-based Redis module
B. A relational database with geospatial extensions for location queries
C. A distributed log system like Kafka to stream location updates
D. A batch processing system like Hadoop to periodically update driver locations

Solution

  1. Step 1: Identify location query requirements

    Real-time matching requires low-latency spatial queries on dynamic driver locations.

  2. Step 2: Evaluate component suitability

    Relational DBs (A) have high latency for frequent updates; Kafka (C) streams data but doesn't index locations; Hadoop (D) is batch-oriented and too slow. In-memory spatial indexes (B) provide fast nearest-neighbor queries and frequent updates.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    In-memory spatial index enables fast real-time location queries [check]
Hint: Real-time location needs fast spatial index, not batch or logs [check]
Common Mistakes:
2. Without the segment manifest (playlist) update mechanism, what exactly breaks in the adaptive bitrate HLS streaming workflow?
medium
A. Clients will download all bitrate variants simultaneously, wasting bandwidth
B. Clients will fail to discover new segments, causing playback stalls or infinite buffering
C. Clients will continue playing old segments indefinitely without switching bitrates
D. Clients will switch bitrates too frequently, causing playback instability

Solution

  1. Step 1: Role of manifest updates in HLS

    Manifest updates inform clients about new segments and available bitrates, enabling selective fetching.

  2. Step 2: Consequence of missing manifest updates

    Without updates, clients cannot know which segments to fetch next, leading them to request all variants to avoid stalls, wasting bandwidth.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Missing manifest updates cause clients to fetch all variants, wasting bandwidth [check]
Hint: No manifest updates -> clients fetch all bitrates, wasting bandwidth [check]
Common Mistakes:
3. For a distributed cache requiring high availability and low latency, which replication strategy is best when network partitions are frequent?
medium
A. Asynchronous replication with eventual consistency
B. No replication, rely on client retries
C. Synchronous replication with strong consistency
D. Quorum-based replication with majority writes

Solution

  1. Step 1: Analyze replication under network partitions

    Synchronous replication blocks writes on slow or unreachable replicas, hurting availability.

  2. Step 2: Asynchronous replication tolerates partitions

    Allows writes to proceed locally and replicates changes later, favoring availability over immediate consistency.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Async replication favors availability during partitions [check]
Hint: Async replication favors availability under partitions [OK]
Common Mistakes:
4. Without the retry queue component in the payment gateway's webhook delivery system, what exact failure occurs when a merchant's endpoint is temporarily down?
medium
A. Webhook notifications are retried immediately in a tight loop, overwhelming the merchant server
B. Webhook delivery blocks the main payment processing thread until the merchant recovers
C. Webhook notifications are queued in memory indefinitely, causing memory leaks and crashes
D. Webhook notifications are lost permanently with no retry, causing inconsistent merchant state

Solution

  1. Step 1: Understand retry queue role

    The retry queue buffers failed webhook notifications for asynchronous retries, ensuring eventual delivery.

  2. Step 2: Consequence of missing retry queue

    Without it, failed webhooks are dropped or lost, causing permanent notification loss and inconsistent merchant state.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Missing retry queue -> permanent webhook loss on failure [check]
Hint: Retry queue prevents permanent webhook loss [check]
Common Mistakes:
5. For a high-traffic API, which approach better captures user experience degradation under load: reporting average latency or reporting P95 latency, and why?
medium
A. P95 latency, because it highlights the slowest 5% of requests affecting users
B. Average latency, because it is easier to compute in real-time
C. Average latency, because it summarizes all requests evenly
D. P95 latency, because it ignores outliers and focuses on median performance

Solution

  1. Step 1: Understand average latency limitations

    Average latency can be skewed by many fast requests and hide slow request impact, thus missing user experience degradation.

  2. Step 2: Why P95 latency is better

    P95 latency captures the slowest 5% of requests, which often cause user frustration and indicate performance issues under load.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    P95 latency reveals slow requests impacting users better than average [check]
Hint: P95 reveals slow requests; average hides tail latency [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard