Bird
Raised Fist0

Distributed File System Design - Metadata Servers and Data Nodes

HLDmediumInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🏗️
Distributed File System Design - Metadata Servers and Data Nodes
Design a scalable distributed file system with metadata servers and data nodes for efficient file storage and retrieval at large scale.
mediumStorage SystemsFreq: high
Prerequisites:System Design FundamentalsCAP TheoremDistributed Systems BasicsStorage Systems Concepts
⚙️
Functional Requirements
  • Support file creation, deletion, and renaming with atomic metadata updates.
  • Efficiently store and retrieve file metadata (file size, permissions, block locations).
  • Distribute file data across multiple data nodes with replication for durability.
  • Provide strong consistency for metadata operations to avoid stale reads.
  • Support concurrent client access with locking or lease mechanisms.
  • Enable scalable metadata management to handle billions of files.
📊
Non-Functional Requirements
  • System must support at least 2 billion files (objects) stored.
  • Metadata read/write latency P99 under 100ms.
  • Metadata servers availability of 99.99% to avoid system-wide downtime.
  • Data replication factor of 3 for durability and fault tolerance.
  • System should handle peak write QPS of approximately 1 million metadata ops/sec.
❓
Clarifying Questions to Ask
  • What is the expected average file size and typical file size distribution?
  • Are metadata operations strongly consistent or eventually consistent?
  • What is the expected read-to-write ratio for metadata operations?
  • Is the system expected to support multi-region deployment or geo-replication?
  • What is the retention period for metadata and file data?
  • Are there any SLAs for data durability or recovery time objectives?
Out of Scope
✗ Design of client-side caching or offline access.
✗ Detailed data node storage internals (e.g., block placement algorithms).
✗ Network-level optimizations or physical hardware design.
✗ Security features like encryption or authentication.
📌
Assumptions
  • 2 billion files stored in the system.
  • Each file has on average 40 metadata operations per day (create, read, update).
  • Average metadata operation size is 200 bytes (including file attributes and block pointers).
  • Replication factor for data blocks is 3.
⚡
QPS Calculation
write qps2B files × 40 ops/day ÷ 86400 sec/day ≈ 925,926 ops/sec
peak write qpsAssuming 2× peak factor: 925,926 × 2 ≈ 1,851,852 ops/sec
read qpsAssuming read-to-write ratio 3:1, read_qps = 3 × 925,926 ≈ 2,777,778 ops/sec
💾
Storage Estimate
daily metadata storage2B files × 40 ops × 200 bytes = 16 TB/day
retention periodMetadata retained for 30 days
total metadata storage16 TB/day × 30 days = 480 TB
with replication480 TB × 3 (replication factor) = 1.44 PB
🖥️
Server Estimate
metadata serversAssuming each metadata server handles 50K ops/sec, need 1,851,852 ÷ 50,000 ≈ 38 metadata servers
data nodesAssuming each data node stores 100 TB, need 1.44 PB ÷ 100 TB = 15 data nodes
🏗️
Architecture Diagram
Architecture Diagram
          +----------------------+
          |      Clients         |
          +----------+-----------+
                     |
                     v
          +----------+-----------+
          |   Load Balancer      |
          +----------+-----------+
                     |
         +-----------+------------+
         |                        |
 +-------v-------+        +-------v-------+
 | Metadata Srv1 |  ...   | Metadata SrvN |
 +-------+-------+        +-------+-------+
         |                        |
         +-----------+------------+
                     |
          +----------+-----------+
          |   Metadata Store     |
          | (e.g., Apache Cassandra) |
          +----------+-----------+
                     |
         +-----------+------------+
         |                        |
 +-------v-------+        +-------v-------+
 |   Data Node1  |  ...   |   Data NodeM  |
 +---------------+        +---------------+
🧩
Components
Clients
Issue file system operations (create, read, write, delete) to the system.
Why thisClients interact with metadata servers for metadata and data nodes for file data, enabling separation of concerns.
Why notDirect client access to data nodes without metadata coordination leads to inconsistent state and complex client logic.
Load Balancer
Distributes client requests evenly across metadata servers to balance load and provide fault tolerance.
Why thisUsing HAProxy or Envoy provides mature, scalable TCP/HTTP load balancing with health checks.
Why notDNS round-robin lacks health checks and fine-grained load distribution, causing hotspots.
Metadata Servers
Handle metadata operations, maintain file namespace, file attributes, and block mappings.
Why thisMetadata servers use Apache Cassandra for scalable, highly available, and partition-tolerant metadata storage with tunable consistency.
Why notRelational DBs like PostgreSQL require manual sharding and do not scale linearly to millions of ops/sec.
Metadata Store (Apache Cassandra)
Stores metadata persistently with replication and supports fast lookups and updates.
Why thisCassandra's distributed architecture supports linear scalability, multi-datacenter replication, and tunable consistency levels.
Why notHDFS NameNode's single point of failure and limited scalability make it unsuitable for 2B files and 1M ops/sec.
Data Nodes
Store actual file data blocks, replicate data for durability, and serve read/write requests.
Why thisData nodes run on commodity servers with local disks, using protocols like gRPC for block transfer and replication.
Why notUsing object stores like S3 directly for data blocks adds latency and reduces control over replication and locality.
🔄
Data Flow
1
Client → Load Balancer
Client sends file operation request (e.g., create file).
Load balancer evenly distributes requests to metadata servers.
2
Load Balancer → Metadata Server
Metadata server receives request, checks namespace and permissions.
Metadata server coordinates metadata updates atomically.
3
Metadata Server → Metadata Store (Cassandra)
Metadata server reads/writes file metadata and block locations.
Cassandra provides scalable, replicated metadata storage.
4
Metadata Server → Data Nodes
Metadata server instructs data nodes to store or retrieve file blocks.
Data nodes handle actual file data replication and storage.
5
Data Nodes → Client
Data nodes send requested file blocks to client.
Clients receive file data after metadata coordination.

Practice

(1/5)
1. In a request trace, at step 4 the latency monitoring component records the request's completion time. If the request took 120ms but the P95 latency threshold is 100ms, what state does the monitoring component mark for this request?
easy
A. Request is ignored because it is below P99 latency
B. Request is marked as within acceptable latency
C. Request is marked as exceeding P95 latency threshold
D. Request is marked as failed due to timeout

Solution

  1. Step 1: Compare request latency to P95 threshold

    The request latency (120ms) exceeds the P95 threshold (100ms), so it is in the slowest 5% tail.

  2. Step 2: Monitoring component's marking logic

    Requests exceeding P95 latency are flagged to identify performance degradation; it is not ignored or marked as failed unless timeout occurs.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    120ms > 100ms P95 -> marked as exceeding P95 latency [check]
Hint: Latency > P95 threshold -> mark as slow request [OK]
Common Mistakes:
2. A messaging system must decouple high-throughput message producers from multiple independent consumers subscribing to different topics. Which component best handles this decoupling and ensures message durability?
easy
A. Use a distributed log-based system like Kafka with topic partitions
B. Use a simple in-memory queue on the producer server to buffer messages
C. Use a direct HTTP push from producer to each consumer
D. Use a relational database with polling by consumers

Solution

  1. Step 1: Identify decoupling and durability needs

    In-memory queues (Use a simple in-memory queue on the producer server to buffer messages) lack durability and don't scale across servers. Direct HTTP push (Use a direct HTTP push from producer to each consumer) tightly couples producer and consumers, causing backpressure issues. Polling a relational DB (Use a relational database with polling by consumers) adds latency and load on DB, unsuitable for high throughput.

  2. Step 2: Recognize Kafka's role

    Kafka's distributed log with topic partitions decouples producers and consumers, provides durability, and supports high throughput with consumer groups.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Kafka partitions enable durable, scalable decoupling [check]
Hint: Distributed logs decouple producers and consumers reliably [OK]
Common Mistakes:
3. For an API Gateway that must support both REST and gRPC clients, which architecture best balances protocol support and transformation complexity?
medium
A. Use gRPC-only API Gateway and require REST clients to upgrade to gRPC
B. Use separate API Gateways for REST and gRPC, each optimized for its protocol
C. Use REST-only API Gateway and require gRPC clients to use a proxy converting gRPC to REST
D. Use a single API Gateway that performs protocol translation and transformation internally

Solution

  1. Step 1: Analyze protocol support requirements

    Supporting both REST and gRPC clients in one gateway reduces operational complexity and centralizes transformation logic.

  2. Step 2: Evaluate alternatives

    Use separate API Gateways for REST and gRPC, each optimized for its protocol increases deployment and maintenance overhead. Use REST-only API Gateway and require gRPC clients to use a proxy converting gRPC to REST adds latency and complexity on client side. Use gRPC-only API Gateway and require REST clients to upgrade to gRPC is impractical as REST clients are widespread.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Single gateway with protocol translation balances support and complexity [check]
Hint: One gateway handles both protocols with internal translation [check]
Common Mistakes:
4. For intent classification in the bot, which architecture best balances latency and model update frequency given a high volume of diverse user queries?
medium
A. Use a single large LLM instance serving all queries synchronously
B. Run multiple LLM replicas behind a load balancer with synchronous inference
C. Deploy a lightweight intent classifier for fast routing, with LLM fallback for complex queries asynchronously
D. Use batch processing of queries offline to update intents periodically

Solution

  1. Step 1: Analyze latency and update frequency constraints

    High volume requires low latency for common intents and flexibility for complex queries.

  2. Step 2: Evaluate architectures

    A single large LLM (A) has high latency and update overhead. Multiple replicas (C) improve throughput but still synchronous latency is high. Batch offline (D) delays updates and user experience. Lightweight classifier with async LLM fallback (B) balances fast routing and complex understanding.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Lightweight classifier + async LLM fallback balances latency and accuracy [check]
Hint: Fast lightweight routing, async LLM fallback for complexity [check]
Common Mistakes:
5. At 10x scale, which component in the blob store architecture is most likely to fail first due to stateful connection limits, and why?
hard
A. Front-end API servers, because they maintain stateful client sessions and cannot easily shard connections
B. Storage nodes, because disk I/O bandwidth saturates with large blob volumes
C. Metadata service, because it cannot shard metadata fast enough across nodes
D. Distributed log system, because partition leaders become overloaded with write requests

Solution

  1. Step 1: Identify stateful bottleneck at front-end API servers

    API servers maintain stateful client sessions (e.g., HTTP keep-alive or TLS), limiting horizontal scaling due to connection affinity.

  2. Step 2: Eliminate other options

    Storage nodes scale with added disks; metadata service shards metadata; distributed log partitions scale with more brokers.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Stateful client sessions limit API server horizontal scaling [check]
Hint: Stateful API servers limit scaling due to connection affinity [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard