Distributed file systems in HLD - System Design Exercise

Design: Distributed File System

Design covers file storage, metadata management, replication, and client access. Does not cover user authentication or network security in detail.

Functional Requirements

FR1: Store and manage files across multiple machines

FR2: Allow concurrent access to files by multiple users

FR3: Provide fault tolerance and data replication

FR4: Support large files and high throughput

FR5: Ensure data consistency and integrity

FR6: Allow easy file metadata management (e.g., file names, permissions)

FR7: Provide fast file read and write operations

FR8: Support scalability to thousands of nodes and millions of files

Non-Functional Requirements

NFR1: System should handle up to 10,000 concurrent clients

NFR2: File read/write latency p99 should be under 200ms

NFR3: Availability target of 99.9% uptime (less than 8.77 hours downtime per year)

NFR4: Data replication factor of at least 3 for fault tolerance

NFR5: Support files up to several terabytes in size

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

❓ Question 6

Key Components

Metadata server or service

Data storage nodes

Replication manager

Client library or interface

Failure detection and recovery system

Cache layer for metadata and data

Design Patterns

Master-slave or distributed metadata management

Data sharding and partitioning

Replication and consensus algorithms (e.g., Paxos, Raft)

Caching for read performance

Eventual consistency vs strong consistency

Reference Architecture

                +---------------------+
                |     Clients          |
                +----------+----------+
                           |
                           v
                +---------------------+
                |  Metadata Server(s)  |<----+
                +----------+----------+     |
                           |                |
          +----------------+----------------+----------------+
          |                |                |                |
+---------v--------+ +-----v-------+ +------v-------+ +------v-------+
| Data Node 1      | | Data Node 2 | | Data Node 3  | | Data Node N  |
+------------------+ +-------------+ +--------------+ +--------------+

Legend:
- Clients request file operations
- Metadata Server manages file info and locations
- Data Nodes store actual file chunks
- Replication ensures copies on multiple Data Nodes

Components

Metadata Server

Distributed consensus system (e.g., Raft)

Stores file metadata, directory structure, and file-to-chunk mappings

Data Nodes

Distributed storage servers

Store actual file chunks and handle read/write requests

Replication Manager

Replication protocol with consensus

Ensure data chunks are replicated across multiple Data Nodes for fault tolerance

Client Library

API or SDK

Interface for clients to interact with the distributed file system transparently

Failure Detector

Heartbeat and monitoring system

Detect failed nodes and trigger recovery or re-replication

Cache Layer

In-memory cache (e.g., Redis or local client cache)

Speed up metadata and data access for frequent operations

Request Flow

1. Client requests file metadata from Metadata Server.

2. Metadata Server returns file location and chunk info.

3. Client requests file chunks directly from Data Nodes.

4. Data Nodes serve file chunks to client.

5. For writes, client sends data to Data Nodes; Replication Manager ensures copies are stored on multiple nodes.

6. Metadata Server updates metadata after successful writes.

7. Failure Detector monitors nodes and triggers re-replication if a Data Node fails.

Database Schema

Entities: - File: id (PK), name, size, permissions, creation_date - Directory: id (PK), name, parent_directory_id (FK) - Chunk: id (PK), file_id (FK), chunk_index, size - ChunkLocation: chunk_id (FK), data_node_id (FK), replica_index - DataNode: id (PK), address, status Relationships: - Directory has many Files and Directories (self-referencing) - File has many Chunks - Chunk has many ChunkLocations (replicas) - DataNode stores many ChunkLocations

Scaling Discussion

Bottlenecks

Metadata Server becomes a single point of failure or bottleneck under heavy load

Network bandwidth limits data transfer between clients and Data Nodes

Replication overhead increases with number of replicas and file size

Failure detection and recovery can be slow with many nodes

Client latency increases with large file sizes and many chunks

Solutions

Use multiple Metadata Servers with sharding or distributed consensus to scale metadata management

Implement data locality and client caching to reduce network load

Optimize replication with asynchronous replication and selective replication policies

Use efficient heartbeat protocols and parallel recovery processes

Support parallel chunk downloads and uploads to improve client throughput

Interview Tips

Time: Spend 10 minutes clarifying requirements and constraints, 20 minutes designing architecture and data flow, 10 minutes discussing scaling and trade-offs, 5 minutes for questions.

Explain how metadata and data are separated and managed

Discuss replication strategy and fault tolerance

Describe how clients interact with the system

Highlight how consistency and concurrency are handled

Address scaling challenges and solutions

Mention trade-offs between consistency, availability, and performance

Practice

1. What is the main role of a metadata server in a distributed file system?

easy

A. To keep track of file locations and directory structure

B. To store the actual file data

C. To provide network connectivity between clients

D. To encrypt files before storage

Solution

1. Step 1: Understand metadata server function

   The metadata server manages information about where files are stored and their organization.

2. Step 2: Differentiate from storage nodes

   Storage nodes hold the actual file data, not metadata servers.

3. Final Answer:

   To keep track of file locations and directory structure -> Option A

4. Quick Check:

   Metadata server = file info tracker [OK]

Hint: Metadata servers track file info, not data storage [OK]

Common Mistakes:

• Confusing metadata server with storage node
• Thinking metadata server stores file data
• Assuming metadata server handles encryption

2. Which of the following is the correct sequence when a client reads a file in a distributed file system?

easy

A. Client contacts only the storage node

B. Client contacts storage node, then metadata server

C. Client contacts only the metadata server

D. Client contacts metadata server, then storage node

Solution

1. Step 1: Identify client interaction flow

   The client first asks the metadata server for file location details.

2. Step 2: Access data from storage node

   After getting location info, the client reads the actual file data from the storage node.

3. Final Answer:

   Client contacts metadata server, then storage node -> Option D

4. Quick Check:

   Read flow = metadata server then storage node [OK]

Hint: Client asks metadata first, then reads data [OK]

Common Mistakes:

• Reversing the order of contacting servers
• Assuming client contacts only one server type
• Ignoring metadata server role in locating files

3. Consider a distributed file system where a file is split into 3 chunks stored on different storage nodes. If the metadata server returns chunk locations as [Node1, Node2, Node3], what happens when the client requests the file?

medium

A. Client reads all chunks sequentially from Node1 only

B. Client reads chunks in parallel from Node1, Node2, and Node3

C. Client reads chunks randomly from any node

D. Client reads only the first chunk from Node1

Solution

1. Step 1: Understand chunk distribution

   The file is divided into chunks stored on separate nodes for parallel access.

2. Step 2: Client reads chunks efficiently

   To speed up reading, the client fetches all chunks at the same time from their respective nodes.

3. Final Answer:

   Client reads chunks in parallel from Node1, Node2, and Node3 -> Option B

4. Quick Check:

   Parallel chunk reads = faster file access [OK]

Hint: Chunks read in parallel from all nodes [OK]

Common Mistakes:

• Assuming client reads all chunks from one node
• Thinking chunks are read randomly
• Ignoring parallelism benefits

4. A distributed file system client tries to read a file but gets an error because the metadata server is unreachable. What is the most likely cause?

medium

A. Client has no network access to metadata server

B. Storage nodes are down

C. File data is corrupted on storage nodes

D. Client requested a non-existent file chunk

Solution

1. Step 1: Identify error source

   The error is due to inability to reach the metadata server, not storage nodes.

2. Step 2: Analyze network access

   Most likely, the client cannot connect to the metadata server because of network issues or firewall rules.

3. Final Answer:

   Client has no network access to metadata server -> Option A

4. Quick Check:

   Metadata server unreachable = network issue [OK]

Hint: Metadata server unreachable means network problem [OK]

Common Mistakes:

• Blaming storage nodes for metadata server errors
• Assuming file corruption causes metadata errors
• Confusing chunk existence with server reachability

5. You are designing a distributed file system for a company with many users reading large files frequently but rarely writing. Which design choice improves read performance the most?

hard

A. Disable caching to ensure fresh data on every read

B. Store all file chunks on a single storage node to simplify access

C. Use multiple metadata servers with caching to reduce lookup latency

D. Limit the number of storage nodes to reduce network overhead

Solution

1. Step 1: Analyze workload pattern

   Many reads and few writes mean caching metadata can speed up file location lookups.

2. Step 2: Evaluate design options

   Multiple metadata servers with caching reduce latency and avoid bottlenecks, improving read speed.

3. Step 3: Reject other options

   Storing all chunks on one node limits scalability; disabling caching slows reads; fewer nodes reduce parallelism.

4. Final Answer:

   Use multiple metadata servers with caching to reduce lookup latency -> Option C

5. Quick Check:

   Metadata caching + multiple servers = faster reads [OK]

Hint: Cache metadata and use multiple servers for fast reads [OK]

Common Mistakes:

• Putting all data on one node limits scalability
• Disabling caching hurts read performance
• Reducing nodes lowers parallel access