Distributed file systems in HLD - System Design Exercise

Design: Distributed File System

Design covers file storage, metadata management, replication, and client access. Does not cover user authentication or network security in detail.

Functional Requirements

FR1: Store and manage files across multiple machines

FR2: Allow concurrent access to files by multiple users

FR3: Provide fault tolerance and data replication

FR4: Support large files and high throughput

FR5: Ensure data consistency and integrity

FR6: Allow easy file metadata management (e.g., file names, permissions)

FR7: Provide fast file read and write operations

FR8: Support scalability to thousands of nodes and millions of files

Non-Functional Requirements

NFR1: System should handle up to 10,000 concurrent clients

NFR2: File read/write latency p99 should be under 200ms

NFR3: Availability target of 99.9% uptime (less than 8.77 hours downtime per year)

NFR4: Data replication factor of at least 3 for fault tolerance

NFR5: Support files up to several terabytes in size

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

❓ Question 6

Key Components

Metadata server or service

Data storage nodes

Replication manager

Client library or interface

Failure detection and recovery system

Cache layer for metadata and data

Design Patterns

Master-slave or distributed metadata management

Data sharding and partitioning

Replication and consensus algorithms (e.g., Paxos, Raft)

Caching for read performance

Eventual consistency vs strong consistency

Reference Architecture

                +---------------------+
                |     Clients          |
                +----------+----------+
                           |
                           v
                +---------------------+
                |  Metadata Server(s)  |<----+
                +----------+----------+     |
                           |                |
          +----------------+----------------+----------------+
          |                |                |                |
+---------v--------+ +-----v-------+ +------v-------+ +------v-------+
| Data Node 1      | | Data Node 2 | | Data Node 3  | | Data Node N  |
+------------------+ +-------------+ +--------------+ +--------------+

Legend:
- Clients request file operations
- Metadata Server manages file info and locations
- Data Nodes store actual file chunks
- Replication ensures copies on multiple Data Nodes

Components

Metadata Server

Distributed consensus system (e.g., Raft)

Stores file metadata, directory structure, and file-to-chunk mappings

Data Nodes

Distributed storage servers

Store actual file chunks and handle read/write requests

Replication Manager

Replication protocol with consensus

Ensure data chunks are replicated across multiple Data Nodes for fault tolerance

Client Library

API or SDK

Interface for clients to interact with the distributed file system transparently

Failure Detector

Heartbeat and monitoring system

Detect failed nodes and trigger recovery or re-replication

Cache Layer

In-memory cache (e.g., Redis or local client cache)

Speed up metadata and data access for frequent operations

Request Flow

1. Client requests file metadata from Metadata Server.

2. Metadata Server returns file location and chunk info.

3. Client requests file chunks directly from Data Nodes.

4. Data Nodes serve file chunks to client.

5. For writes, client sends data to Data Nodes; Replication Manager ensures copies are stored on multiple nodes.

6. Metadata Server updates metadata after successful writes.

7. Failure Detector monitors nodes and triggers re-replication if a Data Node fails.

Database Schema

Entities: - File: id (PK), name, size, permissions, creation_date - Directory: id (PK), name, parent_directory_id (FK) - Chunk: id (PK), file_id (FK), chunk_index, size - ChunkLocation: chunk_id (FK), data_node_id (FK), replica_index - DataNode: id (PK), address, status Relationships: - Directory has many Files and Directories (self-referencing) - File has many Chunks - Chunk has many ChunkLocations (replicas) - DataNode stores many ChunkLocations

Scaling Discussion

Bottlenecks

Metadata Server becomes a single point of failure or bottleneck under heavy load

Network bandwidth limits data transfer between clients and Data Nodes

Replication overhead increases with number of replicas and file size

Failure detection and recovery can be slow with many nodes

Client latency increases with large file sizes and many chunks

Solutions

Use multiple Metadata Servers with sharding or distributed consensus to scale metadata management

Implement data locality and client caching to reduce network load

Optimize replication with asynchronous replication and selective replication policies

Use efficient heartbeat protocols and parallel recovery processes

Support parallel chunk downloads and uploads to improve client throughput

Interview Tips

Time: Spend 10 minutes clarifying requirements and constraints, 20 minutes designing architecture and data flow, 10 minutes discussing scaling and trade-offs, 5 minutes for questions.

Explain how metadata and data are separated and managed

Discuss replication strategy and fault tolerance

Describe how clients interact with the system

Highlight how consistency and concurrency are handled

Address scaling challenges and solutions

Mention trade-offs between consistency, availability, and performance