Blob storage (S3, Azure Blob) in HLD - Architecture Diagram

System Overview - Blob storage (S3, Azure Blob)

This system stores and manages large amounts of unstructured data like images, videos, and documents. It provides scalable, durable, and secure storage accessible over the internet with high availability and low latency.

Architecture Diagram

User
  |
  v
Internet CDN
  |
  v
Load Balancer
  |
  v
API Gateway
  |
  v
Blob Storage Service
  |
  +------------------+
  |                  |
  v                  v
Metadata DB       Object Storage
(Cache Layer)     (Durable Storage)

Components

User

client

End user or application accessing blob storage

Internet CDN

cdn

Caches frequently accessed blobs close to users for faster delivery

Load Balancer

load_balancer

Distributes incoming requests evenly to backend services

API Gateway

api_gateway

Handles authentication, authorization, and routes requests to blob storage service

Blob Storage Service

service

Manages blob upload, download, and metadata operations

Metadata DB (Cache Layer)

cache

Stores metadata like blob names, sizes, and access permissions for quick lookup

Object Storage (Durable Storage)

storage

Stores the actual blob data reliably and durably across multiple servers

Request Flow - 10 Hops

User → Internet CDN

Internet CDN → Load Balancer

Load Balancer → API Gateway

API Gateway → Blob Storage Service

Blob Storage Service → Metadata DB (Cache Layer)

Blob Storage Service → Object Storage (Durable Storage)

Blob Storage Service → API Gateway

API Gateway → Load Balancer

Load Balancer → Internet CDN

Internet CDN → User

Failure Scenario

Component Fails:Object Storage (Durable Storage)

Impact:Blob data cannot be read or written, causing failures in upload/download operations

Mitigation:Use data replication across multiple storage nodes and regions to ensure availability and durability

Architecture Quiz - 3 Questions

Which component is responsible for authenticating user requests?

A API Gateway

B Load Balancer

C Metadata DB

D Internet CDN

Design Principle

This architecture uses a layered approach with caching (CDN and metadata cache) to reduce latency and load on durable storage. It separates concerns by using an API Gateway for security and routing, and durable object storage for reliable data persistence. Load balancing ensures scalability and availability.

Practice

1. What is the main purpose of blob storage services like Amazon S3 or Azure Blob Storage?

easy

A. To manage user identities and permissions

B. To run virtual machines in the cloud

C. To store large files safely and accessibly in the cloud

D. To host websites directly without any storage

Solution

1. Step 1: Understand blob storage function

   Blob storage is designed to save large files like videos, images, and backups in the cloud.

2. Step 2: Compare with other cloud services

   Other services like virtual machines or identity management serve different purposes, not file storage.

3. Final Answer:

   To store large files safely and accessibly in the cloud -> Option C

4. Quick Check:

   Blob storage = large file storage [OK]

Hint: Blob storage is for big files, not computing or identity [OK]

Common Mistakes:

• Confusing blob storage with compute services
• Thinking blob storage manages users
• Assuming blob storage hosts websites directly

2. Which of the following is the correct way to specify a bucket name in Amazon S3?

easy

A. My.bucket.name

B. MyBucketName!

C. my_bucket_name

D. my-bucket-name

Solution

1. Step 1: Review S3 bucket naming rules

   S3 bucket names must be lowercase, can include hyphens, but no underscores or special characters like '!'.

2. Step 2: Check each option

   my-bucket-name uses lowercase letters and hyphens, which is valid. Options A, B, and C are invalid: A has uppercase letters and dots, B has uppercase and '!', C has underscores.

3. Final Answer:

   my-bucket-name -> Option D

4. Quick Check:

   Bucket names use lowercase and hyphens [OK]

Hint: Bucket names: lowercase letters and hyphens only [OK]

Common Mistakes:

• Using uppercase letters in bucket names
• Including underscores or special characters
• Using dots in bucket names

3. Given this AWS CLI command:
aws s3 cp file.txt s3://my-bucket/ --storage-class STANDARD_IA
What does the --storage-class STANDARD_IA option do?

medium

A. Uploads the file to a cheaper, infrequently accessed storage tier

B. Encrypts the file automatically during upload

C. Makes the file publicly accessible

D. Uploads the file with maximum redundancy across regions

Solution

1. Step 1: Understand storage classes in S3

   STANDARD_IA means Standard-Infrequent Access, a cheaper tier for files accessed less often.

2. Step 2: Analyze command effect

   The command uploads file.txt to the bucket using this storage class, saving cost for infrequent access.

3. Final Answer:

   Uploads the file to a cheaper, infrequently accessed storage tier -> Option A

4. Quick Check:

   STANDARD_IA = cheaper, infrequent access [OK]

Hint: STANDARD_IA means cheaper storage for rare access [OK]

Common Mistakes:

• Thinking it encrypts the file
• Assuming it makes file public
• Confusing with multi-region redundancy

4. You try to upload a file to Azure Blob Storage but get an error saying the container name is invalid. Which of these is the most likely cause?

medium

A. The container name contains uppercase letters

B. The file size is too large

C. The storage account is in the wrong region

D. The file name has spaces

Solution

1. Step 1: Recall Azure container naming rules

   Container names must be lowercase letters and numbers only, no uppercase letters allowed.

2. Step 2: Match error cause

   An invalid container name error usually means the name breaks these rules, such as having uppercase letters.

3. Final Answer:

   The container name contains uppercase letters -> Option A

4. Quick Check:

   Azure container names must be lowercase [OK]

Hint: Azure containers must be lowercase, no uppercase letters [OK]

Common Mistakes:

• Blaming file size for container name errors
• Confusing region with naming rules
• Thinking file name affects container name

5. You want to store backup files in Amazon S3 with minimal cost but still need quick access sometimes. Which combination of settings is best?

hard

A. Use STANDARD storage class and enable public access for easy retrieval

B. Use STANDARD_IA storage class with lifecycle rule to move files to GLACIER after 30 days

C. Use GLACIER storage class for all files with no lifecycle rules

D. Use REDUCED_REDUNDANCY storage class and no lifecycle rules

Solution

1. Step 1: Understand storage classes and lifecycle

   STANDARD_IA offers cheaper storage with quick access. GLACIER is very cheap but slow to retrieve.

2. Step 2: Analyze cost and access needs

   Using STANDARD_IA first allows quick access. Moving to GLACIER after 30 days saves cost for older backups.

3. Step 3: Evaluate other options

   GLACIER only delays access too much. STANDARD is costly. REDUCED_REDUNDANCY is deprecated and less reliable.

4. Final Answer:

   Use STANDARD_IA storage class with lifecycle rule to move files to GLACIER after 30 days -> Option B

5. Quick Check:

   STANDARD_IA + lifecycle to GLACIER = cost + access balance [OK]

Hint: Start with STANDARD_IA, then archive to GLACIER for cost savings [OK]

Common Mistakes:

• Choosing GLACIER only and ignoring access speed
• Enabling public access unnecessarily
• Using deprecated REDUCED_REDUNDANCY class