Blob storage (S3, Azure Blob) in HLD - Deep Dive

Overview - Blob storage (S3, Azure Blob)

What is it?

Blob storage is a way to store large amounts of unstructured data like images, videos, documents, or backups. It organizes data as objects called blobs, which can be accessed over the internet. Services like Amazon S3 and Azure Blob Storage provide scalable, durable, and secure storage solutions for these blobs. Users can upload, download, and manage blobs using simple APIs.

Why it matters

Without blob storage, storing and managing large files would be slow, unreliable, and expensive. Traditional file systems or databases struggle with scale and performance for big data. Blob storage solves this by offering a simple, scalable, and cost-effective way to store any amount of data accessible from anywhere. This enables cloud apps, backups, media streaming, and data lakes to work efficiently.

Where it fits

Before learning blob storage, you should understand basic storage concepts like files, databases, and cloud computing. After this, you can explore related topics like content delivery networks (CDNs), data lifecycle management, and distributed file systems. Blob storage is a foundational building block for cloud-native architectures and big data solutions.

Mental Model

Core Idea

Blob storage is like a giant, organized warehouse where each item (blob) is stored in a labeled box (container/bucket) and can be quickly found and retrieved over the internet.

Think of it like...

Imagine a massive library where each book is a blob. The library shelves are containers or buckets, and each book has a unique label (key). You can ask the librarian (API) to fetch, add, or remove any book quickly without searching the entire library.

┌─────────────────────────────┐
│        Blob Storage         │
│ ┌─────────────┐            │
│ │  Bucket/    │            │
│ │ Container 1 │            │
│ │ ┌─────────┐ │            │
│ │ │ Blob A  │ │            │
│ │ ├─────────┤ │            │
│ │ │ Blob B  │ │            │
│ │ └─────────┘ │            │
│ ├─────────────┤            │
│ │  Bucket/    │            │
│ │ Container 2 │            │
│ │ ┌─────────┐ │            │
│ │ │ Blob C  │ │            │
│ │ └─────────┘ │            │
│ └─────────────┘            │
└─────────────────────────────┘

Build-Up - 7 Steps

1

Foundation

What is Blob Storage?

Concept: Introduce the basic idea of blob storage as a way to store unstructured data as objects.

Blob storage stores data as blobs (binary large objects). Unlike files on your computer, blobs are stored in containers or buckets. Each blob has a unique name (key) and can be any size or type of data, like photos, videos, or backups. You access blobs via simple web APIs.

Result

You understand that blob storage is a cloud service for storing large, unstructured files accessible over the internet.

Understanding that blob storage treats data as objects rather than files or blocks is key to grasping its flexibility and scalability.

2

Foundation

Buckets and Containers Explained

Concept: Explain how blobs are grouped inside buckets or containers for organization and access control.

Buckets (S3) or containers (Azure Blob) are like folders that hold blobs. They provide a namespace so blob names are unique within them. Buckets also help manage permissions, lifecycle policies, and billing. You create buckets first, then upload blobs inside them.

Result

You can visualize how blobs are organized and managed inside buckets or containers.

Knowing that buckets are the main organizational unit helps you design storage layouts and control access effectively.

3

Intermediate

Blob Types and Access Patterns

🤔Before reading on: do you think all blobs are stored and accessed the same way? Commit to your answer.

Concept: Introduce different blob types and how access patterns affect storage choices.

Blob storage supports types like block blobs (large files uploaded in parts), append blobs (for logs), and page blobs (random read/write, used for disks). Access can be public, private, or via signed URLs. Choosing the right blob type and access method optimizes performance and cost.

Result

You understand that blob storage is flexible to support different data types and usage scenarios.

Recognizing blob types and access patterns helps you optimize storage for your application's needs and avoid unnecessary costs.

4

Intermediate

Durability and Replication Strategies

🤔Before reading on: do you think blob storage keeps only one copy of your data? Commit to yes or no.

Concept: Explain how blob storage ensures data durability and availability through replication.

Blob storage replicates data across multiple servers and sometimes across regions. Common strategies include locally redundant storage (copies in one data center), zone-redundant (across data centers), and geo-redundant (across regions). This protects data from hardware failures and disasters.

Result

You know how blob storage keeps your data safe and available even if parts of the system fail.

Understanding replication strategies is crucial for designing reliable systems and meeting data durability requirements.

5

Intermediate

Security and Access Control Mechanisms

🤔Before reading on: do you think blob storage data is always public by default? Commit to yes or no.

Concept: Describe how blob storage secures data using authentication, authorization, and encryption.

Blob storage uses access keys, IAM roles, and signed URLs to control who can read or write blobs. Data is encrypted at rest and in transit. You can set bucket policies or container ACLs to restrict access. Auditing logs track usage for compliance.

Result

You understand how to protect your data and control access securely in blob storage.

Knowing security features helps prevent data leaks and ensures compliance with regulations.

6

Advanced

Performance Optimization and Caching

🤔Before reading on: do you think blob storage automatically caches data for faster access? Commit to yes or no.

Concept: Explore how to improve blob storage performance using caching and tuning.

Blob storage itself is highly scalable but can have latency. Using CDNs caches blobs closer to users for faster reads. You can tune blob size, parallel uploads, and request patterns to optimize throughput. Understanding eventual consistency helps design around delays.

Result

You can design systems that deliver blob data quickly and efficiently at scale.

Knowing performance tuning and caching strategies prevents bottlenecks and improves user experience.

7

Expert

Internals of Blob Storage Architecture

🤔Before reading on: do you think blob storage stores blobs as single files on disk? Commit to yes or no.

Concept: Reveal how blob storage systems manage data internally for scalability and durability.

Blob storage breaks large blobs into smaller chunks stored across distributed servers. Metadata tracks blob parts and versions. Systems use consensus protocols to maintain consistency and replication. Data is stored on commodity hardware with error correction. This design balances cost, scale, and reliability.

Result

You gain deep insight into how blob storage works behind the scenes to serve billions of requests reliably.

Understanding internal architecture helps troubleshoot, optimize, and innovate on blob storage solutions.

Under the Hood

Blob storage systems split large files into smaller blocks or pages, each stored on distributed servers. Metadata services keep track of blob composition, versions, and locations. Replication protocols copy data across nodes or regions to ensure durability. Access requests go through front-end servers that authenticate, authorize, and route them to the correct storage nodes. Data is encrypted and checksummed to detect corruption.

Why designed this way?

This design allows blob storage to scale massively while keeping costs low by using commodity hardware. Splitting blobs enables parallel uploads and downloads, improving performance. Replication and consensus protocols ensure data safety despite hardware failures or network issues. Alternatives like monolithic file systems or databases were too slow or expensive at cloud scale.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│ Client/API    │──────▶│ Front-end     │──────▶│ Metadata      │
│ Request       │       │ Servers       │       │ Service       │
└───────────────┘       └───────────────┘       └───────────────┘
                                │                      │
                                ▼                      ▼
                      ┌─────────────────┐      ┌─────────────────┐
                      │ Storage Nodes   │◀────▶│ Replication     │
                      │ (Blob Chunks)   │      │ & Consensus     │
                      └─────────────────┘      └─────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Is blob storage the same as a traditional file system? Commit yes or no.

Common Belief: Blob storage works exactly like a regular file system on your computer.

Reality: Blob storage is object-based, not file-based. It does not support file system features like folders, file locking, or random writes like a disk.

Why it matters: Assuming blob storage is a file system can lead to design mistakes, such as expecting fast random writes or hierarchical folder operations that are slow or unsupported.

Quick: Do you think blob storage automatically backs up your data forever? Commit yes or no.

Common Belief: Once you upload data to blob storage, it is permanently safe without extra steps.

Reality: Blob storage replicates data for durability but does not replace backups. Deleted or corrupted blobs are lost unless you enable versioning or backup solutions.

Why it matters: Relying solely on blob storage replication can cause data loss in accidental deletion or corruption scenarios.

Quick: Is blob storage always fast for any size file? Commit yes or no.

Common Belief: Blob storage delivers instant, high-speed access regardless of file size or location.

Reality: Blob storage performance depends on blob size, network, and region. Large files may require multipart uploads, and latency can vary. Caching or CDNs are needed for consistent speed.

Why it matters: Ignoring performance characteristics can cause slow user experiences or high costs.

Quick: Can anyone access blobs by default? Commit yes or no.

Common Belief: Blobs are public by default and accessible to anyone on the internet.

Reality: Blobs are private by default. Access requires explicit permissions or signed URLs.

Why it matters: Misunderstanding default privacy can lead to accidental data exposure or access failures.

Expert Zone

1

Blob storage systems often use eventual consistency for some operations, meaning changes may take time to appear globally, which affects application design.

2

Choosing the right replication strategy balances cost, latency, and durability; geo-redundant storage is more expensive but protects against regional disasters.

3

Multipart uploads and parallel downloads improve performance but require careful error handling and cleanup of incomplete parts.

When NOT to use

Blob storage is not suitable for low-latency random read/write workloads like databases or virtual machine disks. Use block storage or file storage services instead. Also, for small metadata-heavy files, a database or file system may be more efficient.

Production Patterns

In production, blob storage is used for media hosting with CDN integration, backup and archive with lifecycle policies, big data lakes with tiered storage, and as a source for serverless functions. Access control is often managed via IAM roles and signed URLs for temporary access.

Connections

Content Delivery Network (CDN)

Builds-on

Understanding blob storage helps grasp how CDNs cache and deliver large files globally to reduce latency and bandwidth costs.

Distributed File Systems

Similar pattern

Blob storage shares principles with distributed file systems like data chunking and replication but differs in access methods and consistency models.

Library Cataloging Systems

Analogous system

Knowing how libraries organize books by categories and unique IDs helps understand blob storage's bucket and key naming conventions.

Common Pitfalls

#1 Uploading very large files as a single blob without chunking.

Wrong approach:

Upload a 10GB video file in one HTTP request without multipart upload.

Correct approach:

Split the 10GB video into smaller blocks and upload using multipart upload APIs.

Root cause:

Misunderstanding blob storage limits and ignoring network reliability and performance best practices.

#2 Setting all blobs to public access without restrictions.

Wrong approach:

Configure bucket policy to allow public read access for all blobs.

Correct approach:

Use private buckets and generate signed URLs for controlled temporary access.

Root cause:

Lack of awareness about security defaults and risks of data exposure.

#3 Assuming immediate consistency after blob update.

Wrong approach:

Immediately reading a blob after upload expecting the new version everywhere.

Correct approach:

Design applications to handle eventual consistency delays or use strong consistency features if available.

Root cause:

Not understanding the consistency model of blob storage services.

Key Takeaways

Blob storage stores large unstructured data as objects called blobs inside buckets or containers accessible via web APIs.

It is designed for scalability, durability, and cost-effectiveness by splitting data, replicating it, and using commodity hardware.

Security and access control are critical; blobs are private by default and require proper permissions or signed URLs.

Performance depends on blob size, access patterns, and caching; understanding these helps optimize user experience.

Blob storage is not a file system replacement and has limits; knowing when to use it and its internals enables better system design.

Practice

1. What is the main purpose of blob storage services like Amazon S3 or Azure Blob Storage?

easy

A. To manage user identities and permissions

B. To run virtual machines in the cloud

C. To store large files safely and accessibly in the cloud

D. To host websites directly without any storage

Solution

1. Step 1: Understand blob storage function

   Blob storage is designed to save large files like videos, images, and backups in the cloud.

2. Step 2: Compare with other cloud services

   Other services like virtual machines or identity management serve different purposes, not file storage.

3. Final Answer:

   To store large files safely and accessibly in the cloud -> Option C

4. Quick Check:

   Blob storage = large file storage [OK]

Hint: Blob storage is for big files, not computing or identity [OK]

Common Mistakes:

• Confusing blob storage with compute services
• Thinking blob storage manages users
• Assuming blob storage hosts websites directly

2. Which of the following is the correct way to specify a bucket name in Amazon S3?

easy

A. My.bucket.name

B. MyBucketName!

C. my_bucket_name

D. my-bucket-name

Solution

1. Step 1: Review S3 bucket naming rules

   S3 bucket names must be lowercase, can include hyphens, but no underscores or special characters like '!'.

2. Step 2: Check each option

   my-bucket-name uses lowercase letters and hyphens, which is valid. Options A, B, and C are invalid: A has uppercase letters and dots, B has uppercase and '!', C has underscores.

3. Final Answer:

   my-bucket-name -> Option D

4. Quick Check:

   Bucket names use lowercase and hyphens [OK]

Hint: Bucket names: lowercase letters and hyphens only [OK]

Common Mistakes:

• Using uppercase letters in bucket names
• Including underscores or special characters
• Using dots in bucket names

3. Given this AWS CLI command:
aws s3 cp file.txt s3://my-bucket/ --storage-class STANDARD_IA
What does the --storage-class STANDARD_IA option do?

medium

A. Uploads the file to a cheaper, infrequently accessed storage tier

B. Encrypts the file automatically during upload

C. Makes the file publicly accessible

D. Uploads the file with maximum redundancy across regions

Solution

1. Step 1: Understand storage classes in S3

   STANDARD_IA means Standard-Infrequent Access, a cheaper tier for files accessed less often.

2. Step 2: Analyze command effect

   The command uploads file.txt to the bucket using this storage class, saving cost for infrequent access.

3. Final Answer:

   Uploads the file to a cheaper, infrequently accessed storage tier -> Option A

4. Quick Check:

   STANDARD_IA = cheaper, infrequent access [OK]

Hint: STANDARD_IA means cheaper storage for rare access [OK]

Common Mistakes:

• Thinking it encrypts the file
• Assuming it makes file public
• Confusing with multi-region redundancy

4. You try to upload a file to Azure Blob Storage but get an error saying the container name is invalid. Which of these is the most likely cause?

medium

A. The container name contains uppercase letters

B. The file size is too large

C. The storage account is in the wrong region

D. The file name has spaces

Solution

1. Step 1: Recall Azure container naming rules

   Container names must be lowercase letters and numbers only, no uppercase letters allowed.

2. Step 2: Match error cause

   An invalid container name error usually means the name breaks these rules, such as having uppercase letters.

3. Final Answer:

   The container name contains uppercase letters -> Option A

4. Quick Check:

   Azure container names must be lowercase [OK]

Hint: Azure containers must be lowercase, no uppercase letters [OK]

Common Mistakes:

• Blaming file size for container name errors
• Confusing region with naming rules
• Thinking file name affects container name

5. You want to store backup files in Amazon S3 with minimal cost but still need quick access sometimes. Which combination of settings is best?

hard

A. Use STANDARD storage class and enable public access for easy retrieval

B. Use STANDARD_IA storage class with lifecycle rule to move files to GLACIER after 30 days

C. Use GLACIER storage class for all files with no lifecycle rules

D. Use REDUCED_REDUNDANCY storage class and no lifecycle rules

Solution

1. Step 1: Understand storage classes and lifecycle

   STANDARD_IA offers cheaper storage with quick access. GLACIER is very cheap but slow to retrieve.

2. Step 2: Analyze cost and access needs

   Using STANDARD_IA first allows quick access. Moving to GLACIER after 30 days saves cost for older backups.

3. Step 3: Evaluate other options

   GLACIER only delays access too much. STANDARD is costly. REDUCED_REDUNDANCY is deprecated and less reliable.

4. Final Answer:

   Use STANDARD_IA storage class with lifecycle rule to move files to GLACIER after 30 days -> Option B

5. Quick Check:

   STANDARD_IA + lifecycle to GLACIER = cost + access balance [OK]

Hint: Start with STANDARD_IA, then archive to GLACIER for cost savings [OK]

Common Mistakes:

• Choosing GLACIER only and ignoring access speed
• Enabling public access unnecessarily
• Using deprecated REDUCED_REDUNDANCY class