Blob storage (S3, Azure Blob) in HLD - Deep Dive

Overview - Blob storage (S3, Azure Blob)

What is it?

Blob storage is a way to store large amounts of unstructured data like images, videos, documents, or backups. It organizes data as objects called blobs, which can be accessed over the internet. Services like Amazon S3 and Azure Blob Storage provide scalable, durable, and secure storage solutions for these blobs. Users can upload, download, and manage blobs using simple APIs.

Why it matters

Without blob storage, storing and managing large files would be slow, unreliable, and expensive. Traditional file systems or databases struggle with scale and performance for big data. Blob storage solves this by offering a simple, scalable, and cost-effective way to store any amount of data accessible from anywhere. This enables cloud apps, backups, media streaming, and data lakes to work efficiently.

Where it fits

Before learning blob storage, you should understand basic storage concepts like files, databases, and cloud computing. After this, you can explore related topics like content delivery networks (CDNs), data lifecycle management, and distributed file systems. Blob storage is a foundational building block for cloud-native architectures and big data solutions.

Mental Model

Core Idea

Blob storage is like a giant, organized warehouse where each item (blob) is stored in a labeled box (container/bucket) and can be quickly found and retrieved over the internet.

Think of it like...

Imagine a massive library where each book is a blob. The library shelves are containers or buckets, and each book has a unique label (key). You can ask the librarian (API) to fetch, add, or remove any book quickly without searching the entire library.

┌─────────────────────────────┐
│        Blob Storage         │
│ ┌─────────────┐            │
│ │  Bucket/    │            │
│ │ Container 1 │            │
│ │ ┌─────────┐ │            │
│ │ │ Blob A  │ │            │
│ │ ├─────────┤ │            │
│ │ │ Blob B  │ │            │
│ │ └─────────┘ │            │
│ ├─────────────┤            │
│ │  Bucket/    │            │
│ │ Container 2 │            │
│ │ ┌─────────┐ │            │
│ │ │ Blob C  │ │            │
│ │ └─────────┘ │            │
│ └─────────────┘            │
└─────────────────────────────┘

Build-Up - 7 Steps

1

Foundation

What is Blob Storage?

Concept: Introduce the basic idea of blob storage as a way to store unstructured data as objects.

Blob storage stores data as blobs (binary large objects). Unlike files on your computer, blobs are stored in containers or buckets. Each blob has a unique name (key) and can be any size or type of data, like photos, videos, or backups. You access blobs via simple web APIs.

Result

You understand that blob storage is a cloud service for storing large, unstructured files accessible over the internet.

Understanding that blob storage treats data as objects rather than files or blocks is key to grasping its flexibility and scalability.

2

Foundation

Buckets and Containers Explained

Concept: Explain how blobs are grouped inside buckets or containers for organization and access control.

Buckets (S3) or containers (Azure Blob) are like folders that hold blobs. They provide a namespace so blob names are unique within them. Buckets also help manage permissions, lifecycle policies, and billing. You create buckets first, then upload blobs inside them.

Result

You can visualize how blobs are organized and managed inside buckets or containers.

Knowing that buckets are the main organizational unit helps you design storage layouts and control access effectively.

3

Intermediate

Blob Types and Access Patterns

🤔Before reading on: do you think all blobs are stored and accessed the same way? Commit to your answer.

Concept: Introduce different blob types and how access patterns affect storage choices.

Blob storage supports types like block blobs (large files uploaded in parts), append blobs (for logs), and page blobs (random read/write, used for disks). Access can be public, private, or via signed URLs. Choosing the right blob type and access method optimizes performance and cost.

Result

You understand that blob storage is flexible to support different data types and usage scenarios.

Recognizing blob types and access patterns helps you optimize storage for your application's needs and avoid unnecessary costs.

4

Intermediate

Durability and Replication Strategies

🤔Before reading on: do you think blob storage keeps only one copy of your data? Commit to yes or no.

Concept: Explain how blob storage ensures data durability and availability through replication.

Blob storage replicates data across multiple servers and sometimes across regions. Common strategies include locally redundant storage (copies in one data center), zone-redundant (across data centers), and geo-redundant (across regions). This protects data from hardware failures and disasters.

Result

You know how blob storage keeps your data safe and available even if parts of the system fail.

Understanding replication strategies is crucial for designing reliable systems and meeting data durability requirements.

5

Intermediate

Security and Access Control Mechanisms

🤔Before reading on: do you think blob storage data is always public by default? Commit to yes or no.

Concept: Describe how blob storage secures data using authentication, authorization, and encryption.

Blob storage uses access keys, IAM roles, and signed URLs to control who can read or write blobs. Data is encrypted at rest and in transit. You can set bucket policies or container ACLs to restrict access. Auditing logs track usage for compliance.

Result

You understand how to protect your data and control access securely in blob storage.

Knowing security features helps prevent data leaks and ensures compliance with regulations.

6

Advanced

Performance Optimization and Caching

🤔Before reading on: do you think blob storage automatically caches data for faster access? Commit to yes or no.

Concept: Explore how to improve blob storage performance using caching and tuning.

Blob storage itself is highly scalable but can have latency. Using CDNs caches blobs closer to users for faster reads. You can tune blob size, parallel uploads, and request patterns to optimize throughput. Understanding eventual consistency helps design around delays.

Result

You can design systems that deliver blob data quickly and efficiently at scale.

Knowing performance tuning and caching strategies prevents bottlenecks and improves user experience.

7

Expert

Internals of Blob Storage Architecture

🤔Before reading on: do you think blob storage stores blobs as single files on disk? Commit to yes or no.

Concept: Reveal how blob storage systems manage data internally for scalability and durability.

Blob storage breaks large blobs into smaller chunks stored across distributed servers. Metadata tracks blob parts and versions. Systems use consensus protocols to maintain consistency and replication. Data is stored on commodity hardware with error correction. This design balances cost, scale, and reliability.

Result

You gain deep insight into how blob storage works behind the scenes to serve billions of requests reliably.

Understanding internal architecture helps troubleshoot, optimize, and innovate on blob storage solutions.

Under the Hood

Blob storage systems split large files into smaller blocks or pages, each stored on distributed servers. Metadata services keep track of blob composition, versions, and locations. Replication protocols copy data across nodes or regions to ensure durability. Access requests go through front-end servers that authenticate, authorize, and route them to the correct storage nodes. Data is encrypted and checksummed to detect corruption.

Why designed this way?

This design allows blob storage to scale massively while keeping costs low by using commodity hardware. Splitting blobs enables parallel uploads and downloads, improving performance. Replication and consensus protocols ensure data safety despite hardware failures or network issues. Alternatives like monolithic file systems or databases were too slow or expensive at cloud scale.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│ Client/API    │──────▶│ Front-end     │──────▶│ Metadata      │
│ Request       │       │ Servers       │       │ Service       │
└───────────────┘       └───────────────┘       └───────────────┘
                                │                      │
                                ▼                      ▼
                      ┌─────────────────┐      ┌─────────────────┐
                      │ Storage Nodes   │◀────▶│ Replication     │
                      │ (Blob Chunks)   │      │ & Consensus     │
                      └─────────────────┘      └─────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Is blob storage the same as a traditional file system? Commit yes or no.

Common Belief: Blob storage works exactly like a regular file system on your computer.

Reality: Blob storage is object-based, not file-based. It does not support file system features like folders, file locking, or random writes like a disk.

Why it matters: Assuming blob storage is a file system can lead to design mistakes, such as expecting fast random writes or hierarchical folder operations that are slow or unsupported.

Quick: Do you think blob storage automatically backs up your data forever? Commit yes or no.

Common Belief: Once you upload data to blob storage, it is permanently safe without extra steps.

Reality: Blob storage replicates data for durability but does not replace backups. Deleted or corrupted blobs are lost unless you enable versioning or backup solutions.

Why it matters: Relying solely on blob storage replication can cause data loss in accidental deletion or corruption scenarios.

Quick: Is blob storage always fast for any size file? Commit yes or no.

Common Belief: Blob storage delivers instant, high-speed access regardless of file size or location.

Reality: Blob storage performance depends on blob size, network, and region. Large files may require multipart uploads, and latency can vary. Caching or CDNs are needed for consistent speed.

Why it matters: Ignoring performance characteristics can cause slow user experiences or high costs.

Quick: Can anyone access blobs by default? Commit yes or no.

Common Belief: Blobs are public by default and accessible to anyone on the internet.

Reality: Blobs are private by default. Access requires explicit permissions or signed URLs.

Why it matters: Misunderstanding default privacy can lead to accidental data exposure or access failures.

Expert Zone

1

Blob storage systems often use eventual consistency for some operations, meaning changes may take time to appear globally, which affects application design.

2

Choosing the right replication strategy balances cost, latency, and durability; geo-redundant storage is more expensive but protects against regional disasters.

3

Multipart uploads and parallel downloads improve performance but require careful error handling and cleanup of incomplete parts.

When NOT to use

Blob storage is not suitable for low-latency random read/write workloads like databases or virtual machine disks. Use block storage or file storage services instead. Also, for small metadata-heavy files, a database or file system may be more efficient.

Production Patterns

In production, blob storage is used for media hosting with CDN integration, backup and archive with lifecycle policies, big data lakes with tiered storage, and as a source for serverless functions. Access control is often managed via IAM roles and signed URLs for temporary access.

Connections

Content Delivery Network (CDN)

Builds-on

Understanding blob storage helps grasp how CDNs cache and deliver large files globally to reduce latency and bandwidth costs.

Distributed File Systems

Similar pattern

Blob storage shares principles with distributed file systems like data chunking and replication but differs in access methods and consistency models.

Library Cataloging Systems

Analogous system

Knowing how libraries organize books by categories and unique IDs helps understand blob storage's bucket and key naming conventions.

Common Pitfalls

#1 Uploading very large files as a single blob without chunking.

Wrong approach:

Upload a 10GB video file in one HTTP request without multipart upload.

Correct approach:

Split the 10GB video into smaller blocks and upload using multipart upload APIs.

Root cause:

Misunderstanding blob storage limits and ignoring network reliability and performance best practices.

#2 Setting all blobs to public access without restrictions.

Wrong approach:

Configure bucket policy to allow public read access for all blobs.

Correct approach:

Use private buckets and generate signed URLs for controlled temporary access.

Root cause:

Lack of awareness about security defaults and risks of data exposure.

#3 Assuming immediate consistency after blob update.

Wrong approach:

Immediately reading a blob after upload expecting the new version everywhere.

Correct approach:

Design applications to handle eventual consistency delays or use strong consistency features if available.

Root cause:

Not understanding the consistency model of blob storage services.

Key Takeaways

Blob storage stores large unstructured data as objects called blobs inside buckets or containers accessible via web APIs.

It is designed for scalability, durability, and cost-effectiveness by splitting data, replicating it, and using commodity hardware.

Security and access control are critical; blobs are private by default and require proper permissions or signed URLs.

Performance depends on blob size, access patterns, and caching; understanding these helps optimize user experience.

Blob storage is not a file system replacement and has limits; knowing when to use it and its internals enables better system design.