Snowflake architecture (storage, compute, services layers) - Deep Dive

Overview - Snowflake architecture (storage, compute, services layers)

What is it?

Snowflake architecture is a way to organize how data is stored, processed, and managed in the Snowflake cloud data platform. It separates storage, compute, and services into different layers that work together but can scale independently. This design helps users handle large amounts of data efficiently and run many queries at the same time without slowing down.

Why it matters

Without this architecture, data platforms would struggle to balance storage needs and computing power, causing slow queries and high costs. Snowflake’s design solves this by letting storage grow separately from compute, so companies only pay for what they use and get fast results. This means better performance, flexibility, and cost control for businesses working with big data.

Where it fits

Before learning Snowflake architecture, you should understand basic cloud computing and data storage concepts. After this, you can explore how to write queries in Snowflake, optimize performance, and manage security. This architecture is a foundation for mastering Snowflake’s features and cloud data warehousing.

Mental Model

Core Idea

Snowflake architecture splits data storage, computing power, and management services into separate layers that work independently but together to deliver fast, scalable, and cost-efficient data processing.

Think of it like...

Imagine a restaurant kitchen where the pantry (storage) holds all ingredients, the chefs (compute) prepare meals, and the managers (services) coordinate orders and quality. Each part works separately but must communicate smoothly to serve customers quickly and well.

┌───────────────┐   ┌───────────────┐   ┌───────────────┐
│   Services    │──▶│   Compute     │──▶│   Storage     │
│  Layer       │   │   Layer       │   │   Layer       │
│ (Coordination│   │ (Processing)  │   │ (Data held)   │
│  & Security) │   │               │   │               │
└───────────────┘   └───────────────┘   └───────────────┘

Build-Up - 7 Steps

1

Foundation

Understanding Cloud Data Storage Basics

Concept: Learn what cloud data storage means and why it is important for modern data platforms.

Cloud data storage means saving data on remote servers accessed over the internet instead of local computers. This allows easy access, sharing, and scaling without buying physical hardware. Data is stored in files or tables and can grow as needed.

Result

You understand that data is kept safely and flexibly in the cloud, ready for processing.

Knowing cloud storage basics helps you grasp why separating storage from compute is powerful in Snowflake.

2

Foundation

What Compute Means in Data Platforms

Concept: Discover what compute resources do and how they process data queries.

Compute refers to the servers and CPUs that run programs to analyze or transform data. When you ask a question (query) about data, compute power works to find and calculate the answer. More compute means faster results but costs more.

Result

You see compute as the 'worker' that does the heavy lifting on data.

Understanding compute clarifies why separating it from storage allows flexible scaling and cost control.

3

Intermediate

Services Layer Role in Snowflake

🤔Before reading on: do you think the services layer stores data or manages operations? Commit to your answer.

Concept: Introduce the services layer that manages security, metadata, and coordination in Snowflake.

The services layer is like the brain of Snowflake. It handles user access, query parsing, optimization, and metadata (data about data). It does not store data or run queries but tells compute and storage what to do and when.

Result

You understand that services layer controls and secures the system without holding data or doing heavy processing.

Knowing the services layer’s role helps you see how Snowflake keeps operations smooth and secure while scaling.

4

Intermediate

How Storage Layer Works Independently

🤔Before reading on: do you think storage and compute must always scale together? Commit to your answer.

Concept: Explain that Snowflake’s storage layer is separate and can grow without affecting compute.

Snowflake stores all data in a centralized cloud storage that is separate from compute. This means data can grow very large without slowing down queries. Storage is designed to be durable, secure, and cost-efficient, using cloud providers like AWS, Azure, or GCP.

Result

You see that storage can expand independently, saving costs and improving performance.

Understanding independent storage scaling is key to Snowflake’s flexibility and cost savings.

5

Intermediate

Compute Layer and Virtual Warehouses

🤔Before reading on: do you think all queries share the same compute resources or have separate ones? Commit to your answer.

Concept: Introduce virtual warehouses as separate compute clusters that run queries independently.

Snowflake uses virtual warehouses, which are groups of compute servers that run queries. Each warehouse works independently, so many users can run queries at the same time without waiting. Warehouses can be started, stopped, or resized on demand.

Result

You understand how compute resources can be flexible and isolated for better performance.

Knowing about virtual warehouses explains how Snowflake handles many users and workloads smoothly.

6

Advanced

How Layers Communicate and Coordinate

🤔Before reading on: do you think compute accesses storage directly or through services? Commit to your answer.

Concept: Explain the interaction between services, compute, and storage layers during query execution.

When a query runs, the services layer parses and plans it, then tells the compute layer what to do. Compute fetches data from storage as needed. Results are sent back through services for security checks and returned to the user. This separation allows each layer to focus on its job efficiently.

Result

You see the smooth flow of data and commands between layers during operations.

Understanding this flow clarifies why Snowflake can scale and perform well under heavy use.

7

Expert

Optimizations and Multi-Cluster Warehouses

🤔Before reading on: do you think Snowflake automatically balances load across compute clusters? Commit to your answer.

Concept: Explore advanced features like multi-cluster warehouses that auto-scale compute for concurrency and performance.

Snowflake can run multiple compute clusters for the same warehouse to handle many queries at once. It automatically adds or removes clusters based on workload, preventing slowdowns. This dynamic scaling is managed by the services layer and helps maintain fast response times even with many users.

Result

You understand how Snowflake optimizes compute resources automatically for peak performance.

Knowing about multi-cluster warehouses reveals how Snowflake solves common concurrency problems in data platforms.

Under the Hood

Snowflake’s architecture separates storage, compute, and services into distinct layers connected by secure APIs. Storage uses cloud object storage to hold compressed, columnar data files. Compute runs in virtual warehouses that read data from storage on demand. The services layer manages metadata, security, query parsing, and optimization, coordinating compute and storage without holding data itself. This separation allows independent scaling and efficient resource use.

Why designed this way?

Snowflake was designed to overcome limits of traditional data warehouses that tightly couple storage and compute, causing bottlenecks and high costs. By separating layers, Snowflake enables elastic scaling, better concurrency, and cost savings. Alternatives like monolithic systems were less flexible and more expensive to operate at scale.

┌───────────────┐          ┌───────────────┐          ┌───────────────┐
│   Services    │─────────▶│   Compute     │─────────▶│   Storage     │
│  Layer       │          │   Layer       │          │   Layer       │
│ (Metadata,   │          │ (Virtual      │          │ (Cloud Object │
│  Security,   │          │  Warehouses)  │          │  Storage)     │
│  Coordination)│          │               │          │               │
└───────────────┘          └───────────────┘          └───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Do you think Snowflake stores data inside compute clusters? Commit to yes or no.

Common Belief: Snowflake stores data inside its compute clusters for faster access.

Reality: Snowflake stores all data separately in cloud storage, not inside compute clusters.

Why it matters: Believing data is inside compute can lead to wrong assumptions about scaling and costs, causing inefficient resource use.

Quick: Do you think all users share the same compute resources in Snowflake? Commit to yes or no.

Common Belief: All users run queries on the same compute resources, so heavy use slows everyone down.

Reality: Snowflake uses separate virtual warehouses for compute, so users can run queries independently without interference.

Why it matters: Misunderstanding this can cause poor workload planning and unnecessary scaling costs.

Quick: Do you think the services layer does heavy data processing? Commit to yes or no.

Common Belief: The services layer processes data and runs queries like compute does.

Reality: The services layer only manages metadata, security, and coordination; it does not process data.

Why it matters: Confusing roles can lead to misconfigurations and performance issues.

Quick: Do you think storage and compute must always scale together? Commit to yes or no.

Common Belief: Storage and compute scale together because they are tightly linked.

Reality: Snowflake’s architecture allows storage and compute to scale independently.

Why it matters: Not knowing this can cause overprovisioning and higher costs.

Expert Zone

1

Virtual warehouses can be paused to save costs without losing data or metadata.

2

The services layer caches metadata aggressively to reduce latency and improve query planning speed.

3

Snowflake’s storage uses micro-partitions with automatic clustering, which optimizes query performance without manual tuning.

When NOT to use

Snowflake architecture is not ideal for real-time transactional systems requiring millisecond latency. For such cases, specialized OLTP databases or streaming platforms are better. Also, if you need on-premises deployment, Snowflake’s cloud-only design is not suitable.

Production Patterns

In production, teams use multiple virtual warehouses sized and scheduled for different workloads, such as ETL, reporting, and ad-hoc queries. They leverage multi-cluster warehouses for concurrency and use resource monitors to control costs. The services layer is configured with role-based access control for security.

Connections

Microservices Architecture

Both separate concerns into independent layers or services that communicate via APIs.

Understanding Snowflake’s layered design helps grasp how microservices isolate functions for scalability and maintainability.

Operating System Kernel

The services layer acts like an OS kernel managing resources and coordinating tasks between hardware (storage) and applications (compute).

Seeing the services layer as a kernel clarifies its role in managing metadata, security, and resource allocation.

Restaurant Kitchen Workflow

Similar to how storage, compute, and services layers work, a kitchen separates ingredient storage, cooking, and order management.

This cross-domain connection shows how separating roles improves efficiency and scalability in complex systems.

Common Pitfalls

#1 Trying to scale compute and storage together manually.

Wrong approach:

Manually increasing both storage size and compute warehouse size at the same time for every workload change.

Correct approach:

Scale storage independently as data grows and adjust compute warehouses based on query load separately.

Root cause:

Misunderstanding that storage and compute are separate layers that scale independently.

#2 Using a single virtual warehouse for all workloads.

Wrong approach:

Running all queries on one virtual warehouse regardless of workload type or concurrency needs.

Correct approach:

Create multiple virtual warehouses sized and scheduled for different workloads to avoid resource contention.

Root cause:

Not knowing that virtual warehouses isolate compute resources for better performance.

#3 Assuming services layer processes data directly.

Wrong approach:

Trying to optimize query speed by focusing on services layer settings expecting it to speed up data processing.

Correct approach:

Focus on compute warehouse sizing and query optimization since services layer manages coordination, not data processing.

Root cause:

Confusing the role of the services layer with compute.

Key Takeaways

Snowflake architecture separates storage, compute, and services into independent layers for flexibility and efficiency.

Storage holds all data in cloud object storage, allowing it to scale without affecting compute resources.

Compute runs in virtual warehouses that can be started, stopped, and resized independently to handle different workloads.

The services layer manages metadata, security, and coordination but does not store or process data itself.

This design enables Snowflake to deliver fast, scalable, and cost-effective data processing in the cloud.