Hot-warm-cold architecture in Elasticsearch - Deep Dive

Overview - Hot-warm-cold architecture

What is it?

Hot-warm-cold architecture is a way to organize data storage in Elasticsearch based on how often data is used and how fast it needs to be accessed. It divides data into three layers: hot for new and frequently accessed data, warm for less active data, and cold for old data that is rarely accessed but still kept. This helps manage resources efficiently and keeps search fast.

Why it matters

Without this architecture, all data would be treated the same, making searches slower and more expensive as data grows. It solves the problem of balancing speed and cost by storing data in the right place depending on its age and usage. This means businesses can keep large amounts of data without slowing down or spending too much on storage.

Where it fits

Before learning this, you should understand basic Elasticsearch concepts like indices and nodes. After this, you can explore advanced data lifecycle management and performance tuning in Elasticsearch clusters.

Mental Model

Core Idea

Hot-warm-cold architecture organizes data by usage frequency and age to optimize cost and performance in Elasticsearch.

Think of it like...

It's like a library where new popular books are kept on the front shelves (hot), older but still sometimes read books are on side shelves (warm), and rarely read books are stored in the basement (cold).

┌───────────┐   ┌───────────┐   ┌───────────┐
│   HOT     │→ │   WARM    │→ │   COLD    │
│ (Fast,   │   │ (Slower,  │   │ (Slowest, │
│  expensive)│  │  cheaper) │   │  cheapest)│
└───────────┘   └───────────┘   └───────────┘
   ▲              ▲               ▲
   │              │               │
New &          Less active     Rarely used
frequently     data            data
accessed

Build-Up - 7 Steps

1

Foundation

Understanding Elasticsearch data nodes

Concept: Learn what data nodes are and how they store data in Elasticsearch.

Elasticsearch stores data in units called indices, which are split into shards. These shards live on data nodes, which are servers that hold and manage the data. Each node can have different roles, but data nodes specifically hold the actual data and respond to search requests.

Result

You know that data nodes are the servers where your data lives and that they handle search and indexing.

Understanding data nodes is key because hot-warm-cold architecture depends on placing data on different nodes based on usage.

2

Foundation

What is data aging in Elasticsearch?

Concept: Data aging means data changes in how often it is used over time.

When data is new, it is accessed often for searching or updating. Over time, it becomes less important and is accessed less. Eventually, it might only be kept for records and rarely searched. This natural change is called data aging.

Result

You understand that data usage changes over time, which creates the need to treat data differently.

Knowing data ages helps explain why storing all data the same way is inefficient.

3

Intermediate

Defining hot, warm, and cold tiers

🤔Before reading on: do you think hot data is stored on the fastest or slowest hardware? Commit to your answer.

Concept: Hot, warm, and cold tiers are categories of data storage based on access speed and cost.

Hot tier stores new, frequently accessed data on fast, expensive hardware for quick searches. Warm tier holds older data accessed less often on slower, cheaper hardware. Cold tier keeps old, rarely accessed data on the slowest and cheapest hardware but still searchable.

Result

You can classify data into three tiers and understand their tradeoffs between speed and cost.

Recognizing these tiers helps optimize resource use by matching data needs to hardware capabilities.

4

Intermediate

How data moves between tiers

🤔Before reading on: do you think data moves automatically between tiers or must be moved manually? Commit to your answer.

Concept: Data moves from hot to warm to cold as it ages, often automatically using policies.

Elasticsearch can use Index Lifecycle Management (ILM) policies to automatically move data between tiers based on age or size. For example, after 7 days in hot, data moves to warm; after 30 days, it moves to cold. This keeps the system efficient without manual work.

Result

You understand that data migration between tiers can be automated to save effort and maintain performance.

Knowing automatic data movement prevents manual errors and ensures data is always stored optimally.

5

Intermediate

Hardware and configuration differences per tier

Concept: Each tier uses different hardware and settings to balance cost and performance.

Hot nodes use SSDs and more CPU for fast indexing and searching. Warm nodes use slower disks like HDDs and less CPU since data is less active. Cold nodes use even slower storage and minimal CPU because data is rarely searched. Settings like replica count and refresh intervals also differ per tier.

Result

You can plan hardware and settings for each tier to optimize costs and speed.

Understanding hardware differences helps design clusters that meet performance needs without overspending.

6

Advanced

Tradeoffs and challenges in tiered storage

🤔Before reading on: do you think cold data is always instantly searchable or might it have delays? Commit to your answer.

Concept: Tiered storage improves cost but can introduce delays and complexity.

Cold data may take longer to search because it’s on slower disks or even frozen (offline) storage. Managing ILM policies and ensuring data integrity across tiers adds complexity. Also, wrong tiering can cause performance issues or higher costs if data is misclassified.

Result

You see that tiered storage is a balance, not a perfect solution, requiring careful planning.

Knowing these tradeoffs prepares you to design and maintain efficient, reliable Elasticsearch clusters.

7

Expert

Optimizing hot-warm-cold in large clusters

🤔Before reading on: do you think all indices should have the same lifecycle policy in large clusters? Commit to your answer.

Concept: Large clusters need tailored policies and monitoring for each data type and index.

In big systems, different data sets have different lifecycles and access patterns. Experts create custom ILM policies per index, monitor node health closely, and use shard allocation awareness to keep data balanced. They also tune refresh rates and replica counts per tier to optimize performance and cost.

Result

You understand how to scale hot-warm-cold architecture for complex, real-world Elasticsearch deployments.

Knowing how to customize and monitor tiering policies is essential for maintaining performance and cost efficiency at scale.

Under the Hood

Elasticsearch uses node attributes and shard allocation rules to place indices on nodes tagged as hot, warm, or cold. ILM policies automate index rollover and shard movement by changing index settings and triggering shard relocation. The cluster state tracks node roles and index locations, coordinating searches and writes accordingly.

Why designed this way?

This design arose to handle growing data volumes without linear cost increases. Early Elasticsearch clusters treated all data equally, causing performance bottlenecks and high costs. Separating data by usage allowed better hardware utilization and predictable performance. Alternatives like single-tier storage were too expensive or slow at scale.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│   Hot Nodes   │──────▶│   Warm Nodes  │──────▶│   Cold Nodes  │
│ (Fast SSDs)   │       │ (Slower HDDs) │       │ (Slowest Disk)│
│  Index writes │       │  Read mostly  │       │  Archive data │
│  Frequent     │       │  Less writes  │       │  Rare queries │
└───────────────┘       └───────────────┘       └───────────────┘
        ▲                      ▲                       ▲
        │                      │                       │
   New data             Aging data              Old data
   indexed here         moved here             moved here

Myth Busters - 4 Common Misconceptions

Quick: Is cold data in Elasticsearch always offline and inaccessible? Commit to yes or no.

Common Belief: Cold data is offline and cannot be searched until manually restored.

Reality: Cold data is still searchable but may have slower response times because it is stored on slower hardware or frozen nodes.

Why it matters: Believing cold data is offline can lead to unnecessary manual steps and delays in accessing important historical data.

Quick: Does hot-warm-cold architecture mean you must buy three different types of hardware? Commit to yes or no.

Common Belief: You must have three completely separate hardware setups for hot, warm, and cold tiers.

Reality: While hardware is often different, many clusters use mixed nodes or cloud storage tiers to reduce complexity and cost.

Why it matters: Thinking you need separate hardware can discourage adoption or lead to over-provisioning.

Quick: Does data automatically move between tiers without any configuration? Commit to yes or no.

Common Belief: Data moves between hot, warm, and cold tiers automatically without any setup.

Reality: You must configure Index Lifecycle Management policies to automate data movement; otherwise, data stays where it is.

Why it matters: Assuming automatic movement can cause data to remain on expensive hot nodes longer than needed, increasing costs.

Quick: Is hot data always the largest portion of your Elasticsearch storage? Commit to yes or no.

Common Belief: Hot data makes up most of the storage because it is the newest and most important.

Reality: Hot data is usually a small fraction; warm and cold data often make up the majority of storage as data ages.

Why it matters: Misunderstanding data distribution can lead to poor capacity planning and unexpected costs.

Expert Zone

1

ILM policies can include custom actions like force merging or shrinking indices to optimize storage in warm and cold tiers.

2

Shard allocation awareness can be used to keep replicas on different tiers or availability zones for fault tolerance.

3

Cold tier nodes can be configured as frozen nodes, which keep data on remote storage and load it into memory only when searched.

When NOT to use

Hot-warm-cold architecture is less suitable for small clusters or use cases where all data is equally critical and frequently accessed. In such cases, a single-tier cluster or time-series optimized indices might be better.

Production Patterns

In production, teams often combine hot-warm-cold with rollover indices and snapshot backups. They monitor ILM execution closely and adjust policies based on query patterns and storage costs. Some use cloud storage for cold tier to reduce on-premises hardware needs.

Connections

Data Lifecycle Management

Hot-warm-cold architecture builds on data lifecycle management principles.

Understanding lifecycle management helps grasp how data moves through hot, warm, and cold tiers automatically.

Cache Hierarchy in Computer Architecture

Both organize data storage by speed and cost tradeoffs.

Knowing cache layers (L1, L2, L3) helps understand why Elasticsearch separates data into hot, warm, and cold tiers for efficiency.

Library Book Organization

Both arrange items by usage frequency and accessibility.

Seeing how libraries keep popular books handy and archives in storage clarifies why data is tiered in Elasticsearch.

Common Pitfalls

#1 Not setting ILM policies causes data to stay on hot nodes indefinitely.

Wrong approach:

PUT /_template/my_template
{
"index_patterns": ["logs-*"],
"settings": {
"number_of_shards": 1
}
}

Correct approach:

PUT /_ilm/policy/logs_policy
{
"policy": {
"phases": {
"hot": {"actions": {"rollover": {"max_age": "7d"}}},
"warm": {"actions": {"allocate": {"require": {"data": "warm"}}}},
"cold": {"actions": {"allocate": {"require": {"data": "cold"}}}}
}
}
}

PUT /_template/my_template
{
"index_patterns": ["logs-*"],
"settings": {
"number_of_shards": 1,
"index.lifecycle.name": "logs_policy",
"index.lifecycle.rollover_alias": "logs"
}
}

Root cause:

Learners may not realize that ILM policies must be explicitly created and linked to indices to automate tier movement.

#2 Using the same hardware and settings for all tiers wastes resources.

Wrong approach:

All nodes use SSDs and high CPU with frequent refresh intervals regardless of data age.

Correct approach:

Hot nodes use SSDs and fast CPUs with frequent refreshes; warm nodes use slower disks and less CPU; cold nodes use cheapest storage and minimal CPU with infrequent refreshes.

Root cause:

Misunderstanding that different data ages require different hardware and settings leads to inefficient cluster design.

#3 Assuming cold tier data is instantly searchable like hot data.

Wrong approach:

Expecting sub-second search times on cold nodes without tuning or understanding delays.

Correct approach:

Plan for slower search times on cold nodes and configure frozen indices or caching to improve performance when needed.

Root cause:

Not recognizing the performance tradeoffs of storing data on slower hardware causes unrealistic expectations.

Key Takeaways

Hot-warm-cold architecture organizes Elasticsearch data by usage frequency and age to balance speed and cost.

Data moves through tiers automatically using Index Lifecycle Management policies based on age or size.

Each tier uses different hardware and settings optimized for its access patterns and cost constraints.

Understanding this architecture helps design scalable, efficient Elasticsearch clusters that handle large data volumes.

Misconfigurations or misunderstandings can lead to higher costs, slower searches, or wasted resources.