Index patterns for time-series in Elasticsearch - Deep Dive

Overview - Index patterns for time-series

What is it?

Index patterns for time-series in Elasticsearch are ways to organize and search data that changes over time. They help group many indexes that each hold data for a specific time period, like days or months. This makes it easier to find and analyze data from certain time ranges quickly. Instead of searching all data at once, you search only the relevant time slices.

Why it matters

Without index patterns for time-series, searching through large amounts of time-based data would be slow and inefficient. Imagine trying to find a single day's weather data in a huge pile of years of records without any order. Index patterns let Elasticsearch quickly narrow down to the right data, saving time and computing power. This is crucial for monitoring systems, logs, or any data that grows continuously over time.

Where it fits

Before learning index patterns, you should understand basic Elasticsearch concepts like indexes, documents, and queries. After mastering index patterns, you can explore advanced topics like index lifecycle management, rollups, and optimizing queries for time-series data.

Mental Model

Core Idea

Index patterns group many time-based indexes so you can search and analyze only the relevant time slices efficiently.

Think of it like...

Think of a library where books are organized by year on different shelves. Instead of searching the entire library, you go directly to the shelf for the year you want. Index patterns are like the labels on those shelves that help you find the right books quickly.

┌─────────────────────────────┐
│      Index Pattern          │
│  (e.g., logs-2023.*)        │
├─────────────┬───────────────┤
│ Index: logs-2023.01.01 │ Index: logs-2023.01.02 │
│ Index: logs-2023.01.03 │ Index: logs-2023.01.04 │
└─────────────┴───────────────┘

Searches using the pattern only query the matching daily indexes.

Build-Up - 7 Steps

1

Foundation

Understanding Elasticsearch Index Basics

Concept: Learn what an index is and how data is stored in Elasticsearch.

An index in Elasticsearch is like a database table that stores documents. Each document holds data in JSON format. Indexes help organize data for fast searching. For example, a 'logs' index might store all log entries. You can create many indexes to separate data logically.

Result

You understand that an index is a container for documents and that Elasticsearch uses indexes to organize data.

Knowing what an index is helps you see why grouping indexes by time is useful for managing large datasets.

2

Foundation

What is Time-Series Data?

Concept: Time-series data is information collected over time, often with timestamps.

Examples include temperature readings every hour, website visits per day, or server logs every minute. This data grows continuously and is often analyzed by time ranges, like last week or last month. Handling this data efficiently requires special organization.

Result

You recognize that time-series data is ordered by time and usually large and continuous.

Understanding the nature of time-series data shows why normal indexing might not be enough for performance.

3

Intermediate

Why Use Multiple Indexes for Time-Series?

🤔Before reading on: do you think storing all time-series data in one index or multiple indexes is better for performance? Commit to your answer.

Concept: Splitting time-series data into multiple indexes by time periods improves search speed and management.

Instead of one huge index, Elasticsearch users create daily, weekly, or monthly indexes. For example, logs-2023.01.01, logs-2023.01.02, etc. This limits searches to relevant time slices and helps with deleting old data easily.

Result

You see that multiple indexes reduce search scope and improve efficiency.

Knowing that smaller, time-based indexes speed up queries helps you design scalable systems.

4

Intermediate

How Index Patterns Work in Elasticsearch

🤔Before reading on: do you think an index pattern matches one index or many indexes? Commit to your answer.

Concept: Index patterns use wildcards to match many indexes at once for searching and visualization.

An index pattern like 'logs-2023.*' matches all indexes starting with 'logs-2023.'. When you search using this pattern, Elasticsearch queries all matching indexes. This lets you work with many time-based indexes as if they were one.

Result

You understand that index patterns simplify querying multiple time-based indexes.

Realizing that index patterns act as a shortcut to many indexes helps you manage time-series data easily.

5

Intermediate

Choosing the Right Time Interval for Indexes

🤔Before reading on: do you think daily or monthly indexes are better for all time-series data? Commit to your answer.

Concept: The time interval for indexes balances search speed, storage, and management needs.

Daily indexes give fine control and fast queries for recent data but create many indexes. Monthly indexes reduce index count but can slow searches. Choose based on data volume, query patterns, and retention policies.

Result

You learn to pick index intervals that fit your use case.

Understanding trade-offs in index intervals helps optimize performance and resource use.

6

Advanced

Index Lifecycle Management with Time-Series Patterns

🤔Before reading on: do you think old time-series indexes should be kept forever or deleted/archived? Commit to your answer.

Concept: Index Lifecycle Management (ILM) automates moving indexes through phases like hot, warm, and delete.

ILM policies let you automatically delete or archive old indexes, optimize storage, and improve performance. For example, recent daily indexes stay on fast storage, older ones move to cheaper storage, then get deleted after a set time.

Result

You see how ILM keeps time-series data manageable over time.

Knowing ILM prevents storage bloat and keeps queries fast without manual intervention.

7

Expert

Surprises in Index Pattern Performance and Querying

🤔Before reading on: do you think querying many small indexes is always faster than fewer large indexes? Commit to your answer.

Concept: Querying many small indexes can sometimes be slower due to overhead; Elasticsearch optimizes but has limits.

While smaller indexes reduce data scanned, too many indexes increase coordination overhead. Elasticsearch merges segments internally but too many indexes can cause slower cluster state updates and query planning. Balancing index count and size is key.

Result

You understand that more indexes is not always better and that Elasticsearch has internal optimizations and limits.

Recognizing the trade-off between index count and query overhead helps design efficient time-series storage.

Under the Hood

Elasticsearch stores data in shards within indexes. Each index has metadata and segments holding documents. When you use an index pattern, Elasticsearch expands the pattern to find all matching indexes and queries their shards in parallel. It merges results before returning them. Internally, it manages cluster state with index metadata and optimizes segment merging to keep search fast.

Why designed this way?

Time-series data grows fast and can become huge. Storing all data in one index would slow searches and make management hard. Splitting by time and using patterns lets Elasticsearch scale horizontally and manage data lifecycle. This design balances flexibility, speed, and resource use.

┌───────────────┐
│ Index Pattern │
│  logs-2023.*  │
└──────┬────────┘
       │ Matches multiple indexes
       ▼
┌───────────────┐   ┌───────────────┐   ┌───────────────┐
│ logs-2023.01.01│  │ logs-2023.01.02│  │ logs-2023.01.03│
│ ┌───────────┐ │  │ ┌───────────┐ │  │ ┌───────────┐ │
│ │ Shard 1   │ │  │ │ Shard 1   │ │  │ │ Shard 1   │ │
│ └───────────┘ │  │ └───────────┘ │  │ └───────────┘ │
└───────────────┘   └───────────────┘   └───────────────┘

Elasticsearch queries all shards in parallel and merges results.

Myth Busters - 4 Common Misconceptions

Quick: Do you think using one big index for all time-series data is faster than many small indexes? Commit yes or no.

Common Belief: One big index is always faster because it avoids overhead of multiple indexes.

Reality: Many small time-based indexes are faster for queries limited to recent data because Elasticsearch scans less data and can delete old indexes easily.

Why it matters: Using one big index slows down queries and makes deleting old data complex, causing performance and storage issues.

Quick: Do you think index patterns can match indexes with completely different naming schemes? Commit yes or no.

Common Belief: Index patterns can match any index regardless of naming if they contain the right data.

Reality: Index patterns rely on naming conventions and wildcards; they cannot match indexes with unrelated names.

Why it matters: Without consistent naming, index patterns fail, making searches incomplete or impossible.

Quick: Do you think more indexes always mean better performance? Commit yes or no.

Common Belief: More indexes always improve performance by narrowing search scope.

Reality: Too many indexes increase cluster overhead and can slow down query planning and cluster state updates.

Why it matters: Ignoring this leads to degraded cluster performance and slower searches.

Quick: Do you think deleting documents inside an index is the same as deleting an entire index? Commit yes or no.

Common Belief: Deleting documents inside an index frees up space immediately like deleting the whole index.

Reality: Deleting documents marks them as deleted but space is reclaimed only after segment merges; deleting whole indexes frees space immediately.

Why it matters: Misunderstanding this causes unexpected storage growth and slower searches.

Expert Zone

1

Index patterns depend heavily on consistent naming conventions; even small deviations break pattern matching.

2

Elasticsearch's internal segment merging affects how quickly deleted data space is reclaimed, impacting storage and performance.

3

Cluster state size grows with the number of indexes, so very high index counts can slow cluster operations even if queries are fast.

When NOT to use

Index patterns for time-series are not ideal when data is not time-based or when data volume is small; in such cases, a single index or other data stores like relational databases may be better.

Production Patterns

In production, teams use daily or hourly indexes for logs, combined with ILM policies to delete or archive old data. They use index patterns in Kibana for dashboards and alerts, and optimize index intervals based on query patterns and storage costs.

Connections

Partitioning in Relational Databases

Both split large datasets into smaller parts based on a key (like time) to improve query speed and management.

Understanding partitioning helps grasp why Elasticsearch splits time-series data into multiple indexes.

File Organization in Operating Systems

Index patterns are like directory structures organizing files by date to find them quickly.

Knowing how OS organizes files by folders helps understand how index patterns organize data.

Calendar Systems in Time Management

Both use time intervals (days, months) to segment continuous data or events for easier handling.

Recognizing time segmentation in calendars clarifies why time-series data is split by intervals.

Common Pitfalls

#1 Searching all indexes without using an index pattern.

Wrong approach:

GET /_search
{
"query": { "match_all": {} }
}

Correct approach:

GET /logs-2023.01.*/_search
{
"query": { "match_all": {} }
}

Root cause:

Not using index patterns causes queries to scan all indexes, slowing performance.

#2 Using inconsistent index names breaking the pattern.

Wrong approach:

Indexes named logs-2023-01-01 and log2023.01.02 mixed together with pattern logs-2023.*

Correct approach:

Consistent naming like logs-2023.01.01, logs-2023.01.02 matching pattern logs-2023.*

Root cause:

Inconsistent naming prevents index patterns from matching all relevant indexes.

#3 Creating too many tiny indexes (e.g., per minute) without need.

Wrong approach:

Indexes like logs-2023.01.01-12:00, logs-2023.01.01-12:01, ... for low volume data.

Correct approach:

Use daily or hourly indexes matching data volume and query needs.

Root cause:

Over-splitting increases cluster overhead and slows operations.

Key Takeaways

Index patterns group many time-based indexes to make searching large time-series data efficient.

Splitting data into multiple indexes by time improves query speed and data management.

Consistent naming conventions are essential for index patterns to work correctly.

Index Lifecycle Management automates data retention and storage optimization for time-series indexes.

Balancing index size and count is key to maintaining Elasticsearch cluster performance.