Index patterns for time-series in Elasticsearch - Deep Dive

Overview - Index patterns for time-series

What is it?

Index patterns for time-series in Elasticsearch are ways to organize and search data that changes over time. They help group many indexes that each hold data for a specific time period, like days or months. This makes it easier to find and analyze data from certain time ranges quickly. Instead of searching all data at once, you search only the relevant time slices.

Why it matters

Without index patterns for time-series, searching through large amounts of time-based data would be slow and inefficient. Imagine trying to find a single day's weather data in a huge pile of years of records without any order. Index patterns let Elasticsearch quickly narrow down to the right data, saving time and computing power. This is crucial for monitoring systems, logs, or any data that grows continuously over time.

Where it fits

Before learning index patterns, you should understand basic Elasticsearch concepts like indexes, documents, and queries. After mastering index patterns, you can explore advanced topics like index lifecycle management, rollups, and optimizing queries for time-series data.

Mental Model

Core Idea

Index patterns group many time-based indexes so you can search and analyze only the relevant time slices efficiently.

Think of it like...

Think of a library where books are organized by year on different shelves. Instead of searching the entire library, you go directly to the shelf for the year you want. Index patterns are like the labels on those shelves that help you find the right books quickly.

┌─────────────────────────────┐
│      Index Pattern          │
│  (e.g., logs-2023.*)        │
├─────────────┬───────────────┤
│ Index: logs-2023.01.01 │ Index: logs-2023.01.02 │
│ Index: logs-2023.01.03 │ Index: logs-2023.01.04 │
└─────────────┴───────────────┘

Searches using the pattern only query the matching daily indexes.

Build-Up - 7 Steps

1

Foundation

Understanding Elasticsearch Index Basics

Concept: Learn what an index is and how data is stored in Elasticsearch.

An index in Elasticsearch is like a database table that stores documents. Each document holds data in JSON format. Indexes help organize data for fast searching. For example, a 'logs' index might store all log entries. You can create many indexes to separate data logically.

Result

You understand that an index is a container for documents and that Elasticsearch uses indexes to organize data.

Knowing what an index is helps you see why grouping indexes by time is useful for managing large datasets.

2

Foundation

What is Time-Series Data?

Concept: Time-series data is information collected over time, often with timestamps.

Examples include temperature readings every hour, website visits per day, or server logs every minute. This data grows continuously and is often analyzed by time ranges, like last week or last month. Handling this data efficiently requires special organization.

Result

You recognize that time-series data is ordered by time and usually large and continuous.

Understanding the nature of time-series data shows why normal indexing might not be enough for performance.

3

Intermediate

Why Use Multiple Indexes for Time-Series?

🤔Before reading on: do you think storing all time-series data in one index or multiple indexes is better for performance? Commit to your answer.

Concept: Splitting time-series data into multiple indexes by time periods improves search speed and management.

Instead of one huge index, Elasticsearch users create daily, weekly, or monthly indexes. For example, logs-2023.01.01, logs-2023.01.02, etc. This limits searches to relevant time slices and helps with deleting old data easily.

Result

You see that multiple indexes reduce search scope and improve efficiency.

Knowing that smaller, time-based indexes speed up queries helps you design scalable systems.

4

Intermediate

How Index Patterns Work in Elasticsearch

🤔Before reading on: do you think an index pattern matches one index or many indexes? Commit to your answer.

Concept: Index patterns use wildcards to match many indexes at once for searching and visualization.

An index pattern like 'logs-2023.*' matches all indexes starting with 'logs-2023.'. When you search using this pattern, Elasticsearch queries all matching indexes. This lets you work with many time-based indexes as if they were one.

Result

You understand that index patterns simplify querying multiple time-based indexes.

Realizing that index patterns act as a shortcut to many indexes helps you manage time-series data easily.

5

Intermediate

Choosing the Right Time Interval for Indexes

🤔Before reading on: do you think daily or monthly indexes are better for all time-series data? Commit to your answer.

Concept: The time interval for indexes balances search speed, storage, and management needs.

Daily indexes give fine control and fast queries for recent data but create many indexes. Monthly indexes reduce index count but can slow searches. Choose based on data volume, query patterns, and retention policies.

Result

You learn to pick index intervals that fit your use case.

Understanding trade-offs in index intervals helps optimize performance and resource use.

6

Advanced

Index Lifecycle Management with Time-Series Patterns

🤔Before reading on: do you think old time-series indexes should be kept forever or deleted/archived? Commit to your answer.

Concept: Index Lifecycle Management (ILM) automates moving indexes through phases like hot, warm, and delete.

ILM policies let you automatically delete or archive old indexes, optimize storage, and improve performance. For example, recent daily indexes stay on fast storage, older ones move to cheaper storage, then get deleted after a set time.

Result

You see how ILM keeps time-series data manageable over time.

Knowing ILM prevents storage bloat and keeps queries fast without manual intervention.

7

Expert

Surprises in Index Pattern Performance and Querying

🤔Before reading on: do you think querying many small indexes is always faster than fewer large indexes? Commit to your answer.

Concept: Querying many small indexes can sometimes be slower due to overhead; Elasticsearch optimizes but has limits.

While smaller indexes reduce data scanned, too many indexes increase coordination overhead. Elasticsearch merges segments internally but too many indexes can cause slower cluster state updates and query planning. Balancing index count and size is key.

Result

You understand that more indexes is not always better and that Elasticsearch has internal optimizations and limits.

Recognizing the trade-off between index count and query overhead helps design efficient time-series storage.

Under the Hood

Elasticsearch stores data in shards within indexes. Each index has metadata and segments holding documents. When you use an index pattern, Elasticsearch expands the pattern to find all matching indexes and queries their shards in parallel. It merges results before returning them. Internally, it manages cluster state with index metadata and optimizes segment merging to keep search fast.

Why designed this way?

Time-series data grows fast and can become huge. Storing all data in one index would slow searches and make management hard. Splitting by time and using patterns lets Elasticsearch scale horizontally and manage data lifecycle. This design balances flexibility, speed, and resource use.

┌───────────────┐
│ Index Pattern │
│  logs-2023.*  │
└──────┬────────┘
       │ Matches multiple indexes
       ▼
┌───────────────┐   ┌───────────────┐   ┌───────────────┐
│ logs-2023.01.01│  │ logs-2023.01.02│  │ logs-2023.01.03│
│ ┌───────────┐ │  │ ┌───────────┐ │  │ ┌───────────┐ │
│ │ Shard 1   │ │  │ │ Shard 1   │ │  │ │ Shard 1   │ │
│ └───────────┘ │  │ └───────────┘ │  │ └───────────┘ │
└───────────────┘   └───────────────┘   └───────────────┘

Elasticsearch queries all shards in parallel and merges results.

Myth Busters - 4 Common Misconceptions

Quick: Do you think using one big index for all time-series data is faster than many small indexes? Commit yes or no.

Common Belief: One big index is always faster because it avoids overhead of multiple indexes.

Reality: Many small time-based indexes are faster for queries limited to recent data because Elasticsearch scans less data and can delete old indexes easily.

Why it matters: Using one big index slows down queries and makes deleting old data complex, causing performance and storage issues.

Quick: Do you think index patterns can match indexes with completely different naming schemes? Commit yes or no.

Common Belief: Index patterns can match any index regardless of naming if they contain the right data.

Reality: Index patterns rely on naming conventions and wildcards; they cannot match indexes with unrelated names.

Why it matters: Without consistent naming, index patterns fail, making searches incomplete or impossible.

Quick: Do you think more indexes always mean better performance? Commit yes or no.

Common Belief: More indexes always improve performance by narrowing search scope.

Reality: Too many indexes increase cluster overhead and can slow down query planning and cluster state updates.

Why it matters: Ignoring this leads to degraded cluster performance and slower searches.

Quick: Do you think deleting documents inside an index is the same as deleting an entire index? Commit yes or no.

Common Belief: Deleting documents inside an index frees up space immediately like deleting the whole index.

Reality: Deleting documents marks them as deleted but space is reclaimed only after segment merges; deleting whole indexes frees space immediately.

Why it matters: Misunderstanding this causes unexpected storage growth and slower searches.

Expert Zone

1

Index patterns depend heavily on consistent naming conventions; even small deviations break pattern matching.

2

Elasticsearch's internal segment merging affects how quickly deleted data space is reclaimed, impacting storage and performance.

3

Cluster state size grows with the number of indexes, so very high index counts can slow cluster operations even if queries are fast.

When NOT to use

Index patterns for time-series are not ideal when data is not time-based or when data volume is small; in such cases, a single index or other data stores like relational databases may be better.

Production Patterns

In production, teams use daily or hourly indexes for logs, combined with ILM policies to delete or archive old data. They use index patterns in Kibana for dashboards and alerts, and optimize index intervals based on query patterns and storage costs.

Connections

Partitioning in Relational Databases

Both split large datasets into smaller parts based on a key (like time) to improve query speed and management.

Understanding partitioning helps grasp why Elasticsearch splits time-series data into multiple indexes.

File Organization in Operating Systems

Index patterns are like directory structures organizing files by date to find them quickly.

Knowing how OS organizes files by folders helps understand how index patterns organize data.

Calendar Systems in Time Management

Both use time intervals (days, months) to segment continuous data or events for easier handling.

Recognizing time segmentation in calendars clarifies why time-series data is split by intervals.

Common Pitfalls

#1 Searching all indexes without using an index pattern.

Wrong approach:

GET /_search
{
"query": { "match_all": {} }
}

Correct approach:

GET /logs-2023.01.*/_search
{
"query": { "match_all": {} }
}

Root cause:

Not using index patterns causes queries to scan all indexes, slowing performance.

#2 Using inconsistent index names breaking the pattern.

Wrong approach:

Indexes named logs-2023-01-01 and log2023.01.02 mixed together with pattern logs-2023.*

Correct approach:

Consistent naming like logs-2023.01.01, logs-2023.01.02 matching pattern logs-2023.*

Root cause:

Inconsistent naming prevents index patterns from matching all relevant indexes.

#3 Creating too many tiny indexes (e.g., per minute) without need.

Wrong approach:

Indexes like logs-2023.01.01-12:00, logs-2023.01.01-12:01, ... for low volume data.

Correct approach:

Use daily or hourly indexes matching data volume and query needs.

Root cause:

Over-splitting increases cluster overhead and slows operations.

Key Takeaways

Index patterns group many time-based indexes to make searching large time-series data efficient.

Splitting data into multiple indexes by time improves query speed and data management.

Consistent naming conventions are essential for index patterns to work correctly.

Index Lifecycle Management automates data retention and storage optimization for time-series indexes.

Balancing index size and count is key to maintaining Elasticsearch cluster performance.

Practice

1. What is the main purpose of using index patterns in Elasticsearch for time-series data?

easy

A. To group multiple time-based indexes for easier searching

B. To store data permanently without deletion

C. To create visual dashboards automatically

D. To encrypt data in the indexes

Solution

1. Step 1: Understand index patterns role

   Index patterns help combine many time-based indexes so you can search them all at once.

2. Step 2: Compare options

   Only grouping multiple time-based indexes matches the main purpose; others are unrelated.

3. Final Answer:

   To group multiple time-based indexes for easier searching -> Option A

4. Quick Check:

   Index patterns = group time-based indexes [OK]

Hint: Index patterns group many indexes by time for easy search [OK]

Common Mistakes:

• Thinking index patterns store data permanently
• Confusing index patterns with dashboards
• Assuming index patterns encrypt data

2. Which of the following is the correct syntax for an index pattern matching daily logs from January 2024 in Elasticsearch?

easy

A. logs-2024.01.*

B. logs-2024-01-*

C. logs_2024-01-*

D. logs-01-2024-*

Solution

1. Step 1: Identify date format in index pattern

   Elasticsearch uses dash-separated date parts like YYYY-MM-DD, so logs-2024-01-* matches daily logs in January 2024.

2. Step 2: Check other options

   logs-2024.01.* uses dots instead of dashes, which is incorrect. logs_2024-01-* uses underscore instead of dash. logs-01-2024-* has wrong order of date parts.

3. Final Answer:

   <code>logs-2024-01-*</code> -> Option B

4. Quick Check:

   Date format YYYY-MM-DD with dashes = logs-2024-01-* [OK]

Hint: Use dashes YYYY-MM-DD in index patterns for dates [OK]

Common Mistakes:

• Using dots instead of dashes in date format
• Mixing underscores and dashes incorrectly
• Reordering date parts wrongly

3. Given the index pattern metrics-2023.12.* and indexes metrics-2023.12.01, metrics-2023.11.30, metrics-2023.12.15, which indexes will match the pattern?

medium

A. metrics-2023.11.30 only

B. All three indexes

C. metrics-2023.12.01 and metrics-2023.12.15

D. None of the indexes

Solution

1. Step 1: Understand the pattern metrics-2023.12.*

   This pattern matches indexes starting with 'metrics-2023.12.' followed by anything, so it matches December 2023 indexes.

2. Step 2: Check each index

   metrics-2023.12.01 and metrics-2023.12.15 match December 2023; metrics-2023.11.30 is November, so it does not match.

3. Final Answer:

   <code>metrics-2023.12.01</code> and <code>metrics-2023.12.15</code> -> Option C

4. Quick Check:

   Pattern matches December indexes only = metrics-2023.12.01 and metrics-2023.12.15 [OK]

Hint: Wildcards match only after specified prefix [OK]

Common Mistakes:

• Assuming wildcard matches all months
• Ignoring exact prefix before wildcard
• Confusing dot and dash separators

4. You created an index pattern logs-2023-1* to match logs from January 2023, but it also matches October and November indexes. What is the likely mistake?

medium

A. The dash (-) should be a dot (.) in the pattern

B. Index patterns cannot match months, only years

C. The pattern should be logs-2023-1? instead of *

D. Using wildcard * after 1 matches months starting with 1 and 10

Solution

1. Step 1: Analyze wildcard usage in logs-2023-1*

   The wildcard * matches any characters after '1', so it matches '1...' (January) but also '10', '11' because '10' and '11' start with '1' in pattern matching.

2. Step 2: Understand why October and November match

   Because the pattern matches any string starting with '1', it also matches '10' and '11' incorrectly.

3. Final Answer:

   Using wildcard * after 1 matches months starting with 1 and 10 -> Option D

4. Quick Check:

   Wildcard * after 1 matches more than intended = Using wildcard * after 1 matches months starting with 1 and 10 [OK]

Hint: Wildcard * matches any characters, not just one digit [OK]

Common Mistakes:

• Thinking * matches only one character
• Confusing dash and dot separators
• Believing index patterns can't match months

5. You want to create an index pattern that matches all daily indexes for the year 2024 named like events-2024-03-15. Which pattern correctly matches all days in March 2024 but excludes other months?

hard

A. events-2024-03-??

B. events-2024-03-* and events-2024-04-*

C. events-2024-03-??*

D. events-2024-03-*

Solution

1. Step 1: Understand pattern to match daily dates

   Daily indexes have two-digit day numbers, so ?? matches exactly two characters (days 01 to 31).

2. Step 2: Compare options

   events-2024-03-* uses *, which matches any length, possibly including invalid days. events-2024-03-?? uses ?? to match exactly two digits. events-2024-03-??* allows extra characters after two digits, which is incorrect. events-2024-03-* and events-2024-04-* includes April, which should be excluded.

3. Final Answer:

   <code>events-2024-03-??</code> -> Option A

4. Quick Check:

   Two-question marks match exact two-digit days = events-2024-03-?? [OK]

Hint: Use ?? to match exactly two characters (days) in index pattern [OK]

Common Mistakes:

• Using * which matches any length, not exact days
• Including extra months by mistake
• Using patterns that allow invalid day formats