Index patterns for time-series in Elasticsearch - Step-by-Step Execution

Concept Flow - Index patterns for time-series

Start: Time-series data arrives

↓

Create index pattern with date wildcard

↓

Elasticsearch matches indices by pattern

↓

Query runs on matched indices

↓

Results returned for time range

↓

End

Data arrives continuously. We create an index pattern using date wildcards. Elasticsearch matches indices by this pattern. Queries run on matched indices and return results.

Execution Sample

Elasticsearch

GET /logs-2023.06.*/_search
{
  "query": { "range": { "@timestamp": { "gte": "2023-06-01", "lt": "2023-07-01" } } }
}

This query searches all indices starting with 'logs-2023.06.' for documents in June 2023.

Execution Table

Step	Action	Index Pattern Used	Indices Matched	Query Time Range	Result
1	Receive query	logs-2023.06.*	logs-2023.06.01, logs-2023.06.02, ..., logs-2023.06.30	2023-06-01 to 2023-07-01	Ready to search
2	Elasticsearch matches indices	logs-2023.06.*	logs-2023.06.01, logs-2023.06.02, ..., logs-2023.06.30	2023-06-01 to 2023-07-01	Indices matched successfully
3	Run query on matched indices	logs-2023.06.*	logs-2023.06.01, logs-2023.06.02, ..., logs-2023.06.30	2023-06-01 to 2023-07-01	Documents in June returned
4	Return results	logs-2023.06.*	logs-2023.06.01, logs-2023.06.02, ..., logs-2023.06.30	2023-06-01 to 2023-07-01	Results sent to user

💡 Query completes after searching all indices matching the pattern for the specified time range.

Variable Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	Final
Index Pattern	logs-2023.06.*	logs-2023.06.*	logs-2023.06.*	logs-2023.06.*	logs-2023.06.*
Matched Indices	none	none	logs-2023.06.01, logs-2023.06.02, ..., logs-2023.06.30	logs-2023.06.01, logs-2023.06.02, ..., logs-2023.06.30	logs-2023.06.01, logs-2023.06.02, ..., logs-2023.06.30
Query Time Range	none	2023-06-01 to 2023-07-01	2023-06-01 to 2023-07-01	2023-06-01 to 2023-07-01	2023-06-01 to 2023-07-01
Results	none	none	none	Documents in June returned	Documents in June returned

Key Moments - 3 Insights

Why do we use a wildcard like 'logs-2023.06.*' instead of a single index name?

Because time-series data is split into many daily indices, the wildcard matches all daily indices for June 2023, ensuring the query covers the entire month (see execution_table rows 1 and 2).

What happens if the query time range does not match any indices?

Elasticsearch finds no indices matching the pattern and time range, so the query returns no results. This is shown by no matched indices in variable_tracker after Step 2.

Can the index pattern include multiple months or years?

Yes, by adjusting the wildcard, for example 'logs-2023.*' matches all indices in 2023. The pattern controls which indices Elasticsearch searches (see concept_flow and execution_table).

Visual Quiz - 3 Questions

Look at the execution_table at Step 2, which indices are matched by the pattern 'logs-2023.06.*'?

A logs-2023.06.01 to logs-2023.06.30

B logs-2023.05.01 to logs-2023.05.31

C logs-2022.06.01 to logs-2022.06.30

D logs-2023.07.01 to logs-2023.07.31

💡 Hint

Check the 'Indices Matched' column at Step 2 in the execution_table.

According to variable_tracker, what is the value of 'Results' after Step 3?

A Documents in July

B none

C Documents in June returned

D All documents

💡 Hint

Look at the 'Results' row in variable_tracker after Step 3.

If the index pattern was changed to 'logs-2023.*', how would the matched indices change at Step 2?

A Only June 2023 indices matched

B All indices from 2023 matched

C No indices matched

D Only July 2023 indices matched

💡 Hint

Consider how the wildcard affects the 'Indices Matched' column in execution_table Step 2.

Concept Snapshot

Index patterns use wildcards to match multiple time-series indices.
Example: logs-2023.06.* matches all June 2023 daily indices.
Queries run on matched indices for the requested time range.
This approach efficiently handles large time-series data.
Adjust patterns to cover desired date ranges.

Full Transcript

This visual execution shows how Elasticsearch uses index patterns with wildcards to query time-series data. Data is stored in daily indices named with dates. A pattern like 'logs-2023.06.*' matches all indices for June 2023. When a query runs with a time range, Elasticsearch finds all indices matching the pattern and searches them. Results from all matched indices are combined and returned. Variables like 'Index Pattern', 'Matched Indices', 'Query Time Range', and 'Results' change step-by-step as the query executes. Key points include why wildcards are used, what happens if no indices match, and how to adjust patterns for different date ranges. The quiz tests understanding of matched indices and results at different steps.