Index patterns for time-series in Elasticsearch - Step-by-Step Execution

Concept Flow - Index patterns for time-series

Start: Time-series data arrives

↓

Create index pattern with date wildcard

↓

Elasticsearch matches indices by pattern

↓

Query runs on matched indices

↓

Results returned for time range

↓

End

Data arrives continuously. We create an index pattern using date wildcards. Elasticsearch matches indices by this pattern. Queries run on matched indices and return results.

Execution Sample

Elasticsearch

GET /logs-2023.06.*/_search
{
  "query": { "range": { "@timestamp": { "gte": "2023-06-01", "lt": "2023-07-01" } } }
}

This query searches all indices starting with 'logs-2023.06.' for documents in June 2023.

Execution Table

Step	Action	Index Pattern Used	Indices Matched	Query Time Range	Result
1	Receive query	logs-2023.06.*	logs-2023.06.01, logs-2023.06.02, ..., logs-2023.06.30	2023-06-01 to 2023-07-01	Ready to search
2	Elasticsearch matches indices	logs-2023.06.*	logs-2023.06.01, logs-2023.06.02, ..., logs-2023.06.30	2023-06-01 to 2023-07-01	Indices matched successfully
3	Run query on matched indices	logs-2023.06.*	logs-2023.06.01, logs-2023.06.02, ..., logs-2023.06.30	2023-06-01 to 2023-07-01	Documents in June returned
4	Return results	logs-2023.06.*	logs-2023.06.01, logs-2023.06.02, ..., logs-2023.06.30	2023-06-01 to 2023-07-01	Results sent to user

💡 Query completes after searching all indices matching the pattern for the specified time range.

Variable Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	Final
Index Pattern	logs-2023.06.*	logs-2023.06.*	logs-2023.06.*	logs-2023.06.*	logs-2023.06.*
Matched Indices	none	none	logs-2023.06.01, logs-2023.06.02, ..., logs-2023.06.30	logs-2023.06.01, logs-2023.06.02, ..., logs-2023.06.30	logs-2023.06.01, logs-2023.06.02, ..., logs-2023.06.30
Query Time Range	none	2023-06-01 to 2023-07-01	2023-06-01 to 2023-07-01	2023-06-01 to 2023-07-01	2023-06-01 to 2023-07-01
Results	none	none	none	Documents in June returned	Documents in June returned

Key Moments - 3 Insights

Why do we use a wildcard like 'logs-2023.06.*' instead of a single index name?

Because time-series data is split into many daily indices, the wildcard matches all daily indices for June 2023, ensuring the query covers the entire month (see execution_table rows 1 and 2).

What happens if the query time range does not match any indices?

Elasticsearch finds no indices matching the pattern and time range, so the query returns no results. This is shown by no matched indices in variable_tracker after Step 2.

Can the index pattern include multiple months or years?

Yes, by adjusting the wildcard, for example 'logs-2023.*' matches all indices in 2023. The pattern controls which indices Elasticsearch searches (see concept_flow and execution_table).

Visual Quiz - 3 Questions

Look at the execution_table at Step 2, which indices are matched by the pattern 'logs-2023.06.*'?

A logs-2023.06.01 to logs-2023.06.30

B logs-2023.05.01 to logs-2023.05.31

C logs-2022.06.01 to logs-2022.06.30

D logs-2023.07.01 to logs-2023.07.31

💡 Hint

Check the 'Indices Matched' column at Step 2 in the execution_table.

According to variable_tracker, what is the value of 'Results' after Step 3?

A Documents in July

B none

C Documents in June returned

D All documents

💡 Hint

Look at the 'Results' row in variable_tracker after Step 3.

If the index pattern was changed to 'logs-2023.*', how would the matched indices change at Step 2?

A Only June 2023 indices matched

B All indices from 2023 matched

C No indices matched

D Only July 2023 indices matched

💡 Hint

Consider how the wildcard affects the 'Indices Matched' column in execution_table Step 2.

Concept Snapshot

Index patterns use wildcards to match multiple time-series indices.
Example: logs-2023.06.* matches all June 2023 daily indices.
Queries run on matched indices for the requested time range.
This approach efficiently handles large time-series data.
Adjust patterns to cover desired date ranges.

Full Transcript

This visual execution shows how Elasticsearch uses index patterns with wildcards to query time-series data. Data is stored in daily indices named with dates. A pattern like 'logs-2023.06.*' matches all indices for June 2023. When a query runs with a time range, Elasticsearch finds all indices matching the pattern and searches them. Results from all matched indices are combined and returned. Variables like 'Index Pattern', 'Matched Indices', 'Query Time Range', and 'Results' change step-by-step as the query executes. Key points include why wildcards are used, what happens if no indices match, and how to adjust patterns for different date ranges. The quiz tests understanding of matched indices and results at different steps.

Practice

1. What is the main purpose of using index patterns in Elasticsearch for time-series data?

easy

A. To group multiple time-based indexes for easier searching

B. To store data permanently without deletion

C. To create visual dashboards automatically

D. To encrypt data in the indexes

Solution

1. Step 1: Understand index patterns role

   Index patterns help combine many time-based indexes so you can search them all at once.

2. Step 2: Compare options

   Only grouping multiple time-based indexes matches the main purpose; others are unrelated.

3. Final Answer:

   To group multiple time-based indexes for easier searching -> Option A

4. Quick Check:

   Index patterns = group time-based indexes [OK]

Hint: Index patterns group many indexes by time for easy search [OK]

Common Mistakes:

• Thinking index patterns store data permanently
• Confusing index patterns with dashboards
• Assuming index patterns encrypt data

2. Which of the following is the correct syntax for an index pattern matching daily logs from January 2024 in Elasticsearch?

easy

A. logs-2024.01.*

B. logs-2024-01-*

C. logs_2024-01-*

D. logs-01-2024-*

Solution

1. Step 1: Identify date format in index pattern

   Elasticsearch uses dash-separated date parts like YYYY-MM-DD, so logs-2024-01-* matches daily logs in January 2024.

2. Step 2: Check other options

   logs-2024.01.* uses dots instead of dashes, which is incorrect. logs_2024-01-* uses underscore instead of dash. logs-01-2024-* has wrong order of date parts.

3. Final Answer:

   <code>logs-2024-01-*</code> -> Option B

4. Quick Check:

   Date format YYYY-MM-DD with dashes = logs-2024-01-* [OK]

Hint: Use dashes YYYY-MM-DD in index patterns for dates [OK]

Common Mistakes:

• Using dots instead of dashes in date format
• Mixing underscores and dashes incorrectly
• Reordering date parts wrongly

3. Given the index pattern metrics-2023.12.* and indexes metrics-2023.12.01, metrics-2023.11.30, metrics-2023.12.15, which indexes will match the pattern?

medium

A. metrics-2023.11.30 only

B. All three indexes

C. metrics-2023.12.01 and metrics-2023.12.15

D. None of the indexes

Solution

1. Step 1: Understand the pattern metrics-2023.12.*

   This pattern matches indexes starting with 'metrics-2023.12.' followed by anything, so it matches December 2023 indexes.

2. Step 2: Check each index

   metrics-2023.12.01 and metrics-2023.12.15 match December 2023; metrics-2023.11.30 is November, so it does not match.

3. Final Answer:

   <code>metrics-2023.12.01</code> and <code>metrics-2023.12.15</code> -> Option C

4. Quick Check:

   Pattern matches December indexes only = metrics-2023.12.01 and metrics-2023.12.15 [OK]

Hint: Wildcards match only after specified prefix [OK]

Common Mistakes:

• Assuming wildcard matches all months
• Ignoring exact prefix before wildcard
• Confusing dot and dash separators

4. You created an index pattern logs-2023-1* to match logs from January 2023, but it also matches October and November indexes. What is the likely mistake?

medium

A. The dash (-) should be a dot (.) in the pattern

B. Index patterns cannot match months, only years

C. The pattern should be logs-2023-1? instead of *

D. Using wildcard * after 1 matches months starting with 1 and 10

Solution

1. Step 1: Analyze wildcard usage in logs-2023-1*

   The wildcard * matches any characters after '1', so it matches '1...' (January) but also '10', '11' because '10' and '11' start with '1' in pattern matching.

2. Step 2: Understand why October and November match

   Because the pattern matches any string starting with '1', it also matches '10' and '11' incorrectly.

3. Final Answer:

   Using wildcard * after 1 matches months starting with 1 and 10 -> Option D

4. Quick Check:

   Wildcard * after 1 matches more than intended = Using wildcard * after 1 matches months starting with 1 and 10 [OK]

Hint: Wildcard * matches any characters, not just one digit [OK]

Common Mistakes:

• Thinking * matches only one character
• Confusing dash and dot separators
• Believing index patterns can't match months

5. You want to create an index pattern that matches all daily indexes for the year 2024 named like events-2024-03-15. Which pattern correctly matches all days in March 2024 but excludes other months?

hard

A. events-2024-03-??

B. events-2024-03-* and events-2024-04-*

C. events-2024-03-??*

D. events-2024-03-*

Solution

1. Step 1: Understand pattern to match daily dates

   Daily indexes have two-digit day numbers, so ?? matches exactly two characters (days 01 to 31).

2. Step 2: Compare options

   events-2024-03-* uses *, which matches any length, possibly including invalid days. events-2024-03-?? uses ?? to match exactly two digits. events-2024-03-??* allows extra characters after two digits, which is incorrect. events-2024-03-* and events-2024-04-* includes April, which should be excluded.

3. Final Answer:

   <code>events-2024-03-??</code> -> Option A

4. Quick Check:

   Two-question marks match exact two-digit days = events-2024-03-?? [OK]

Hint: Use ?? to match exactly two characters (days) in index pattern [OK]

Common Mistakes:

• Using * which matches any length, not exact days
• Including extra months by mistake
• Using patterns that allow invalid day formats