Index refresh interval in Elasticsearch - Deep Dive

Overview - Index refresh interval

What is it?

The index refresh interval in Elasticsearch is the time period between automatic refreshes of an index. A refresh makes recent changes searchable by creating a new segment visible to search queries. This setting controls how often Elasticsearch makes new data available for search without manual intervention.

Why it matters

Without the index refresh interval, new or updated data would not become searchable automatically, causing delays in seeing fresh information. It balances the need for up-to-date search results with system performance. If refreshes happen too often, it can slow down indexing; if too rare, search results become stale.

Where it fits

Before learning about index refresh interval, you should understand basic Elasticsearch concepts like indices, documents, and segments. After this, you can explore advanced performance tuning, such as bulk indexing strategies and segment merging.

Mental Model

Core Idea

The index refresh interval is the timer that tells Elasticsearch when to make recent changes visible to search by creating a new searchable segment.

Think of it like...

It's like a newspaper printing schedule: the index refresh interval is the time between printing new editions that include the latest news, so readers see fresh stories at regular times.

┌───────────────────────────────┐
│ Elasticsearch Index            │
│                               │
│  ┌───────────────┐            │
│  │ Segment 1     │<──────────── Refresh interval triggers
│  ├───────────────┤            │
│  │ Segment 2     │            │
│  └───────────────┘            │
│       ▲                       │
│       │ New data buffered     │
│       │ until refresh         │
└───────┴───────────────────────┘

Build-Up - 7 Steps

1

Foundation

What is an index refresh in Elasticsearch

Concept: Introduce the basic idea of what a refresh does in Elasticsearch.

In Elasticsearch, data is stored in indices made up of segments. When you add or update documents, these changes are first kept in memory and not immediately searchable. A refresh operation creates a new segment that includes these changes, making them visible to search queries.

Result

After a refresh, new or updated documents become searchable.

Understanding that data changes are not instantly searchable helps explain why refreshes are necessary.

2

Foundation

Default refresh interval and its role

Concept: Explain the default timing and its impact on search and indexing.

By default, Elasticsearch refreshes each index every 1 second. This means new data becomes searchable roughly every second. This default balances freshness of search results with system load from frequent refreshes.

Result

New data is searchable about 1 second after indexing by default.

Knowing the default helps you understand when you might want to adjust the refresh interval.

3

Intermediate

How refresh interval affects indexing performance

🤔Before reading on: do you think a shorter refresh interval always improves search speed or can it slow down indexing? Commit to your answer.

Concept: Explore the tradeoff between refresh frequency and indexing speed.

Frequent refreshes create more segments, which means Elasticsearch spends more time merging and managing them. This can slow down indexing throughput. Conversely, longer intervals reduce refresh overhead but delay when data becomes searchable.

Result

Short refresh intervals improve search freshness but can reduce indexing speed; longer intervals improve indexing speed but delay search visibility.

Understanding this tradeoff helps you tune Elasticsearch for your workload needs.

4

Intermediate

Changing the refresh interval dynamically

🤔Before reading on: do you think you can change the refresh interval while Elasticsearch is running or only at index creation? Commit to your answer.

Concept: Show how to update the refresh interval on an existing index.

You can change the refresh interval of an index anytime using the update settings API. For example, setting it to '-1' disables automatic refreshes, useful during bulk indexing. Later, you can reset it to a positive value to resume automatic refreshes.

Result

Refresh interval can be adjusted on the fly to optimize indexing or search needs.

Knowing you can change this setting dynamically allows flexible performance tuning.

5

Intermediate

Manual refresh and its use cases

Concept: Explain how to trigger a refresh manually and when to do it.

Besides automatic refreshes, you can manually refresh an index using the refresh API. This is useful after bulk indexing when you disabled automatic refreshes to speed up indexing but want to make data searchable immediately afterward.

Result

Manual refresh makes recent changes searchable instantly on demand.

Understanding manual refresh lets you control search visibility precisely during heavy indexing.

6

Advanced

Impact of refresh interval on search consistency

🤔Before reading on: do you think search results always include the latest indexed documents immediately? Commit to your answer.

Concept: Discuss how refresh interval affects what search queries see.

Because refreshes happen periodically, search queries only see data up to the last refresh. Documents indexed after the last refresh are invisible to search until the next refresh. This means search results are eventually consistent, not instantly consistent.

Result

Search results reflect data as of the last refresh, causing a slight delay in visibility.

Knowing this helps set realistic expectations about data freshness in search results.

7

Expert

Internal segment creation and refresh optimization

🤔Before reading on: do you think refresh creates a full new index or just a small segment? Commit to your answer.

Concept: Reveal how refresh creates new segments and how Elasticsearch optimizes this process.

A refresh does not rebuild the entire index but creates a new small segment with recent changes. Elasticsearch merges segments in the background to optimize search speed and storage. Frequent refreshes create many small segments, increasing merge overhead, while infrequent refreshes create fewer, larger segments.

Result

Refresh creates incremental segments; segment merging balances performance and resource use.

Understanding segment mechanics explains why refresh interval tuning affects both indexing and search efficiency.

Under the Hood

Elasticsearch stores data in immutable segments. When documents are indexed, they are first written to an in-memory buffer and transaction log. A refresh operation flushes this buffer, creating a new segment that becomes visible to search. Segments are merged asynchronously to optimize performance. The refresh interval controls how often this flush happens automatically.

Why designed this way?

This design balances write performance and search freshness. Immediate visibility of every change would require costly segment rebuilding. Using segments and periodic refreshes allows fast indexing and near real-time search. Alternatives like immediate refresh would degrade performance significantly.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│ In-Memory     │       │ Refresh       │       │ Searchable    │
│ Buffer       ├──────▶│ Operation     ├──────▶│ Segments      │
└───────────────┘       └───────────────┘       └───────────────┘
        ▲                      │                       ▲
        │                      │                       │
        │  New documents        │                       │
        └──────────────────────┘                       │
                                                       │
                                               Segment merging
                                                       │
                                                       ▼
                                            ┌─────────────────┐
                                            │ Optimized Index │
                                            └─────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does setting refresh interval to 0 make data instantly searchable? Commit yes or no.

Common Belief: Setting the refresh interval to 0 means data is searchable immediately after indexing.

Reality: A refresh interval of 0 is invalid; the minimum is 1ms or disabling with -1. Data becomes searchable only after a refresh occurs, not instantly.

Why it matters: Believing data is instantly searchable leads to incorrect assumptions about search result freshness and can cause bugs in applications expecting immediate visibility.

Quick: Does disabling refresh interval improve both indexing and search speed? Commit yes or no.

Common Belief: Disabling automatic refreshes always improves both indexing and search speed.

Reality: Disabling refreshes improves indexing speed but delays search visibility, making search results stale until a manual refresh or re-enabling automatic refresh.

Why it matters: Misunderstanding this can cause stale search results, confusing users or systems relying on fresh data.

Quick: Does a shorter refresh interval reduce disk space usage? Commit yes or no.

Common Belief: Shorter refresh intervals reduce disk space usage by cleaning segments faster.

Reality: Shorter intervals create more small segments, increasing disk usage and merge overhead. Longer intervals reduce segment count and disk fragmentation.

Why it matters: Ignoring this can lead to unexpected disk space growth and degraded performance.

Quick: Does a refresh operation block indexing or searching? Commit yes or no.

Common Belief: Refresh operations block indexing and searching until complete.

Reality: Refresh is lightweight and does not block indexing or searching; it creates new segments without stopping ongoing operations.

Why it matters: Believing refresh blocks operations may cause unnecessary hesitation in tuning refresh intervals.

Expert Zone

1

Frequent refreshes increase segment count, which can degrade search performance due to overhead in managing many small segments.

2

Disabling refresh during bulk indexing and manually refreshing afterward is a common pattern to maximize indexing throughput without sacrificing search freshness.

3

Refresh interval tuning depends heavily on workload type: real-time search needs favor shorter intervals, while heavy indexing favors longer intervals.

When NOT to use

Do not use very short refresh intervals for heavy bulk indexing workloads; instead, disable automatic refresh and use manual refresh after indexing. For near real-time analytics, consider using the _flush API or other mechanisms to balance durability and visibility.

Production Patterns

In production, teams often set refresh interval to -1 during large data imports to speed indexing, then reset to 1s for normal operation. Monitoring segment count and merge activity helps optimize refresh settings. Some use different refresh intervals per index based on data freshness needs.

Connections

Caching mechanisms

Both manage freshness and performance tradeoffs by controlling update visibility timing.

Understanding refresh intervals helps grasp how caches decide when to update stored data for users.

Eventual consistency in distributed systems

Refresh interval creates a delay before data is visible, similar to how eventual consistency delays data propagation.

Knowing refresh intervals clarifies why search results may lag behind writes, a key concept in distributed data systems.

Print publishing cycles

Both use scheduled updates to batch changes for efficiency and user experience.

Seeing refresh intervals like print cycles helps understand balancing update frequency with resource use.

Common Pitfalls

#1 Setting refresh interval to 0 to get instant search results.

Wrong approach:

PUT /my_index/_settings
{
"index": {
"refresh_interval": "0"
}
}

Correct approach:

PUT /my_index/_settings
{
"index": {
"refresh_interval": "1s"
}
}

Root cause:

Misunderstanding that 0 is invalid and that refresh interval controls timing, not instant visibility.

#2 Disabling refresh during bulk indexing but forgetting to manually refresh afterward.

Wrong approach:

PUT /my_index/_settings
{
"index": {
"refresh_interval": "-1"
}
}

// Bulk indexing done
// No manual refresh called

Correct approach:

PUT /my_index/_settings
{
"index": {
"refresh_interval": "-1"
}
}

// Bulk indexing done
POST /my_index/_refresh

Root cause:

Not realizing that disabling refresh delays search visibility until manual refresh.

#3 Setting very short refresh intervals during heavy indexing causing slow performance.

Wrong approach:

PUT /my_index/_settings
{
"index": {
"refresh_interval": "100ms"
}
}

Correct approach:

PUT /my_index/_settings
{
"index": {
"refresh_interval": "30s"
}
}

Root cause:

Ignoring the overhead of frequent refreshes and segment merges on indexing throughput.

Key Takeaways

The index refresh interval controls how often Elasticsearch makes recent changes searchable by creating new segments.

Balancing refresh interval affects both search freshness and indexing performance, requiring tuning based on workload.

You can dynamically change the refresh interval and manually trigger refreshes to optimize indexing and search visibility.

Refresh operations create new segments incrementally without blocking indexing or searching.

Understanding refresh intervals clarifies why search results are eventually consistent, not instantly updated.

Practice

1. What does the index.refresh_interval setting control in Elasticsearch?

easy

A. The number of shards in the index

B. The size limit of the index

C. How often the index makes new data searchable

D. The maximum number of replicas

Solution

1. Step 1: Understand the role of index.refresh_interval

   This setting controls the frequency at which Elasticsearch refreshes the index to make newly indexed data searchable.

2. Step 2: Compare with other options

   The other options relate to the number of shards, size limits, and replicas, which are unrelated to refresh timing.

3. Final Answer:

   How often the index makes new data searchable -> Option C

4. Quick Check:

   Refresh interval = data searchable frequency [OK]

Hint: Refresh interval means how often new data appears [OK]

Common Mistakes:

• Confusing refresh interval with shard count
• Thinking it controls index size
• Mixing it up with replica settings

2. Which of the following is the correct way to set the refresh interval to 5 seconds in an Elasticsearch index settings JSON?

easy

A. { "refresh_interval": 5 }

B. { "index": { "refresh_interval": "5000" } }

C. { "index": { "refresh_interval": 5 } }

D. { "index": { "refresh_interval": "5s" } }

Solution

1. Step 1: Identify correct JSON structure for refresh interval

   The refresh interval must be a string with time units, inside the index object.

2. Step 2: Validate options

   { "index": { "refresh_interval": "5s" } } uses "5s" (5 seconds) correctly as a string with units. Plain numbers like 5 without units are invalid. { "index": { "refresh_interval": "5000" } } uses "5000" without units, which is incorrect. Missing the index object is also invalid.

3. Final Answer:

   { "index": { "refresh_interval": "5s" } } -> Option D

4. Quick Check:

   Refresh interval needs string with units [OK]

Hint: Use string with time unit like "5s" for refresh interval [OK]

Common Mistakes:

• Using number without time unit
• Placing refresh_interval outside index object
• Using milliseconds as plain number string

3. Given the following index setting:

{ "index": { "refresh_interval": "30s" } }

What happens if you index a document and immediately search for it within 10 seconds?

medium

A. The document will not be found until after 30 seconds

B. The document will be found immediately

C. The document will never be found

D. The document will be found after 10 seconds

Solution

1. Step 1: Understand refresh interval effect on search

   With a 30-second refresh interval, Elasticsearch refreshes the index every 30 seconds to make new data searchable.

2. Step 2: Analyze timing of search after indexing

   If you search within 10 seconds, the index has not refreshed yet, so the new document is not visible.

3. Final Answer:

   The document will not be found until after 30 seconds -> Option A

4. Quick Check:

   Refresh interval delays new data visibility [OK]

Hint: Search before refresh interval means no new data visible [OK]

Common Mistakes:

• Assuming instant search visibility
• Confusing refresh interval with indexing speed
• Thinking document is never searchable

4. You set index.refresh_interval to -1 to disable automatic refresh during heavy indexing. After indexing, you want to make all data searchable immediately. What is the correct way to do this?

medium

A. Set index.refresh_interval back to 0

B. Run a manual _refresh API call on the index

C. Restart the Elasticsearch cluster

D. Delete and recreate the index

Solution

1. Step 1: Understand disabling refresh with -1

   Setting refresh_interval to -1 disables automatic refresh, so new data is not searchable until manually refreshed.

2. Step 2: Identify how to make data searchable immediately

   Using the _refresh API triggers an immediate refresh, making all indexed data searchable without restarting or recreating.

3. Final Answer:

   Run a manual _refresh API call on the index -> Option B

4. Quick Check:

   Manual refresh needed when auto refresh disabled [OK]

Hint: Use _refresh API to make data searchable after disabling refresh [OK]

Common Mistakes:

• Setting refresh_interval to 0 instead of calling _refresh
• Restarting cluster unnecessarily
• Deleting index instead of refreshing

5. You have an index with heavy write load and want to optimize indexing speed without losing data visibility for search. Which approach best balances performance and freshness?

hard

A. Set index.refresh_interval to a higher value during indexing, then manually refresh after bulk load

B. Set index.refresh_interval to 0 to refresh after every write

C. Disable refresh permanently by setting index.refresh_interval to -1 and never refresh

D. Delete the index and create a new one for each bulk load

Solution

1. Step 1: Understand trade-off between refresh interval and indexing speed

   Frequent refreshes slow indexing but improve data freshness; less frequent refreshes speed indexing but delay visibility.

2. Step 2: Choose best practice for heavy write load

   Setting a higher refresh interval during bulk indexing reduces refresh overhead, then manually refreshing after bulk load balances speed and search freshness.

3. Final Answer:

   Set index.refresh_interval to a higher value during indexing, then manually refresh after bulk load -> Option A

4. Quick Check:

   Adjust refresh interval for bulk, then manual refresh [OK]

Hint: Increase refresh interval during bulk, refresh manually after [OK]

Common Mistakes:

• Setting refresh_interval to 0 causes slow indexing
• Disabling refresh permanently loses search freshness
• Deleting index wastes resources