Runtime fields in Elasticsearch - Deep Dive

Overview - Runtime fields

What is it?

Runtime fields in Elasticsearch are fields that are computed on the fly when you run a search query. They do not exist in the stored data but are created dynamically using scripts or expressions. This lets you add new fields or transform existing data without changing the original documents. Runtime fields help you explore and analyze data flexibly without reindexing.

Why it matters

Without runtime fields, you would need to reindex your entire dataset every time you want to add or change a field. This can be slow, costly, and sometimes impossible if you don't control the data source. Runtime fields let you experiment and adapt your data views instantly, saving time and resources while keeping your original data untouched.

Where it fits

Before learning runtime fields, you should understand Elasticsearch basics like documents, fields, and queries. After runtime fields, you can explore advanced scripting, performance tuning, and index mappings. Runtime fields fit between basic querying and advanced data modeling in Elasticsearch.

Mental Model

Core Idea

Runtime fields are like temporary labels you create on your data at search time, letting you see or use new information without changing the stored data.

Think of it like...

Imagine you have a photo album with pictures (your stored data). Runtime fields are like putting sticky notes on photos when you look at them, adding comments or tags that aren't part of the original photo but help you find or understand them better.

┌─────────────────────────────┐
│       Stored Documents       │
│  (fixed data on disk)        │
└─────────────┬───────────────┘
              │
              ▼
┌─────────────────────────────┐
│      Runtime Fields Layer    │
│  (computed during search)    │
└─────────────┬───────────────┘
              │
              ▼
┌─────────────────────────────┐
│        Search Results        │
│  (includes runtime fields)   │
└─────────────────────────────┘

Build-Up - 7 Steps

1

Foundation

What are runtime fields

Concept: Introduction to runtime fields as dynamic fields computed during search.

Runtime fields are fields you define in your search queries that Elasticsearch calculates on the fly. They don't exist in your stored documents but appear in the search results as if they were real fields. You write simple scripts or expressions to tell Elasticsearch how to compute these fields.

Result

You can see new fields in your search results without changing or reindexing your data.

Understanding runtime fields lets you add flexible data views instantly, which is powerful for exploration and quick analysis.

2

Foundation

How runtime fields differ from stored fields

Concept: Distinguishing between stored fields and runtime fields in Elasticsearch.

Stored fields are part of your documents saved on disk. They are fixed until you reindex or update the document. Runtime fields are calculated each time you run a query and do not consume storage space. They allow temporary transformations or new data without permanent changes.

Result

You realize runtime fields save storage and allow dynamic data views, unlike stored fields.

Knowing this difference helps you decide when to use runtime fields versus changing your index.

3

Intermediate

Defining runtime fields with painless scripts

🤔Before reading on: do you think runtime fields can use any programming language or only specific ones? Commit to your answer.

Concept: Using Elasticsearch's Painless scripting language to define runtime fields.

Runtime fields use Painless, a safe and fast scripting language built into Elasticsearch. You write expressions or small scripts to compute values. For example, you can create a runtime field that calculates the length of a text field or converts a string to uppercase.

Result

You can create custom fields that transform or compute data dynamically during search.

Understanding Painless scripting is key to unlocking the full power of runtime fields.

4

Intermediate

Using runtime fields in queries and aggregations

🤔Before reading on: do you think runtime fields can be used in aggregations like normal fields? Commit to your answer.

Concept: Runtime fields can be used in search queries, filters, and aggregations just like stored fields.

Once defined, runtime fields behave like normal fields in your queries. You can filter documents based on runtime field values or aggregate data (like counting or averaging) using runtime fields. This lets you analyze data in new ways without changing your index.

Result

Your queries and aggregations can include dynamic data computed at search time.

Knowing runtime fields integrate fully with queries and aggregations expands your analysis capabilities.

5

Intermediate

Performance considerations of runtime fields

🤔Before reading on: do you think runtime fields are faster, slower, or the same speed as stored fields? Commit to your answer.

Concept: Runtime fields add computation overhead during search, affecting performance.

Because runtime fields are calculated on the fly, they require CPU time during each search. This can slow down queries compared to using stored fields. The complexity of the script and the number of documents searched affect the impact. Use runtime fields for flexibility but monitor performance.

Result

You understand runtime fields trade flexibility for some query speed.

Knowing the performance cost helps you balance flexibility and speed in your Elasticsearch usage.

6

Advanced

Runtime fields in index mappings and scripts

🤔Before reading on: do you think runtime fields can be permanently added to an index mapping? Commit to your answer.

Concept: Runtime fields can be defined in index mappings to be reusable across queries.

You can add runtime fields to your index mapping so they exist for all queries without redefining them each time. This centralizes the logic and makes runtime fields part of your data model, though still computed at search time. You write the Painless script once in the mapping.

Result

Runtime fields become reusable and consistent across your Elasticsearch queries.

Knowing runtime fields can be part of mappings helps you design flexible yet maintainable data models.

7

Expert

Advanced runtime field scripting and limitations

🤔Before reading on: do you think runtime fields can access external data or only the current document? Commit to your answer.

Concept: Runtime fields have scripting limits and cannot access external data or modify stored documents.

Runtime field scripts run per document and can only access that document's data. They cannot call external services or change stored data. Complex logic must fit within Painless's sandbox. Also, some data types or operations are unsupported or slow. Understanding these limits helps avoid surprises.

Result

You write efficient, safe runtime field scripts that respect Elasticsearch constraints.

Knowing runtime fields' boundaries prevents misuse and performance pitfalls in production.

Under the Hood

When you run a search, Elasticsearch loads stored documents from disk. For each document, it runs the runtime field script using the document's data as input. The script computes the field value dynamically in memory. This value is then used in filtering, sorting, or aggregations as if it were a stored field. The process repeats for every matching document during the query execution.

Why designed this way?

Runtime fields were designed to provide flexibility without costly reindexing. Reindexing large datasets is slow and resource-intensive. By computing fields at search time, Elasticsearch lets users experiment and adapt quickly. The design balances flexibility with performance by limiting scripts to safe, per-document computations.

┌───────────────┐
│ Search Query  │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Load Document │
│  from Storage │
└──────┬────────┘
       │
       ▼
┌─────────────────────────────┐
│ Run Runtime Field Script     │
│ (Painless script per doc)    │
└──────┬──────────────────────┘
       │
       ▼
┌───────────────┐
│ Use Computed  │
│ Field in Query│
└───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Do runtime fields permanently add data to your documents? Commit yes or no.

Common Belief: Runtime fields add new data permanently to documents like stored fields.

Reality: Runtime fields only compute data during search and do not change stored documents.

Why it matters: Thinking runtime fields change data can lead to confusion about data consistency and indexing.

Quick: Are runtime fields always faster than stored fields? Commit yes or no.

Common Belief: Runtime fields are faster because they avoid storage overhead.

Reality: Runtime fields are slower because they compute values on the fly during each search.

Why it matters: Assuming runtime fields are faster can cause unexpected slow queries in production.

Quick: Can runtime fields access data from other documents or external sources? Commit yes or no.

Common Belief: Runtime fields can access any data, including external sources or other documents.

Reality: Runtime fields can only access the current document's data and cannot call external services.

Why it matters: Expecting external data access can lead to failed scripts and errors.

Quick: Do runtime fields replace the need for index mappings? Commit yes or no.

Common Belief: Runtime fields make index mappings unnecessary.

Reality: Runtime fields complement but do not replace index mappings; mappings define stored data structure.

Why it matters: Ignoring mappings can cause data modeling issues and limit query performance.

Expert Zone

1

Runtime fields can be combined with stored fields to optimize performance and flexibility, using runtime fields only for rarely needed dynamic data.

2

Scripts in runtime fields run in a sandboxed environment with limited access to ensure security and stability, which restricts some programming constructs.

3

Defining runtime fields in index mappings allows sharing logic across queries but requires careful versioning and testing to avoid breaking searches.

When NOT to use

Avoid runtime fields when you need high query performance on frequently used fields or when the field value rarely changes; instead, use stored fields with proper indexing. Also, do not use runtime fields for complex joins or external data lookups; use ingest pipelines or external processing instead.

Production Patterns

In production, runtime fields are often used for quick data exploration, temporary fixes, or adding experimental fields without reindexing. They are combined with stored fields for performance and defined in mappings for consistency. Monitoring query performance and script complexity is standard practice to avoid slowdowns.

Connections

Computed Columns in SQL

Runtime fields are similar to computed columns that calculate values on query time rather than storing them.

Understanding computed columns helps grasp how runtime fields provide dynamic data without storage overhead.

Lazy Evaluation in Programming

Runtime fields use lazy evaluation by computing values only when needed during search.

Knowing lazy evaluation clarifies why runtime fields save resources until the data is actually requested.

Sticky Notes on Physical Documents

Like adding temporary notes to documents without altering the original, runtime fields add temporary data views.

This connection helps appreciate the non-destructive and flexible nature of runtime fields.

Common Pitfalls

#1 Using complex scripts that slow down queries significantly.

Wrong approach:

{
"runtime_mappings": {
"slow_field": {
"type": "keyword",
"script": "for (int i=0; i<1000000; i++) { emit('value'); }"
}
}
}

Correct approach:

{
"runtime_mappings": {
"simple_field": {
"type": "keyword",
"script": "emit(doc['existing_field'].value)"
}
}
}

Root cause:

Misunderstanding that runtime field scripts run per document and heavy loops cause performance issues.

#2 Expecting runtime fields to update stored data permanently.

Wrong approach:

Using runtime fields to try to fix data errors permanently without reindexing.

Correct approach:

Fix data errors by reindexing or updating documents; use runtime fields only for temporary views.

Root cause:

Confusing runtime fields as a data storage mechanism rather than a dynamic computation.

#3 Defining runtime fields with unsupported data types or operations.

Wrong approach:

{
"runtime_mappings": {
"unsupported_field": {
"type": "geo_point",
"script": "emit('invalid')"
}
}
}

Correct approach:

{
"runtime_mappings": {
"valid_field": {
"type": "keyword",
"script": "emit(doc['text_field'].value)"
}
}
}

Root cause:

Not knowing runtime fields support only certain data types and operations.

Key Takeaways

Runtime fields let you add or transform data dynamically during search without changing stored documents.

They use Painless scripts to compute values per document at query time, offering flexibility but with some performance cost.

Runtime fields can be defined temporarily in queries or permanently in index mappings for reuse.

They cannot modify stored data or access external sources, and complex scripts can slow down searches.

Understanding runtime fields helps you explore data quickly and design adaptable Elasticsearch solutions.