Ingest pipelines in Elasticsearch

Introduction

Ingest pipelines help you prepare and change data before saving it in Elasticsearch. They make data clean and ready to use.

You want to add or change fields in your data automatically before storing it.

You need to remove or fix wrong data before it goes into Elasticsearch.

You want to convert data formats, like dates or numbers, during data input.

You want to enrich data by adding information from other sources before saving.

You want to log or track data changes as it enters Elasticsearch.

Syntax

Elasticsearch

{
  "description": "optional description",
  "processors": [
    {
      "processor_type": {
        "field": "field_name",
        "target_field": "new_field_name",
        "other_options": "value"
      }
    },
    {
      "another_processor": {
        "field": "field_name",
        "options": "value"
      }
    }
  ]
}

Processors run in order, one after another.

You can use many types of processors like set, rename, remove, convert, and more.

Examples

This pipeline adds a new field called new_field with the value "hello" to each document.

Elasticsearch

{
  "description": "Add a new field",
  "processors": [
    {
      "set": {
        "field": "new_field",
        "value": "hello"
      }
    }
  ]
}

This pipeline changes the name of a field from old_name to new_name.

Elasticsearch

{
  "processors": [
    {
      "rename": {
        "field": "old_name",
        "target_field": "new_name"
      }
    }
  ]
}

This pipeline converts the age field to an integer type.

Elasticsearch

{
  "processors": [
    {
      "convert": {
        "field": "age",
        "type": "integer"
      }
    }
  ]
}

Sample Program

This example creates a pipeline named sample_pipeline that adds a status field with value "new" and renames the user field to username. Then it simulates running this pipeline on a document.

Elasticsearch

PUT _ingest/pipeline/sample_pipeline
{
  "description": "Pipeline to add and rename fields",
  "processors": [
    {
      "set": {
        "field": "status",
        "value": "new"
      }
    },
    {
      "rename": {
        "field": "user",
        "target_field": "username"
      }
    }
  ]
}

POST _ingest/pipeline/sample_pipeline/_simulate
{
  "docs": [
    {
      "_source": {
        "user": "alice",
        "message": "Hello world"
      }
    }
  ]
}

Important Notes

Use the _simulate API to test pipelines without indexing data.

Processors run in the order you list them, so plan carefully.

If a processor fails, the whole pipeline stops unless you handle errors.

Summary

Ingest pipelines let you change data before saving it in Elasticsearch.

You create pipelines with processors that do tasks like add, rename, or convert fields.

Test pipelines with the simulate API to see changes before indexing.