Ingest processors (grok, date, rename) in Elasticsearch

Introduction

Ingest processors help you change or organize data before saving it. They make data easier to search and understand.

You want to extract parts of a text message into separate fields.

You need to convert a text date into a real date format.

You want to change a field name to a clearer or standard name.

Syntax

Elasticsearch

{
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": ["%{COMMONAPACHELOG}"]
      }
    },
    {
      "date": {
        "field": "timestamp",
        "formats": ["dd/MMM/yyyy:HH:mm:ss Z"]
      }
    },
    {
      "rename": {
        "field": "old_field_name",
        "target_field": "new_field_name"
      }
    }
  ]
}

The grok processor extracts data using patterns from text.

The date processor converts text to date format for better sorting and searching.

The rename processor changes field names to keep data clear.

Examples

This grok example extracts client IP, method, and request path from a log message.

Elasticsearch

{
  "grok": {
    "field": "log",
    "patterns": ["%{IP:client} %{WORD:method} %{URIPATHPARAM:request}"]
  }
}

This date processor converts an ISO8601 string into a date object.

Elasticsearch

{
  "date": {
    "field": "event_time",
    "formats": ["ISO8601"]
  }
}

This rename processor changes the field name from 'temp' to 'temperature'.

Elasticsearch

{
  "rename": {
    "field": "temp",
    "target_field": "temperature"
  }
}

Sample Program

This pipeline extracts fields from a common Apache log message, converts the timestamp to a date, and renames the client IP field to 'ip_address' for clarity.

Elasticsearch

{
  "description": "Example pipeline using grok, date, and rename processors",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": ["%{COMMONAPACHELOG}"]
      }
    },
    {
      "date": {
        "field": "timestamp",
        "formats": ["dd/MMM/yyyy:HH:mm:ss Z"]
      }
    },
    {
      "rename": {
        "field": "clientip",
        "target_field": "ip_address"
      }
    }
  ]
}

Important Notes

Grok patterns are reusable templates to find parts of text easily.

Date formats must match the text exactly or the processor will fail.

Renaming helps keep your data consistent and easy to understand.

Summary

Ingest processors change data before saving it to make searching easier.

Grok extracts parts of text, date converts text to date, rename changes field names.

Use these processors together to clean and organize your data automatically.