Ingest pipelines in Elasticsearch - Mini Project: Build & Apply

Create and Use an Elasticsearch Ingest Pipeline

📖 Scenario: You work as a data engineer. You need to preprocess incoming log data before storing it in Elasticsearch. This preprocessing will add a timestamp and convert a field to lowercase.

🎯 Goal: Build an Elasticsearch ingest pipeline that adds a timestamp and converts the user field to lowercase. Then simulate sending a document through this pipeline and see the processed output.

📋 What You'll Learn

Create an ingest pipeline named log_pipeline with two processors: set and lowercase

The set processor must add a field ingest_timestamp with the current timestamp

The lowercase processor must convert the user field to lowercase

Simulate ingesting a document with user field set to JohnDoe through the pipeline

Print the resulting document after processing

💡 Why This Matters

🌍 Real World

Ingest pipelines help preprocess and enrich data before indexing in Elasticsearch, making search and analysis more effective.

💼 Career

Data engineers and Elasticsearch administrators use ingest pipelines to automate data transformations and improve data quality.

1

Create the ingest pipeline with a set processor

Create an ingest pipeline named log_pipeline with a set processor that adds a field called ingest_timestamp with the value {{_ingest.timestamp}}.

Elasticsearch

# Create the ingest pipeline named log_pipeline with a set processor
# Your code here

Need a hint?

Use the PUT _ingest/pipeline/log_pipeline API to create the pipeline. The set processor adds a field with the current ingest timestamp.

2

Add a lowercase processor to the pipeline

Update the log_pipeline to add a lowercase processor that converts the user field to lowercase. Keep the existing set processor.

Elasticsearch

PUT _ingest/pipeline/log_pipeline
{
  "description": "Add ingest timestamp and lowercase user",
  "processors": [
    {
      "set": {
        "field": "ingest_timestamp",
        "value": "{{_ingest.timestamp}}"
      }
    },
    {
      "lowercase": {
        "field": "user"
      }
    }
  ]
}
# Your code here

Need a hint?

Add a second processor object with "lowercase" and specify the user field.

3

Simulate ingesting a document through the pipeline

Use the _ingest/pipeline/log_pipeline/_simulate API to send a document with "user": "JohnDoe" through the pipeline.

Elasticsearch

POST _ingest/pipeline/log_pipeline/_simulate
{
  "docs": [
    {
      "_source": {
        "user": "JohnDoe"
      }
    }
  ]
}
# Your code here

Need a hint?

Use the _simulate API with a document containing the user field set to JohnDoe.

4

Print the processed document output

Print the output of the simulation showing the document after processing by the pipeline.

Elasticsearch

# Print the processed document output
# Your code here

Need a hint?

Look at the docs array in the response. The user field should be lowercase and ingest_timestamp should be present.