Pipeline testing in Elasticsearch - Mini Project: Build & Apply

Pipeline testing

📖 Scenario: You work with Elasticsearch ingest pipelines to process data before indexing. You want to test a pipeline to ensure it transforms documents correctly.

🎯 Goal: Create a test document and an ingest pipeline, then run the pipeline on the document to see the output.

📋 What You'll Learn

Create a test document with specific fields

Create an ingest pipeline with a processor

Use the _simulate API to test the pipeline with the document

Print the output of the pipeline simulation

💡 Why This Matters

🌍 Real World

Testing ingest pipelines helps ensure data is transformed correctly before indexing in Elasticsearch.

💼 Career

Many jobs working with Elasticsearch require creating and testing ingest pipelines to prepare data for search.

1

Create a test document

Create a JSON object called test_document with these exact fields and values: "user": "alice", "message": "Hello World", and "timestamp": "2024-06-01T12:00:00Z".

Elasticsearch

# Your code here

Need a hint?

Use a Python dictionary with the exact keys and values.

2

Create an ingest pipeline

Create a JSON object called pipeline with an "processors" list containing one processor: a "set" processor that adds a field "processed" with value true.

Elasticsearch

test_document = {
  "user": "alice",
  "message": "Hello World",
  "timestamp": "2024-06-01T12:00:00Z"
}
# Your code here

Need a hint?

Use a dictionary with a processors list containing a set processor.

3

Simulate the pipeline with the test document

Create a JSON object called simulate_request with keys "pipeline" set to the pipeline object, and "docs" set to a list containing one object with key "_source" set to test_document.

Elasticsearch

test_document = {
  "user": "alice",
  "message": "Hello World",
  "timestamp": "2024-06-01T12:00:00Z"
}
pipeline = {
  "processors": [
    {
      "set": {
        "field": "processed",
        "value": true
      }
    }
  ]
}
# Your code here

Need a hint?

Wrap the pipeline and test document in the simulate_request dictionary as shown.

4

Print the simulation output

Print the simulate_request object to show the pipeline simulation input.

Elasticsearch

test_document = {
  "user": "alice",
  "message": "Hello World",
  "timestamp": "2024-06-01T12:00:00Z"
}
pipeline = {
  "processors": [
    {
      "set": {
        "field": "processed",
        "value": true
      }
    }
  ]
}
simulate_request = {
  "pipeline": pipeline,
  "docs": [
    {
      "_source": test_document
    }
  ]
}
# Your code here

Need a hint?

Use print(simulate_request) to display the JSON object.