Application performance monitoring in Elasticsearch - Step-by-Step Execution

Concept Flow - Application performance monitoring

Start Application

↓

Collect Metrics & Traces

↓

Send Data to Elasticsearch

↓

Store & Index Data

↓

Analyze & Visualize in Kibana

↓

Detect Issues & Alert

↓

Optimize Application

↓

Repeat Monitoring Cycle

This flow shows how application data is collected, sent to Elasticsearch, stored, analyzed, and used to improve performance continuously.

Execution Sample

Elasticsearch

POST /apm-7.17.0/_doc
{
  "transaction": {
    "name": "GET /api/user",
    "duration": 120
  },
  "@timestamp": "2024-06-01T12:00:00Z"
}

This example sends a transaction document to Elasticsearch APM index representing a user API call with its duration.

Execution Table

Step	Action	Data Sent	Elasticsearch Response	System State Change
1	Start Application	N/A	N/A	Application running, ready to collect data
2	Collect Metrics & Traces	Transaction data: {name: 'GET /api/user', duration: 120}	N/A	Metrics collected in memory
3	Send Data to Elasticsearch	POST /apm-7.17.0/_doc with transaction data	201 Created	Data indexed in Elasticsearch
4	Store & Index Data	N/A	N/A	Data stored and searchable in APM index
5	Analyze & Visualize in Kibana	Query APM index	Query results with transaction metrics	Dashboard updated with latest data
6	Detect Issues & Alert	N/A	N/A	Alerts triggered if thresholds exceeded
7	Optimize Application	N/A	N/A	Developers improve code based on insights
8	Repeat Monitoring Cycle	N/A	N/A	Continuous monitoring ongoing

💡 Monitoring continues indefinitely to track application performance and detect issues

Variable Tracker

Variable	Start	After Step 2	After Step 3	After Step 5	Final
transaction_data	None	{"name": "GET /api/user", "duration": 120}	Sent to Elasticsearch	Stored and queried in Kibana	Used for alerts and optimization
application_state	Stopped	Running	Running	Running	Running

Key Moments - 3 Insights

Why do we send transaction data to Elasticsearch instead of storing it locally?

Because Elasticsearch indexes and stores data efficiently, making it easy to search, analyze, and visualize large volumes of performance data as shown in execution_table step 3 and 4.

How does Kibana help in application performance monitoring?

Kibana queries the indexed data in Elasticsearch and creates visual dashboards, helping to spot trends and issues quickly, as seen in execution_table step 5.

What triggers alerts in this monitoring flow?

Alerts are triggered when performance metrics exceed predefined thresholds, indicated in execution_table step 6, helping teams react fast to problems.

Visual Quiz - 3 Questions

Look at the execution table, what response does Elasticsearch give after sending transaction data?

A 404 Not Found

B 500 Internal Server Error

C 201 Created

D 200 OK

💡 Hint

Check the 'Elasticsearch Response' column at Step 3 in the execution_table

At which step does Kibana update the dashboard with the latest data?

A Step 5

B Step 3

C Step 6

D Step 2

💡 Hint

Look at the 'Action' column in execution_table for where visualization happens

If the application stops collecting metrics, which variable in variable_tracker would remain 'None'?

A application_state

B transaction_data

C Elasticsearch Response

D Alerts

💡 Hint

Refer to variable_tracker row for 'transaction_data' and its values after Step 2

Concept Snapshot

Application Performance Monitoring (APM) with Elasticsearch:
- Collects app metrics and traces
- Sends data to Elasticsearch APM index
- Stores and indexes for fast search
- Visualizes data in Kibana dashboards
- Detects issues and triggers alerts
- Enables continuous app optimization

Full Transcript

Application Performance Monitoring (APM) with Elasticsearch involves collecting performance data like transaction durations from your running application. This data is sent to Elasticsearch where it is stored and indexed for fast searching. Kibana then queries this data to create visual dashboards that help you understand how your app performs. Alerts can be set up to notify you when performance degrades. This cycle repeats continuously to help developers optimize the application based on real-time insights.

Practice

1. What is the main purpose of Application Performance Monitoring (APM) in Elasticsearch?

easy

A. To track application speed and detect errors

B. To store user login credentials securely

C. To manage Elasticsearch cluster nodes

D. To backup Elasticsearch indexes automatically

Solution

1. Step 1: Understand APM's role

   APM is designed to monitor how fast an application runs and to find any errors it produces.

2. Step 2: Match purpose with options

   Only To track application speed and detect errors describes tracking speed and errors, which fits APM's main goal.

3. Final Answer:

   To track application speed and detect errors -> Option A

4. Quick Check:

   APM purpose = Track speed and errors [OK]

Hint: APM = watch app speed and errors [OK]

Common Mistakes:

• Confusing APM with security or backup tools
• Thinking APM manages cluster nodes
• Assuming APM stores user credentials

2. Which Elasticsearch query syntax correctly calculates the average response time from APM data?

easy

A. POST /apm-*/_update_by_query {"script": {"source": "ctx._source.duration = 0"}}

B. GET /apm-*/_search {"query": {"match_all": {}}, "size":10}

C. GET /apm-*/_search {"size":0, "aggs": {"avg_response_time": {"avg": {"field": "transaction.duration.us"}}}}

D. GET /apm-*/_search {"aggs": {"max_response_time": {"max": {"field": "transaction.duration.us"}}}}

Solution

1. Step 1: Identify aggregation for average

   The query uses "avg" aggregation on the field "transaction.duration.us" which stores response times in microseconds.

2. Step 2: Confirm query structure

   Size is 0 to avoid returning documents, focusing only on aggregation results, which is correct for average calculation.

3. Final Answer:

   GET /apm-*/_search {"size":0, "aggs": {"avg_response_time": {"avg": {"field": "transaction.duration.us"}}}} -> Option C

4. Quick Check:

   Average aggregation query = GET /apm-*/_search {"size":0, "aggs": {"avg_response_time": {"avg": {"field": "transaction.duration.us"}}}} [OK]

Hint: Average uses "avg" aggregation with size 0 [OK]

Common Mistakes:

• Using match_all without aggregation
• Using update_by_query instead of search
• Using max aggregation instead of avg

3. Given this Elasticsearch aggregation query on APM data, what is the expected output?

GET /apm-*/_search
{
  "size": 0,
  "aggs": {
    "avg_response_time": {
      "avg": { "field": "transaction.duration.us" }
    }
  }
}

medium

A. {"aggregations":{"max_response_time":{"value":500000}}}

B. {"hits":{"total":100}}

C. {"error":"Field not found"}

D. {"aggregations":{"avg_response_time":{"value":250000}}}

Solution

1. Step 1: Understand aggregation type

   The query requests the average of the field "transaction.duration.us" which holds response times in microseconds.

2. Step 2: Match output to aggregation

   The output shows an aggregation named "avg_response_time" with a numeric value representing the average, matching {"aggregations":{"avg_response_time":{"value":250000}}}.

3. Final Answer:

   {"aggregations":{"avg_response_time":{"value":250000}}} -> Option D

4. Quick Check:

   Average aggregation output = {"aggregations":{"avg_response_time":{"value":250000}}} [OK]

Hint: Aggregation output shows "aggregations" with average value [OK]

Common Mistakes:

• Confusing hits total with aggregation result
• Expecting max instead of avg
• Assuming error without checking field existence

4. You run this Elasticsearch query to get average response time but get an error: Fielddata is disabled on text fields by default. What is the likely cause?

medium

A. Trying to aggregate on a text field instead of a numeric field

B. Using the wrong index pattern in the query

C. Missing authentication credentials

D. Query syntax error in aggregation block

Solution

1. Step 1: Analyze error message

   The error says fielddata is disabled on text fields, which means aggregation was attempted on a text field.

2. Step 2: Understand aggregation requirements

   Aggregations like average require numeric fields, so using a text field causes this error.

3. Final Answer:

   Trying to aggregate on a text field instead of a numeric field -> Option A

4. Quick Check:

   Fielddata error = Aggregation on text field [OK]

Hint: Average needs numeric field, not text [OK]

Common Mistakes:

• Blaming index pattern or auth for this error
• Assuming syntax error without checking field type
• Ignoring field data type requirements

5. You want to monitor the average response time for your app but only for transactions with errors. Which Elasticsearch query snippet correctly filters and calculates this?

hard

A. { "size": 0, "query": { "term": { "error.id": "" } }, "aggs": { "avg_response_time": { "avg": { "field": "transaction.duration.us" } } } }

B. { "size": 0, "query": { "exists": { "field": "error.id" } }, "aggs": { "avg_response_time": { "avg": { "field": "transaction.duration.us" } } } }

C. { "size": 0, "query": { "match_all": {} }, "aggs": { "avg_response_time": { "avg": { "field": "transaction.duration.us" } } } }

D. { "size": 0, "query": { "term": { "transaction.status": "success" } }, "aggs": { "avg_response_time": { "avg": { "field": "transaction.duration.us" } } } }

Solution

1. Step 1: Identify filter for transactions with errors

   Transactions with errors have a non-empty "error.id" field, so we use "exists" query on "error.id".

2. Step 2: Confirm aggregation on filtered data

   The aggregation calculates average response time only on filtered documents, which is correct.

3. Final Answer:

   { "size": 0, "query": { "exists": { "field": "error.id" } }, "aggs": { "avg_response_time": { "avg": { "field": "transaction.duration.us" } } } } -> Option B

4. Quick Check:

   Filter errors with exists + avg aggregation = { "size": 0, "query": { "exists": { "field": "error.id" } }, "aggs": { "avg_response_time": { "avg": { "field": "transaction.duration.us" } } } } [OK]

Hint: Use exists query to filter error transactions [OK]

Common Mistakes:

• Using empty term query instead of exists
• Calculating average without filtering errors
• Filtering for success instead of errors