Stats and extended stats in Elasticsearch - Step-by-Step Execution

Concept Flow - Stats and extended stats

Start Query

↓

Select Field

↓

Apply Stats Aggregation

↓

Calculate Basic Stats

↓

Calculate Extended Stats

↓

Return Results

The query starts by selecting a field, then applies stats aggregation to calculate basic and extended statistics, and finally returns the results.

Execution Sample

Elasticsearch

{
  "aggs": {
    "stats_example": {
      "extended_stats": { "field": "price" }
    }
  }
}

This Elasticsearch query calculates extended statistics on the 'price' field.

Execution Table

Step	Action	Field	Calculation	Result
1	Start aggregation	price	N/A	N/A
2	Calculate count	price	Count all documents with price	5
3	Calculate min	price	Find minimum price	10
4	Calculate max	price	Find maximum price	50
5	Calculate avg	price	Calculate average price	30
6	Calculate sum	price	Sum all price values	150
7	Calculate sum_of_squares	price	Sum of each price squared	5500
8	Calculate variance	price	Variance of price values	250
9	Calculate std_deviation	price	Standard deviation of price	15.81
10	Return all stats	price	All calculated stats	{"count":5,"min":10,"max":50,"avg":30,"sum":150,"sum_of_squares":5500,"variance":250,"std_deviation":15.81}

💡 All extended stats calculated and returned for the 'price' field.

Variable Tracker

Variable	Start	After Step 2	After Step 3	After Step 4	After Step 5	After Step 6	After Step 7	After Step 8	After Step 9	Final
count	0	5	5	5	5	5	5	5	5	5
min	∞	∞	10	10	10	10	10	10	10	10
max	-∞	-∞	-∞	50	50	50	50	50	50	50
avg	N/A	N/A	N/A	N/A	30	30	30	30	30	30
sum	0	0	0	0	0	150	150	150	150	150
sum_of_squares	0	0	0	0	0	0	5500	5500	5500	5500
variance	N/A	N/A	N/A	N/A	N/A	N/A	N/A	250	250	250
std_deviation	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	15.81	15.81

Key Moments - 3 Insights

Why do we calculate sum_of_squares before variance?

Sum_of_squares is needed to compute variance; as shown in steps 7 and 8 of the execution_table, variance calculation depends on sum_of_squares.

What does 'count' represent in the stats aggregation?

Count is the number of documents with the field present, as shown in step 2 of the execution_table where count becomes 5.

Why is standard deviation calculated after variance?

Standard deviation is the square root of variance, so variance must be calculated first (step 8), then std_deviation is computed (step 9).

Visual Quiz - 3 Questions

Look at the execution_table, what is the value of 'max' after step 4?

A 50

B 10

C 30

D 5

💡 Hint

Check the 'max' variable value in the execution_table row for step 4.

At which step is the average ('avg') calculated?

A Step 3

B Step 5

C Step 7

D Step 9

💡 Hint

Look at the 'avg' calculation in the execution_table rows.

If the count was 10 instead of 5, which variable would change in the variable_tracker?

A min

B std_deviation

C count

D max

💡 Hint

Refer to the 'count' variable changes in the variable_tracker.

Concept Snapshot

Stats and extended stats aggregation in Elasticsearch:
- Use 'stats' or 'extended_stats' aggregation on a numeric field.
- Calculates count, min, max, avg, sum (basic stats).
- Extended stats add sum_of_squares, variance, std_deviation.
- Useful for quick numeric summaries in search results.
- Syntax: {"aggs": {"name": {"extended_stats": {"field": "fieldname"}}}}

Full Transcript

This visual execution trace shows how Elasticsearch calculates stats and extended stats on a numeric field. The process starts by selecting the field, then counts documents with that field, finds minimum and maximum values, calculates average and sum, then computes sum_of_squares, variance, and standard deviation. Each step updates variables tracked in the variable_tracker. Key moments clarify why sum_of_squares is needed before variance and why standard deviation follows variance. The visual quiz tests understanding of variable values at specific steps and how changes affect results. The concept snapshot summarizes how to use stats and extended_stats aggregations in Elasticsearch queries.

Practice

1. What does the stats aggregation in Elasticsearch provide by default?

easy

A. Count, min, max, avg, and sum

B. Variance and standard deviation

C. Sum of squares only

D. Median and mode

Solution

1. Step 1: Understand the purpose of stats aggregation

   The stats aggregation provides basic statistical data about numeric fields.

2. Step 2: Identify the default metrics included

   By default, it returns count, min, max, avg, and sum values.

3. Final Answer:

   Count, min, max, avg, and sum -> Option A

4. Quick Check:

   Stats aggregation = basic stats [OK]

Hint: Stats aggregation gives basic stats like count and sum [OK]

Common Mistakes:

• Confusing stats with extended_stats
• Thinking stats includes variance
• Assuming median is included

2. Which of the following is the correct syntax to add an extended stats aggregation named my_stats on the field price?

easy

A. { "aggs": { "my_stats": { "extended_stats": { "field": "price" } } } }

B. { "aggs": { "my_stats": { "stats": { "field": "price" } } } }

C. { "aggs": { "my_stats": { "extended_stats": { "field_name": "price" } } } }

D. { "aggregations": { "my_stats": { "stats": { "field": "price" } } } }

Solution

1. Step 1: Check the aggregation type and field key

   The aggregation type must be extended_stats and the field key must be field.

2. Step 2: Verify the JSON structure

   The correct syntax uses aggs as the root key for aggregations, not aggregations.

3. Final Answer:

   { "aggs": { "my_stats": { "extended_stats": { "field": "price" } } } } -> Option A

4. Quick Check:

   Use extended_stats with field inside aggs [OK]

Hint: Use "aggs" and "extended_stats" with "field" key [OK]

Common Mistakes:

• Using "field_name" instead of "field"
• Using "aggregations" instead of "aggs"
• Confusing stats with extended_stats

3. Given this Elasticsearch aggregation query:

{
  "aggs": {
    "price_stats": {
      "extended_stats": {
        "field": "price"
      }
    }
  }
}

Which of the following metrics will NOT be included in the response?

medium

A. Sum of squares

B. Median

C. Standard deviation

D. Variance

Solution

1. Step 1: Identify metrics returned by extended_stats

   The extended_stats aggregation returns count, min, max, avg, sum, sum_of_squares, variance, and std_deviation.

2. Step 2: Check which metric is missing

   Median is not included in either stats or extended_stats aggregations.

3. Final Answer:

   Median -> Option B

4. Quick Check:

   Median not in extended_stats [OK]

Hint: Extended stats exclude median, include variance and std deviation [OK]

Common Mistakes:

• Assuming median is part of extended_stats
• Confusing sum_of_squares with sum
• Thinking variance is not included

4. You wrote this aggregation:

{
  "aggs": {
    "my_stats": {
      "extended_stats": {
        "field_name": "age"
      }
    }
  }
}

Why does this query fail?

medium

A. The aggregation name cannot be "my_stats"

B. The root key should be "aggregations" instead of "aggs"

C. extended_stats does not support the field "age"

D. The field key should be "field", not "field_name"

Solution

1. Step 1: Check the field key in extended_stats

   The correct key to specify the field is field, not field_name.

2. Step 2: Verify other parts of the query

   The aggregation name "my_stats" is valid, and "aggs" is the correct root key. The field "age" can be used if it exists.

3. Final Answer:

   The field key should be "field", not "field_name" -> Option D

4. Quick Check:

   Use "field" key for field name [OK]

Hint: Use "field" key, not "field_name" in aggregations [OK]

Common Mistakes:

• Using "field_name" instead of "field"
• Changing aggregation root key incorrectly
• Assuming field name is invalid without checking

5. You want to find the average price and also understand the price spread using variance and standard deviation in one query. Which aggregation setup is best?

hard

A. Use stats aggregation on price

B. Use percentiles aggregation on price

C. Use extended_stats aggregation on price

D. Use terms aggregation on price

Solution

1. Step 1: Identify metrics needed

   You want average, variance, and standard deviation, which describe spread and central tendency.

2. Step 2: Choose aggregation that provides all metrics

   extended_stats provides avg, variance, std_deviation, sum_of_squares, and more in one aggregation.

3. Step 3: Compare other options

   stats lacks variance and std deviation; terms groups data; percentiles gives percentile info, not variance.

4. Final Answer:

   Use extended_stats aggregation on price -> Option C

5. Quick Check:

   Extended stats = avg + variance + std deviation [OK]

Hint: Extended stats give avg and spread metrics together [OK]

Common Mistakes:

• Using stats and missing variance
• Confusing terms with stats
• Choosing percentiles for variance