Source filtering in Elasticsearch - Mini Project: Build & Apply

Source filtering in Elasticsearch queries

📖 Scenario: You work with a collection of books stored in Elasticsearch. Each book has details like title, author, year, and summary. Sometimes you want to get only specific fields from the search results to save bandwidth and focus on what matters.

🎯 Goal: Build an Elasticsearch query that searches for books by author and uses source filtering to return only the title and year fields in the results.

📋 What You'll Learn

Create a query that searches for books where the author is exactly 'Jane Austen'.

Add source filtering to include only the title and year fields in the search results.

Use the Elasticsearch JSON query format.

💡 Why This Matters

🌍 Real World

Source filtering helps reduce the amount of data sent over the network by returning only the fields you need from Elasticsearch search results.

💼 Career

Many jobs working with Elasticsearch require efficient queries that return only relevant data to improve performance and reduce costs.

1

Create the basic search query

Create a JSON object called query with a match clause that searches for author equal to "Jane Austen".

Elasticsearch

{
  "query": {
    "match": {
      "author": "Jane Austen"
    }
  }
}
# Your code here

Hint

Use the match keyword inside query to find documents where the author is 'Jane Austen'.

2

Add source filtering to include specific fields

Add a _source field to the JSON object that includes only the fields "title" and "year".

Elasticsearch

{
  "query": {
    "match": {
      "author": "Jane Austen"
    }
  },
  "_source": ["title", "year"]
}
# Your code here

Hint

The _source field controls which fields are returned. Use a list with the field names.

3

Add a filter to exclude the summary field instead

Modify the _source field to exclude the "summary" field instead of including fields. Use the exclude syntax inside _source.

Elasticsearch

{
  "query": {
    "match": {
      "author": "Jane Austen"
    }
  },
  "_source": {
    "exclude": ["summary"]
  }
}
# Your code here

Hint

Use _source as an object with an exclude list to hide fields.

4

Print the final query JSON

Print the complete JSON query object exactly as it is, including the query and _source with exclude.

Elasticsearch

# Print the JSON query
print(query)
# Your code here

Hint

Use print(query) to show the final JSON object.

Practice

1. What is the main purpose of _source filtering in Elasticsearch?

easy

A. To control which parts of the document are returned in search results

B. To filter documents before indexing

C. To change the document content during indexing

D. To sort search results by field values

Solution

1. Step 1: Understand _source filtering role

   _source filtering controls which fields of a document are included in the search response.

2. Step 2: Differentiate from other features

   It does not affect indexing or sorting, only what data is returned after a search.

3. Final Answer:

   To control which parts of the document are returned in search results -> Option A

4. Quick Check:

   Source filtering = control returned fields [OK]

Hint: Remember: source filtering limits returned fields only [OK]

Common Mistakes:

• Confusing source filtering with indexing filters
• Thinking it changes document content
• Assuming it sorts results

2. Which of the following is the correct syntax to exclude the field password from the search results using source filtering?

easy

A. "_source": { "filter": ["password"] }

B. "_source": { "include": ["password"] }

C. "_source": { "excludes": ["password"] }

D. "_source": { "remove": ["password"] }

Solution

1. Step 1: Recall source filtering syntax

   To exclude fields, use "excludes" inside _source with a list of field names.

2. Step 2: Match correct syntax

   "_source": { "excludes": ["password"] } uses "excludes": ["password"], which is the correct way to exclude the password field.

3. Final Answer:

   "_source": { "excludes": ["password"] } -> Option C

4. Quick Check:

   Exclude fields = use "excludes" key [OK]

Hint: Use "excludes" key to remove fields from results [OK]

Common Mistakes:

• Using "include" to exclude fields
• Using non-existent keys like "filter" or "remove"
• Wrong JSON structure

3. Given this Elasticsearch query snippet, what fields will be returned in the search results?

{
  "_source": { "includes": ["title", "author"] },
  "query": { "match_all": {} }
}

medium

A. All fields will be returned

B. All fields except title and author will be returned

C. No fields will be returned

D. Only the fields title and author will be returned

Solution

1. Step 1: Understand includes in _source

   The includes key specifies which fields to return from the document.

2. Step 2: Apply to given query

   Since only title and author are included, only these fields will appear in the results.

3. Final Answer:

   Only the fields title and author will be returned -> Option D

4. Quick Check:

   includes fields = return only those fields [OK]

Hint: includes means return only listed fields [OK]

Common Mistakes:

• Thinking includes excludes fields
• Assuming all fields return by default
• Confusing includes with excludes

4. Identify the error in this source filtering snippet:

{
  "_source": {
    "includes": "title,author"
  },
  "query": { "match_all": {} }
}

medium

A. The _source key is misspelled

B. The includes field should be a list, not a comma-separated string

C. The query is missing a filter clause

D. The match_all query is invalid

Solution

1. Step 1: Check includes value type

   The includes field expects an array of strings, not a single comma-separated string.

2. Step 2: Identify correct format

   It should be ["title", "author"] instead of "title,author".

3. Final Answer:

   The includes field should be a list, not a comma-separated string -> Option B

4. Quick Check:

   includes must be an array, not a string [OK]

Hint: Use arrays for includes/excludes fields, not strings [OK]

Common Mistakes:

• Using comma-separated strings instead of arrays
• Misspelling _source key
• Confusing query errors with source filtering

5. You want to return all fields except password and ssn from documents, but also include only the name and email fields explicitly. Which source filtering configuration achieves this correctly?

hard

A. "_source": { "includes": ["name", "email"], "excludes": ["password", "ssn"] }

B. "_source": { "include": ["name", "email", "password", "ssn"] }

C. "_source": { "exclude": ["password", "ssn"] }

D. "_source": { "include": ["name", "email"] }

Solution

1. Step 1: Understand combined includes and excludes

   Using both includes and excludes lets you specify fields to return and fields to omit.

2. Step 2: Analyze options

   "_source": { "includes": ["name", "email"], "excludes": ["password", "ssn"] } includes name and email but excludes password and ssn, matching the requirement.

3. Final Answer:

   "_source": { "includes": ["name", "email"], "excludes": ["password", "ssn"] } -> Option A

4. Quick Check:

   includes + excludes together filters fields precisely [OK]

Hint: Combine includes and excludes to fine-tune returned fields [OK]

Common Mistakes:

• Using only includes or excludes, missing fields
• Including fields meant to be excluded
• Misunderstanding combined filtering behavior