Why documents are the unit of data in Elasticsearch - Why It Works This Way

Overview - Why documents are the unit of data

What is it?

In Elasticsearch, data is stored and managed as documents. A document is a basic unit of information that contains structured data in a format like JSON. Each document represents a single entity or record, such as a user profile or a product listing. This approach makes it easy to store, search, and retrieve data efficiently.

Why it matters

Using documents as the unit of data allows Elasticsearch to handle complex and varied information flexibly. Without documents, data would be stored in rigid tables or rows, making it harder to index and search quickly. Documents enable fast, full-text search and easy scaling, which is crucial for applications like search engines and analytics platforms.

Where it fits

Before learning why documents are the unit of data, you should understand basic data storage concepts and JSON format. After this, you can explore how Elasticsearch indexes documents and performs searches, and then learn about mapping and querying documents in detail.

Mental Model

Core Idea

A document is a self-contained package of data that Elasticsearch stores, indexes, and searches as a single unit.

Think of it like...

Think of a document like a single page in a filing cabinet, where each page holds all the information about one person or item. Instead of searching through the whole cabinet, you quickly find the right page with the details you need.

┌───────────────┐
│   Document    │
│ ┌───────────┐ │
│ │ Field 1   │ │
│ │ Field 2   │ │
│ │ Field 3   │ │
│ └───────────┘ │
└───────────────┘
Each document contains multiple fields with data.

Build-Up - 6 Steps

1

Foundation

Understanding the Document Concept

Concept: Introduce what a document is in Elasticsearch and why it is the basic data unit.

A document in Elasticsearch is a JSON object that holds data about one entity. For example, a document could represent a book with fields like title, author, and year. Documents are stored in indexes and can be searched individually.

Result

You understand that documents are the smallest pieces of data Elasticsearch works with.

Knowing that documents are self-contained helps you see why Elasticsearch can quickly find and retrieve data without scanning unrelated information.

2

Foundation

JSON Format as Document Structure

Concept: Explain why JSON is used to structure documents.

Documents use JSON because it is easy to read and write, supports nested data, and works well with web technologies. JSON fields can be simple values or complex objects, allowing flexible data representation.

Result

You can recognize how data is organized inside a document and why JSON fits this role.

Understanding JSON's flexibility clarifies how documents can represent diverse data types and structures.

3

Intermediate

Documents vs. Traditional Rows

🤔Before reading on: do you think documents are just like rows in a table or something different? Commit to your answer.

Concept: Compare documents with traditional database rows to highlight differences.

Unlike rows in a table, documents can have different fields and nested data. This means each document can be unique, and Elasticsearch does not require a fixed schema. This flexibility supports varied and evolving data.

Result

You see that documents allow more adaptable data storage than rigid tables.

Knowing this difference helps you appreciate why Elasticsearch is suited for dynamic and complex data.

4

Intermediate

Indexing Documents for Fast Search

🤔Before reading on: do you think Elasticsearch searches documents by scanning all data or using an index? Commit to your answer.

Concept: Introduce how documents are indexed to enable quick searching.

When a document is added, Elasticsearch creates an index of its fields and values. This index is like a map that points to where data lives, so searches can jump directly to matching documents without scanning everything.

Result

You understand that indexing makes document search fast and efficient.

Understanding indexing reveals why documents are practical units for search engines.

5

Advanced

Document Immutability and Updates

🤔Before reading on: do you think Elasticsearch changes documents in place or replaces them? Commit to your answer.

Concept: Explain how documents are updated and why they are treated as immutable.

Elasticsearch treats documents as immutable, meaning it does not change them directly. Instead, it creates a new version and marks the old one for deletion. This approach simplifies concurrency and indexing but means updates are actually replacements.

Result

You learn how document immutability affects data updates and performance.

Knowing this prevents confusion about how Elasticsearch handles changes and why some operations may be slower.

6

Expert

Document Storage and Sharding Internals

🤔Before reading on: do you think documents are stored all together or split across shards? Commit to your answer.

Concept: Dive into how documents are physically stored and distributed in Elasticsearch clusters.

Documents are stored in shards, which are parts of an index spread across nodes. Each shard holds a subset of documents. This distribution allows Elasticsearch to scale horizontally and handle large data volumes efficiently.

Result

You understand the internal storage and distribution of documents in Elasticsearch.

Understanding sharding clarifies how Elasticsearch balances load and maintains performance at scale.

Under the Hood

Elasticsearch stores each document as a JSON object indexed by an inverted index structure. When a document is added, its fields are tokenized and mapped to terms in the index, pointing to the document's location. Documents are immutable; updates create new versions. The index is split into shards, each managed by a node, enabling distributed storage and search.

Why designed this way?

Documents as units allow flexible, schema-less data storage suited for varied real-world data. Immutability simplifies concurrency and indexing. Sharding supports horizontal scaling and fault tolerance. Alternatives like fixed tables or mutable records would limit flexibility and scalability.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│   Document 1  │──────▶│   Shard 1     │──────▶│   Node A      │
│   Document 2  │──────▶│   Shard 2     │──────▶│   Node B      │
│   Document 3  │──────▶│   Shard 3     │──────▶│   Node C      │
└───────────────┘       └───────────────┘       └───────────────┘
Each document is indexed and stored in a shard distributed across nodes.

Myth Busters - 4 Common Misconceptions

Quick: Do you think all documents in an index must have the same fields? Commit to yes or no.

Common Belief: All documents in an Elasticsearch index must have the same fields and structure.

Reality: Documents in the same index can have different fields and structures because Elasticsearch is schema-flexible.

Why it matters: Assuming uniform fields limits how you design your data and can cause confusion when indexing diverse data.

Quick: Do you think updating a document changes it instantly in place? Commit to yes or no.

Common Belief: When you update a document, Elasticsearch modifies it directly in storage.

Reality: Elasticsearch treats documents as immutable and replaces the old document with a new version during updates.

Why it matters: Misunderstanding this can lead to wrong expectations about update speed and data consistency.

Quick: Do you think Elasticsearch searches all documents linearly? Commit to yes or no.

Common Belief: Elasticsearch searches by scanning every document one by one.

Reality: Elasticsearch uses inverted indexes to jump directly to matching documents without scanning all data.

Why it matters: Believing in linear search underestimates Elasticsearch's speed and can misguide optimization efforts.

Quick: Do you think documents are stored on a single server only? Commit to yes or no.

Common Belief: All documents in an index are stored on one server or node.

Reality: Documents are split across multiple shards and nodes to distribute load and increase reliability.

Why it matters: Ignoring this can cause misunderstandings about scaling and fault tolerance.

Expert Zone

1

Documents can have nested objects and arrays, but querying nested fields requires special handling to avoid incorrect matches.

2

Elasticsearch merges segments of indexed documents in the background to optimize search speed and storage, which affects how quickly deleted documents free space.

3

Mapping conflicts can occur if documents with the same field name have different data types, requiring careful index design.

When NOT to use

Using documents as the unit is less suitable when strict relational integrity or complex multi-table joins are needed; traditional relational databases with normalized tables are better in those cases.

Production Patterns

In production, documents are designed to be denormalized, containing all needed data to avoid joins. Index templates and mappings enforce field types. Shard count and replication are tuned for performance and reliability. Bulk APIs are used for efficient document ingestion.

Connections

JSON Data Format

Documents are structured as JSON objects, building directly on JSON syntax and semantics.

Understanding JSON helps grasp how documents store complex, nested data flexibly.

Inverted Index

Documents are indexed using inverted indexes to enable fast full-text search.

Knowing inverted indexes explains why documents can be searched quickly despite large data volumes.

Library Cataloging Systems

Like documents in Elasticsearch, library cards represent individual books with metadata for quick lookup.

Seeing documents as catalog cards helps understand how indexing and retrieval work in search systems.

Common Pitfalls

#1 Trying to update a document by changing fields directly without reindexing.

Wrong approach:

POST /index/_update/1 { "doc": { "title": "New Title" } } // expecting in-place change

Correct approach:

POST /index/_doc/1 { "title": "New Title", "other_fields": "..." } // reindex entire document

Root cause:

Misunderstanding that Elasticsearch treats documents as immutable and requires full document replacement on update.

#2 Assuming all documents must have identical fields and failing to index documents with new fields.

Wrong approach:

Indexing documents with different fields without updating mappings, causing errors or ignored fields.

Correct approach:

Define dynamic mappings or update mappings to accommodate new fields before indexing varied documents.

Root cause:

Confusing Elasticsearch's flexible schema with fixed relational schemas.

#3 Searching without using the inverted index, expecting linear scan.

Wrong approach:

Using scripts or filters that force scanning all documents for simple keyword search.

Correct approach:

Use full-text queries that leverage inverted indexes for efficient search.

Root cause:

Not leveraging Elasticsearch's indexing capabilities properly.

Key Takeaways

Documents are the basic units of data in Elasticsearch, storing all information about one entity in JSON format.

Using documents allows flexible, schema-less data storage that can handle varied and nested data easily.

Documents are indexed for fast search using inverted indexes, enabling quick retrieval without scanning all data.

Elasticsearch treats documents as immutable, replacing them on updates to simplify concurrency and indexing.

Documents are distributed across shards and nodes, allowing Elasticsearch to scale and remain reliable.

Practice

1. Why are documents considered the unit of data in Elasticsearch?

easy

A. Because documents cannot be searched individually

B. Because documents are only used for storing metadata

C. Because documents are the same as database tables

D. Because each document holds all related information together in one place

Solution

1. Step 1: Understand the role of documents in Elasticsearch

   Documents store all related data fields together, making them self-contained units.

2. Step 2: Compare documents to other data units

   Unlike tables or metadata, documents hold actual data and can be searched independently.

3. Final Answer:

   Because each document holds all related information together in one place -> Option D

4. Quick Check:

   Documents = self-contained data units [OK]

Hint: Remember: documents bundle related data together [OK]

Common Mistakes:

• Thinking documents are just metadata
• Confusing documents with tables
• Assuming documents can't be searched alone

2. Which of the following is the correct way to identify a document in Elasticsearch?

easy

A. Using the document's file path on disk

B. Using the document's creation timestamp only

C. Using a unique document ID within an index

D. Using the document's size in bytes

Solution

1. Step 1: Identify how documents are stored

   Documents are stored inside indexes and each has a unique ID for identification.

2. Step 2: Eliminate incorrect options

   File paths, timestamps, or size are not used as unique identifiers in Elasticsearch.

3. Final Answer:

   Using a unique document ID within an index -> Option C

4. Quick Check:

   Unique ID = document identifier [OK]

Hint: Unique ID inside index identifies documents [OK]

Common Mistakes:

• Confusing file path with document ID
• Using timestamps as unique keys
• Assuming size identifies documents

3. Given an Elasticsearch index with documents representing books, which statement best explains why documents are efficient for searching?

medium

A. Because documents require manual indexing before searching

B. Because each document contains all book details, searches can target whole books quickly

C. Because documents only store keywords, not full data

D. Because documents are stored as separate files on disk, making file search faster

Solution

1. Step 1: Understand document structure for books

   Each document holds all details about a book, like title, author, and summary.

2. Step 2: Analyze search efficiency

   Having all data in one document allows Elasticsearch to quickly find relevant books without joining data.

3. Final Answer:

   Because each document contains all book details, searches can target whole books quickly -> Option B

4. Quick Check:

   Complete data per document = fast search [OK]

Hint: Whole data in one document speeds search [OK]

Common Mistakes:

• Thinking documents are separate files
• Believing documents store only keywords
• Assuming manual indexing is needed every time

4. You have an Elasticsearch document missing its unique ID. What problem will this cause?

medium

A. The document cannot be uniquely identified or updated

B. The document will be stored twice automatically

C. The document will be ignored during searches

D. The document will be converted into multiple smaller documents

Solution

1. Step 1: Understand the role of unique IDs

   Unique IDs allow Elasticsearch to find, update, or delete specific documents.

2. Step 2: Identify the impact of missing ID

   Without an ID, Elasticsearch cannot uniquely identify the document, causing update or delete failures.

3. Final Answer:

   The document cannot be uniquely identified or updated -> Option A

4. Quick Check:

   Missing ID = no unique identification [OK]

Hint: Missing ID means no unique document tracking [OK]

Common Mistakes:

• Assuming document duplicates automatically
• Thinking document is ignored in search
• Believing document splits into parts

5. You want to design an Elasticsearch index for storing user profiles. Why should each profile be a separate document rather than splitting data across multiple documents?

hard

A. Because storing all profile data in one document makes retrieval and updates simpler and faster

B. Because multiple documents per profile improve search speed by splitting data

C. Because Elasticsearch cannot store more than one document per user

D. Because documents must always be very small in size

Solution

1. Step 1: Consider data grouping in Elasticsearch

   Each document should hold all related data for easy access and management.

2. Step 2: Evaluate impact on retrieval and updates

   One document per profile means fewer queries and simpler updates, improving performance.

3. Step 3: Reject incorrect assumptions

   Splitting data into multiple documents complicates queries and slows down updates.

4. Final Answer:

   Because storing all profile data in one document makes retrieval and updates simpler and faster -> Option A

5. Quick Check:

   One document per profile = simpler management [OK]

Hint: Keep related data in one document for speed [OK]

Common Mistakes:

• Thinking multiple documents per profile speed search
• Believing Elasticsearch limits one document per user
• Assuming documents must be very small always