Field and document level security in Elasticsearch - Deep Dive

Overview - Field and document level security

What is it?

Field and document level security in Elasticsearch controls what parts of data users can see or access. Document level security limits access to whole documents based on rules. Field level security restricts access to specific fields inside documents. Together, they help protect sensitive information while allowing flexible data sharing.

Why it matters

Without these controls, users might see data they shouldn't, risking privacy and security breaches. For example, a user might see confidential customer details or internal notes they are not authorized to view. Field and document level security ensure that users only see data relevant to their role, protecting sensitive information and complying with regulations.

Where it fits

Before learning this, you should understand basic Elasticsearch concepts like indices, documents, and roles. After mastering this, you can explore advanced security features like audit logging, encryption, and fine-grained access control in Elasticsearch.

Mental Model

Core Idea

Field and document level security filter data so users only see the exact parts they are allowed to access.

Think of it like...

It's like a library where some books are locked in cabinets (document level security), and inside open books, some pages are hidden or blurred out (field level security) depending on who is reading.

┌───────────────────────────────┐
│          Elasticsearch         │
│  ┌───────────────┐            │
│  │ Documents     │            │
│  │ ┌───────────┐ │            │
│  │ │ Fields    │ │            │
│  │ │ ┌───────┐ │ │            │
│  │ │ │ Data  │ │ │            │
│  │ │ └───────┘ │ │            │
│  │ └───────────┘ │            │
│  └───────────────┘            │
│                               │
│  Document Level Security:     │
│  Filters whole documents      │
│                               │
│  Field Level Security:        │
│  Filters specific fields      │
└───────────────────────────────┘

Build-Up - 7 Steps

1

Foundation

Understanding Elasticsearch Data Structure

Concept: Learn what documents and fields are in Elasticsearch.

Elasticsearch stores data as documents inside indices. Each document is like a record or a row in a table. Documents have fields, which are like columns or attributes. For example, a document about a person might have fields like name, age, and email.

Result

You can identify documents and fields as the basic units of data in Elasticsearch.

Knowing the data structure is essential because security controls work by filtering documents or fields.

2

Foundation

Basics of Elasticsearch Security Roles

Concept: Roles define what users can do and see in Elasticsearch.

Elasticsearch uses roles to control access. A role can allow or deny actions like reading or writing data. Roles can also limit access to certain indices. This is the first layer of security before field or document level controls.

Result

You understand that roles are the starting point for controlling user access.

Roles set the stage for more detailed controls like field and document level security.

3

Intermediate

Document Level Security Explained

🤔Before reading on: do you think document level security hides parts of a document or whole documents? Commit to your answer.

Concept: Document level security filters entire documents based on rules.

Document level security uses queries to decide which documents a user can see. For example, a sales user might only see documents where the region field matches their area. Documents that don't match are completely hidden from that user.

Result

Users only see documents allowed by the security rules, hiding all others.

Understanding that document level security works by filtering whole documents helps you design precise access controls.

4

Intermediate

Field Level Security Explained

🤔Before reading on: do you think field level security hides whole documents or parts inside documents? Commit to your answer.

Concept: Field level security hides or shows specific fields inside documents.

Field level security lets you specify which fields a user can see or cannot see. For example, you might hide the 'salary' field from regular employees but show it to HR. The document is visible, but some fields are masked or removed.

Result

Users see documents but only with allowed fields visible.

Knowing field level security works inside documents allows fine-grained control over sensitive data.

5

Intermediate

Combining Field and Document Level Security

Concept: You can use both security levels together for precise access control.

Elasticsearch allows combining document and field level security in roles. For example, a user might only see documents from their region (document level) and within those documents, only see non-sensitive fields (field level). This layered approach protects data effectively.

Result

Access is filtered both by which documents and which fields are visible.

Combining both levels creates powerful, flexible security tailored to user needs.

6

Advanced

Implementing Security with Role Mappings

🤔Before reading on: do you think role mappings assign roles to users directly or through groups? Commit to your answer.

Concept: Role mappings connect users or groups to roles with security rules.

In Elasticsearch, you assign roles to users or groups via role mappings. These mappings determine which security rules apply. For example, you can map a role with document and field level security to a user group like 'sales'. This makes managing access easier at scale.

Result

Users get the correct access based on their assigned roles and mappings.

Understanding role mappings helps manage security efficiently in real environments.

7

Expert

Performance and Limitations of Field and Document Security

🤔Before reading on: do you think applying field and document level security affects query speed significantly? Commit to your answer.

Concept: Security filtering can impact Elasticsearch performance and has limitations.

Applying document and field level security adds extra filtering steps during queries. This can slow down searches, especially on large datasets or complex rules. Also, some features like aggregations may behave differently or be limited when security filters apply. Knowing these trade-offs helps design balanced security.

Result

You recognize the performance cost and functional limits of fine-grained security.

Knowing these limits prevents surprises and guides better security and performance trade-offs.

Under the Hood

Elasticsearch enforces field and document level security by modifying the query execution process. When a user sends a search request, Elasticsearch applies the document level security filter as an additional query layer that excludes unauthorized documents. Then, field level security removes or masks unauthorized fields from the returned documents before sending results. This happens transparently during query execution, ensuring users only receive allowed data.

Why designed this way?

This design allows flexible, fine-grained access control without duplicating data or creating separate indices. Applying filters at query time keeps data centralized and consistent. Alternatives like duplicating data per user or pre-filtering data would be inefficient and hard to maintain. The chosen approach balances security, flexibility, and performance.

User Query
   │
   ▼
┌─────────────────────────────┐
│ Elasticsearch Query Engine   │
│ ┌─────────────────────────┐ │
│ │ Document Level Filter   │ │
│ │ (exclude unauthorized)  │ │
│ └─────────────────────────┘ │
│ ┌─────────────────────────┐ │
│ │ Field Level Filter      │ │
│ │ (remove unauthorized)   │ │
│ └─────────────────────────┘ │
└─────────────────────────────┘
   │
   ▼
Filtered Results to User

Myth Busters - 4 Common Misconceptions

Quick: Does document level security hide parts of a document or whole documents? Commit to your answer.

Common Belief: Document level security hides parts of a document from users.

Reality: Document level security hides entire documents, not parts of them.

Why it matters: Misunderstanding this can lead to incorrect security rules, exposing sensitive fields inside documents users should not see.

Quick: Can field level security prevent users from seeing documents entirely? Commit to your answer.

Common Belief: Field level security can hide entire documents from users.

Reality: Field level security only hides specific fields inside documents; it does not hide whole documents.

Why it matters: Confusing this can cause gaps in security where users see documents they should not access.

Quick: Does applying field and document level security have no impact on query performance? Commit to your answer.

Common Belief: Security filtering does not affect Elasticsearch query speed.

Reality: Applying these security filters adds overhead and can slow down queries, especially with complex rules or large data.

Why it matters: Ignoring performance impact can cause slow user experiences and system strain.

Quick: Can you rely on field and document level security alone for all Elasticsearch security needs? Commit to your answer.

Common Belief: Field and document level security alone fully protect Elasticsearch data.

Reality: They are important but should be combined with other security layers like transport encryption, authentication, and audit logging.

Why it matters: Relying on only these controls can leave gaps in overall system security.

Expert Zone

1

Field level security can cause unexpected behavior with nested or object fields if not carefully configured.

2

Document level security filters are applied at query time, so complex filters can impact cluster performance and caching.

3

Role inheritance and overlapping permissions can create subtle access combinations that require careful testing.

When NOT to use

Avoid using field and document level security when performance is critical and data sensitivity is low; instead, use index-level security or separate indices. For very complex access rules, consider external proxy filtering or data masking solutions.

Production Patterns

In production, teams often combine document level security with field level security in roles mapped to user groups. They test security rules with sample queries and monitor performance impact. Audit logging tracks access to sensitive fields and documents. Security rules are updated as business needs evolve.

Connections

Role-Based Access Control (RBAC)

Field and document level security build on RBAC by adding fine-grained data filtering.

Understanding RBAC helps grasp how Elasticsearch assigns security rules before filtering data.

Data Masking

Field level security is a form of data masking by hiding sensitive fields from users.

Knowing data masking concepts clarifies why and how fields are hidden instead of removed.

Information Hiding in Software Engineering

Both concepts aim to restrict access to internal details to protect integrity and privacy.

Recognizing this connection shows how security principles apply broadly beyond databases.

Common Pitfalls

#1 Trying to hide sensitive fields by only using document level security.

Wrong approach:

role {
indices: ['*']
privileges: ['read']
query: { term: { region: 'us-east' } }
// No field level security defined
}

Correct approach:

role {
indices: ['*']
privileges: ['read']
query: { term: { region: 'us-east' } }
field_security: { grant: ['name', 'email', 'region'] }
}

Root cause:

Confusing document filtering with field filtering leads to sensitive fields being exposed inside allowed documents.

#2 Assigning roles directly to users without using role mappings.

Wrong approach:

user {
username: 'alice'
roles: ['sales_role']
// No role mapping used
}

Correct approach:

role_mapping {
roles: ['sales_role']
users: ['alice']
}

Root cause:

Not using role mappings reduces flexibility and complicates managing access for multiple users or groups.

#3 Assuming field level security hides nested object fields automatically.

Wrong approach:

field_security: { grant: ['user.*'] } // expecting all nested fields visible

Correct approach:

field_security: { grant: ['user.name', 'user.email'] } // explicitly list nested fields

Root cause:

Misunderstanding how nested fields are handled causes unexpected data exposure or hiding.

Key Takeaways

Field and document level security let you control exactly what data users can see in Elasticsearch.

Document level security hides entire documents based on rules, while field level security hides specific fields inside documents.

Combining both provides powerful, flexible protection for sensitive data.

These security features work by filtering data during query execution, which can affect performance.

Proper role mappings and testing are essential to manage and verify security effectively.