API key management in Elasticsearch - Deep Dive

Overview - API key management

What is it?

API key management in Elasticsearch is the process of creating, storing, and controlling keys that allow applications or users to access Elasticsearch securely. These keys act like passwords but are designed specifically for programmatic access. They help control who can do what in Elasticsearch without sharing user passwords. This makes it easier and safer to connect apps to Elasticsearch.

Why it matters

Without API key management, applications would need to use user passwords or less secure methods to access Elasticsearch. This increases the risk of unauthorized access and data leaks. Proper API key management ensures that access is limited, monitored, and can be revoked easily, protecting sensitive data and maintaining system security. It also simplifies access control for developers and administrators.

Where it fits

Before learning API key management, you should understand basic Elasticsearch concepts like clusters, nodes, and security basics such as users and roles. After mastering API key management, you can explore advanced security topics like role-based access control, audit logging, and integrating Elasticsearch with external identity providers.

Mental Model

Core Idea

API key management is about securely issuing and controlling special passwords that let programs access Elasticsearch with limited permissions.

Think of it like...

It's like giving a valet key to your car: it lets someone drive the car but only in a limited way, without giving full access to everything.

┌─────────────────────────────┐
│       Elasticsearch          │
│  ┌───────────────┐          │
│  │ API Key Store │◄─────────┤
│  └───────────────┘          │
│          ▲                  │
│          │                  │
│  ┌───────┴────────┐         │
│  │ Client App/API │         │
│  └────────────────┘         │
└─────────────────────────────┘

Flow:
Client sends API key → Elasticsearch checks key in API Key Store → Grants limited access

Build-Up - 6 Steps

1

Foundation

What is an API Key in Elasticsearch

Concept: Introduce the basic idea of an API key as a special token for access.

An API key in Elasticsearch is a unique string that acts like a password but is meant for programs or apps to access Elasticsearch. Instead of using a username and password, apps use API keys to prove who they are. Each key can have specific permissions, so it controls what the app can do.

Result

You understand that API keys are special access tokens different from user passwords.

Knowing that API keys are separate from user credentials helps you see how Elasticsearch keeps program access secure and manageable.

2

Foundation

Creating and Using API Keys

Concept: Learn how to create API keys and use them to access Elasticsearch.

You create an API key by sending a request to Elasticsearch with the permissions you want the key to have. Elasticsearch returns the key string. Your app then includes this key in its requests to Elasticsearch to authenticate. For example, you can create a key that only allows reading data from certain indexes.

Result

You can generate an API key and use it to connect an app to Elasticsearch with controlled access.

Understanding the creation and use process shows how API keys enable secure, permission-limited access.

3

Intermediate

API Key Permissions and Scopes

🤔Before reading on: Do you think an API key can have unlimited access by default or must permissions be explicitly set? Commit to your answer.

Concept: API keys have specific permissions that limit what actions they can perform and on which data.

When creating an API key, you specify roles or privileges that define what the key can do. For example, a key might only allow searching certain indexes or writing data but not deleting. This limits damage if the key is leaked. Permissions are set using role descriptors similar to user roles.

Result

API keys enforce fine-grained access control, restricting actions and data visibility.

Knowing that API keys are not all-powerful by default helps prevent security risks and encourages least-privilege access.

4

Intermediate

Managing API Keys Lifecycle

🤔Before reading on: Do you think API keys in Elasticsearch expire automatically or must be revoked manually? Commit to your answer.

Concept: API keys have a lifecycle: creation, usage, expiration, and revocation.

You can set an expiration time when creating an API key, after which it stops working. You can also revoke keys manually if you suspect they are compromised. Elasticsearch keeps track of active keys and their status. Proper lifecycle management ensures keys don't stay valid forever, reducing risk.

Result

You can control how long API keys work and disable them when needed.

Understanding lifecycle management is key to maintaining security over time and responding to threats.

5

Advanced

Security Best Practices for API Keys

🤔Before reading on: Should API keys be shared openly in code or stored securely? Commit to your answer.

Concept: Learn how to handle API keys safely to avoid leaks and misuse.

API keys should never be hardcoded in public code or shared openly. Store them securely using environment variables or secret managers. Use the least privilege principle when assigning permissions. Rotate keys regularly and monitor their usage for unusual activity. These practices reduce the chance of unauthorized access.

Result

You know how to protect API keys in real projects to keep Elasticsearch safe.

Knowing security best practices prevents common mistakes that lead to data breaches.

6

Expert

Internal Mechanics of API Key Authentication

🤔Before reading on: Do you think Elasticsearch stores API keys as plain text or uses a secure method? Commit to your answer.

Concept: Explore how Elasticsearch stores and verifies API keys internally.

Elasticsearch stores API keys securely by hashing the key material, not in plain text. When a request comes with an API key, Elasticsearch hashes the provided key and compares it to stored hashes. This prevents attackers from reading keys even if they access the storage. The system also links keys to roles and expiration metadata for access control.

Result

You understand the secure storage and verification process behind API keys.

Knowing the internal security mechanisms builds trust in Elasticsearch's design and helps diagnose issues.

Under the Hood

When an API key is created, Elasticsearch generates a unique identifier and a secret key. The secret is hashed and stored securely. The key is associated with specific roles and expiration metadata. When a client sends a request with an API key, Elasticsearch extracts the key, hashes it, and compares it to stored hashes. If matched and not expired or revoked, Elasticsearch grants access based on the key's roles. This process happens quickly to authenticate every request without exposing secrets.

Why designed this way?

Storing hashed keys prevents attackers from stealing usable keys if the storage is compromised. Associating keys with roles and expiration allows fine-grained, time-limited access control. This design balances security, performance, and flexibility. Alternatives like storing plain keys or using only user credentials were less secure or less convenient for programmatic access.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│ Client sends │──────▶│ Elasticsearch  │──────▶│ API Key Store │
│ API key      │       │ hashes key    │       │ stores hashed │
│              │       │ and checks    │       │ keys + roles  │
└───────────────┘       └───────────────┘       └───────────────┘
         │                      │                      ▲
         │                      │                      │
         │                      └──────────────────────┘
         │                             Checks roles & expiration
         ▼
┌─────────────────┐
│ Access granted  │
│ if valid &      │
│ permissions ok  │
└─────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Do you think API keys automatically expire after a set time? Commit to yes or no.

Common Belief: API keys in Elasticsearch expire automatically after creation.

Reality: API keys do not expire unless you explicitly set an expiration time when creating them or revoke them manually.

Why it matters: Assuming automatic expiration can lead to forgotten keys remaining active indefinitely, increasing security risks.

Quick: Do you think API keys grant full user privileges by default? Commit to yes or no.

Common Belief: API keys have the same permissions as the user who created them by default.

Reality: API keys only have the permissions explicitly assigned during creation, which can be less than the creator's full privileges.

Why it matters: Believing keys have full permissions can cause over-permissioning, leading to potential misuse or data leaks.

Quick: Do you think storing API keys in plain text is safe if your server is secure? Commit to yes or no.

Common Belief: It's safe to store API keys in plain text as long as the server is protected.

Reality: Elasticsearch stores API keys hashed to protect them even if storage is accessed by attackers; plain text storage is insecure.

Why it matters: Storing keys in plain text risks exposure if the server is compromised, leading to unauthorized access.

Quick: Do you think API keys can be used to authenticate users interactively? Commit to yes or no.

Common Belief: API keys are meant for user login and interactive sessions.

Reality: API keys are designed for programmatic access, not for interactive user authentication.

Why it matters: Misusing API keys for user login can cause confusion and weaken security controls.

Expert Zone

1

API keys can be created with limited privileges that differ from the creator's roles, enabling fine-grained delegation.

2

Elasticsearch allows multiple API keys per user or application, which can be rotated independently for better security.

3

API key revocation is immediate but requires careful tracking in distributed clusters to ensure all nodes recognize the change.

When NOT to use

Avoid using API keys when interactive user authentication is needed; use user credentials or OAuth instead. For very short-lived access, consider using tokens from an external identity provider. If you need complex multi-factor authentication, API keys alone are insufficient.

Production Patterns

In production, API keys are often stored securely in environment variables or secret management systems. Teams rotate keys regularly and monitor usage logs for anomalies. Keys are scoped narrowly to specific indexes and actions. Some systems automate key creation and revocation as part of deployment pipelines.

Connections

OAuth 2.0 Access Tokens

Both are tokens used for programmatic access with limited permissions.

Understanding API keys helps grasp OAuth tokens, as both control access without sharing passwords.

Password Management

API keys serve a similar purpose to passwords but are designed for machines, not humans.

Knowing how API keys differ from passwords clarifies why separate management and security practices are needed.

Physical Security Badges

API keys are like badges that grant access to specific rooms or areas.

This cross-domain link shows how limiting access by role and time is a universal security principle.

Common Pitfalls

#1 Hardcoding API keys in public code repositories.

Wrong approach:

curl -H 'Authorization: ApiKey ABC123...' http://localhost:9200/_search

Correct approach:

Store the API key in an environment variable and reference it securely in code or scripts.

Root cause:

Lack of awareness about secure storage leads to accidental exposure of keys.

#2 Creating API keys with overly broad permissions.

Wrong approach:

POST /_security/api_key
{
"name": "all_access_key",
"role_descriptors": {
"all_access": {
"cluster": ["all"],
"index": [{"names": ["*"], "privileges": ["all"]}]
}
}
}

Correct approach:

Limit permissions to only needed actions and indexes, e.g., read-only on specific indexes.

Root cause:

Not applying the principle of least privilege increases risk if the key is compromised.

#3 Assuming API keys expire automatically and forgetting to revoke them.

Wrong approach:

Creating keys without expiration and never revoking them.

Correct approach:

Set expiration times on keys and revoke unused keys promptly.

Root cause:

Misunderstanding key lifecycle management leads to stale, vulnerable keys.

Key Takeaways

API key management in Elasticsearch provides secure, limited access for applications without sharing user passwords.

Each API key has specific permissions and an optional expiration, enabling fine control over access.

Proper lifecycle management, including creation, usage, expiration, and revocation, is essential for security.

Storing API keys securely and following best practices prevents accidental leaks and misuse.

Understanding the internal hashing and verification mechanisms builds confidence in Elasticsearch's security design.

Practice

1. What is the primary purpose of an API key in Elasticsearch?

easy

A. To monitor Elasticsearch cluster health

B. To store data inside Elasticsearch indices

C. To allow applications to securely access Elasticsearch with specific permissions

D. To backup Elasticsearch data automatically

Solution

1. Step 1: Understand API key role

   API keys are secret tokens used to authenticate and authorize applications.

2. Step 2: Identify purpose in Elasticsearch

   They grant controlled access to Elasticsearch resources based on assigned roles.

3. Final Answer:

   To allow applications to securely access Elasticsearch with specific permissions -> Option C

4. Quick Check:

   API key = secure app access [OK]

Hint: API keys control app access permissions [OK]

Common Mistakes:

• Confusing API keys with data storage
• Thinking API keys monitor cluster health
• Assuming API keys handle backups

2. Which of the following is the correct Elasticsearch API call to create an API key?

easy

A. DELETE /_security/api_key

B. GET /_security/api_key/create

C. PUT /_security/api_key

D. POST /_security/api_key

Solution

1. Step 1: Recall API key creation syntax

   Elasticsearch uses POST method to create resources like API keys.

2. Step 2: Match correct endpoint

   The correct endpoint for creating an API key is POST /_security/api_key.

3. Final Answer:

   POST /_security/api_key -> Option D

4. Quick Check:

   POST + /_security/api_key = create key [OK]

Hint: Use POST to create API keys in Elasticsearch [OK]

Common Mistakes:

• Using GET or DELETE for creation
• Confusing endpoint paths
• Using PUT instead of POST

3. Given this API key creation request body, what will be the name of the created API key?

{
  "name": "my-app-key",
  "role_descriptors": {
    "my-role": {
      "cluster": ["all"],
      "index": [{"names": ["logs-*"], "privileges": ["read"]}]
    }
  }
}

medium

A. my-app-key

B. my-role

C. logs-*

D. all

Solution

1. Step 1: Identify the API key name field

   The "name" field in the request body sets the API key's name.

2. Step 2: Read the value of the "name" field

   The value is "my-app-key", which becomes the API key's name.

3. Final Answer:

   my-app-key -> Option A

4. Quick Check:

   API key name = "name" field value [OK]

Hint: API key name is in the "name" field [OK]

Common Mistakes:

• Confusing role name with API key name
• Using index pattern as key name
• Mistaking privileges for name

4. You try to delete an API key using this request: DELETE /_security/api_key?id=12345 but get an error. What is the likely cause?

medium

A. API key names cannot be deleted, only IDs

B. API key ID must be passed in the request body, not as a query parameter

C. DELETE method is not supported for API keys

D. You must use GET method to delete API keys

Solution

1. Step 1: Check API key deletion syntax

   Elasticsearch requires the API key ID in the request body JSON, not as a URL query parameter.

2. Step 2: Understand method support

   DELETE method is supported, but parameters must be correctly passed in the body.

3. Final Answer:

   API key ID must be passed in the request body, not as a query parameter -> Option B

4. Quick Check:

   Delete API key ID in body, not URL [OK]

Hint: Pass API key ID in JSON body for deletion [OK]

Common Mistakes:

• Passing ID as URL query parameter
• Using wrong HTTP method
• Confusing API key name with ID

5. You want to create an API key that only allows reading from indices starting with "sales-" and no cluster privileges. Which role descriptor is correct in the request body?

hard

A. { "role_descriptors": { "read_sales": { "cluster": [], "index": [{ "names": ["sales-*"], "privileges": ["read"] }] } } }

B. { "role_descriptors": { "read_sales": { "cluster": ["all"], "index": [{ "names": ["sales-*"], "privileges": ["write"] }] } } }

C. { "role_descriptors": { "read_sales": { "cluster": ["monitor"], "index": [{ "names": ["sales-*"], "privileges": ["all"] }] } } }

D. { "role_descriptors": { "read_sales": { "cluster": ["all"], "index": [{ "names": ["*"], "privileges": ["read"] }] } } }

Solution

1. Step 1: Identify required privileges

   The API key should have no cluster privileges and only read privileges on indices starting with "sales-".

2. Step 2: Match role descriptor to requirements

   { "role_descriptors": { "read_sales": { "cluster": [], "index": [{ "names": ["sales-*"], "privileges": ["read"] }] } } } has empty cluster privileges and read privilege on "sales-*" indices, matching the requirement.

3. Final Answer:

   { "role_descriptors": { "read_sales": { "cluster": [], "index": [{ "names": ["sales-*"], "privileges": ["read"] }] } } } -> Option A

4. Quick Check:

   No cluster + read sales-* = { "role_descriptors": { "read_sales": { "cluster": [], "index": [{ "names": ["sales-*"], "privileges": ["read"] }] } } } [OK]

Hint: Empty cluster array means no cluster privileges [OK]

Common Mistakes:

• Giving cluster all privileges by mistake
• Using write or all privileges instead of read
• Applying privileges to wrong index patterns