Cross-cluster search in Elasticsearch - Deep Dive

Overview - Cross-cluster search

What is it?

Cross-cluster search allows you to search data stored in multiple Elasticsearch clusters as if they were one. It lets you send a single search request that retrieves results from different clusters without moving or copying data. This helps when your data is spread across locations or clusters for scale or organization.

Why it matters

Without cross-cluster search, you would need to manually gather data from each cluster and combine it yourself, which is slow and complex. This feature saves time and effort by making distributed data searchable instantly, improving decision-making and user experience. It also supports scaling and data isolation while keeping search unified.

Where it fits

Before learning cross-cluster search, you should understand basic Elasticsearch concepts like clusters, nodes, indices, and search queries. After mastering it, you can explore advanced topics like cross-cluster replication, security settings for remote clusters, and optimizing distributed search performance.

Mental Model

Core Idea

Cross-cluster search lets you query multiple Elasticsearch clusters at once, treating them like one big cluster for searching.

Think of it like...

Imagine you want to find a book in a library system with many branches. Instead of visiting each branch separately, cross-cluster search is like having a librarian who checks all branches at once and brings you the combined list of books.

┌─────────────────────────────┐
│       Client Search Query    │
└──────────────┬──────────────┘
               │
   ┌───────────┴───────────┐
   │ Cross-Cluster Search   │
   └───────┬─────────┬─────┘
           │         │
 ┌─────────┴─┐ ┌─────┴────────┐
 │ Cluster A │ │  Cluster B   │
 │ (Index 1) │ │ (Index 2)    │
 └───────────┘ └──────────────┘
           │         │
   ┌───────┴─────────┴─────┐
   │  Combined Search Results│
   └────────────────────────┘

Build-Up - 7 Steps

1

Foundation

Understanding Elasticsearch Clusters

Concept: Learn what an Elasticsearch cluster is and how it stores data.

An Elasticsearch cluster is a group of one or more servers (nodes) that hold your data and provide search capabilities. Each cluster manages its own indices, which are collections of documents. Clusters are independent and do not share data by default.

Result

You understand that data is stored and searched within a single cluster, and clusters operate separately.

Knowing that clusters are independent units helps you see why searching across multiple clusters needs a special method.

2

Foundation

Basics of Elasticsearch Search Queries

Concept: Learn how to write a simple search query in Elasticsearch.

A search query in Elasticsearch asks the cluster to find documents matching certain criteria. For example, a query can look for all documents where the field 'title' contains 'database'. You send this query to one cluster, and it returns matching documents.

Result

You can perform basic searches within a single cluster and get relevant results.

Understanding how queries work within one cluster sets the stage for extending searches across clusters.

3

Intermediate

Configuring Remote Clusters for Search

🤔Before reading on: do you think you can search another cluster without any setup? Commit to yes or no.

Concept: Introduce the idea of connecting clusters by configuring remote cluster settings.

To search multiple clusters, you must tell your local cluster about the remote clusters. This is done by adding remote cluster configurations, which include the remote cluster's name and address. This setup allows your cluster to send search requests to the remote cluster securely.

Result

Your local cluster knows how to reach remote clusters and can include them in search queries.

Understanding that clusters must be connected before searching across them prevents confusion about why cross-cluster search doesn't work out of the box.

4

Intermediate

Writing Cross-Cluster Search Queries

🤔Before reading on: do you think you need separate queries for each cluster or one combined query? Commit to your answer.

Concept: Learn how to write a single query that searches multiple clusters using special index syntax.

In cross-cluster search, you use a special prefix to specify remote clusters in your index names. For example, 'clusterA:index1,clusterB:index2' tells Elasticsearch to search 'index1' in 'clusterA' and 'index2' in 'clusterB' together. You send one query, and Elasticsearch merges the results.

Result

You can retrieve combined search results from multiple clusters with one query.

Knowing the syntax to target multiple clusters in one query simplifies distributed search and avoids manual result merging.

5

Intermediate

Handling Result Merging and Sorting

🤔Before reading on: do you think Elasticsearch merges results from clusters automatically or requires manual merging? Commit to your answer.

Concept: Understand how Elasticsearch combines and sorts results from different clusters in cross-cluster search.

Elasticsearch merges results from all queried clusters and sorts them according to your query's criteria, like relevance or date. This happens automatically, so you get a unified, ordered list as if all data was in one place.

Result

Search results appear as a single sorted list, hiding the complexity of multiple clusters.

Knowing that Elasticsearch handles merging and sorting helps you trust cross-cluster search results without extra work.

6

Advanced

Security and Access Control in Cross-Cluster Search

🤔Before reading on: do you think cross-cluster search ignores security settings on remote clusters? Commit to yes or no.

Concept: Learn how security is managed when searching across clusters, including authentication and permissions.

Cross-cluster search respects security settings on remote clusters. You must configure credentials and roles so the local cluster can access remote data safely. This prevents unauthorized access and ensures compliance with data policies.

Result

Cross-cluster search works securely, protecting data across clusters.

Understanding security integration prevents accidental data leaks and builds trust in distributed search.

7

Expert

Performance Considerations and Limitations

🤔Before reading on: do you think cross-cluster search always performs as fast as single-cluster search? Commit to yes or no.

Concept: Explore how network latency, cluster load, and query complexity affect cross-cluster search performance.

Cross-cluster search involves network calls to remote clusters, which can add latency. Large result sets or complex queries increase load and slow responses. Elasticsearch provides settings to limit connections and timeouts. Understanding these helps optimize performance and avoid bottlenecks.

Result

You can design cross-cluster search setups that balance speed and completeness.

Knowing performance trade-offs helps you make informed decisions about when and how to use cross-cluster search effectively.

Under the Hood

Cross-cluster search works by the local cluster acting as a coordinator. When you send a query, it parses the target indices and identifies which belong to remote clusters. It then forwards sub-queries to those clusters over the network, collects their responses, merges and sorts the results, and returns the combined list to you. This coordination happens transparently, using configured remote cluster connections and security credentials.

Why designed this way?

Elasticsearch was designed as a distributed system with independent clusters for scalability and fault isolation. Cross-cluster search was added to allow unified search without forcing data centralization, which can be costly or impractical. The design balances flexibility, security, and performance by keeping clusters autonomous but searchable together.

┌───────────────┐
│ Client Query  │
└───────┬───────┘
        │
┌───────▼─────────────┐
│ Local Cluster (Coord)│
│  ┌───────────────┐  │
│  │Identify Indices│  │
│  └──────┬────────┘  │
│         │           │
│ ┌───────▼───────┐   │
│ │Local Indices  │   │
│ └───────────────┘   │
│         │           │
│ ┌───────▼────────┐  │
│ │Remote Clusters │  │
│ │  ┌───────────┐│  │
│ │  │Cluster A  ││  │
│ │  └───────────┘│  │
│ │  ┌───────────┐│  │
│ │  │Cluster B  ││  │
│ │  └───────────┘│  │
│ └───────────────┘  │
│         │           │
│  Merge & Sort Results│
└─────────┬───────────┘
          │
    ┌─────▼─────┐
    │ Final     │
    │ Results   │
    └───────────┘

Myth Busters - 4 Common Misconceptions

Quick: Do you think cross-cluster search copies data between clusters automatically? Commit to yes or no.

Common Belief: Cross-cluster search copies or moves data from remote clusters to the local cluster before searching.

Reality: Cross-cluster search does not move or copy data; it queries remote clusters live and merges results on the fly.

Why it matters: Believing data is copied can lead to unnecessary data duplication, storage costs, and confusion about data freshness.

Quick: Do you think cross-cluster search ignores security settings on remote clusters? Commit to yes or no.

Common Belief: Cross-cluster search bypasses security controls on remote clusters, so once connected, all data is accessible.

Reality: Security settings and permissions on remote clusters are enforced during cross-cluster search; proper credentials are required.

Why it matters: Ignoring security can cause data breaches or unauthorized access, risking compliance and trust.

Quick: Do you think cross-cluster search always performs as fast as searching a single cluster? Commit to yes or no.

Common Belief: Cross-cluster search is just as fast as searching a single cluster because Elasticsearch is distributed.

Reality: Cross-cluster search can be slower due to network latency and the overhead of merging results from multiple clusters.

Why it matters: Expecting single-cluster speed can cause frustration and poor system design if performance limits are not considered.

Quick: Do you think you can search any cluster without configuring it as a remote cluster? Commit to yes or no.

Common Belief: You can search any Elasticsearch cluster remotely without prior configuration.

Reality: You must configure remote cluster connections before cross-cluster search can access them.

Why it matters: Not configuring remote clusters leads to failed queries and wasted troubleshooting time.

Expert Zone

1

Cross-cluster search respects index-level settings like aliases and mappings, which can affect how results are merged and interpreted.

2

Remote cluster connections use a dedicated transport protocol optimized for Elasticsearch, not standard HTTP, improving efficiency and security.

3

Cross-cluster search can be combined with cross-cluster replication to create resilient, searchable multi-region deployments.

When NOT to use

Avoid cross-cluster search when ultra-low latency is critical or when data volumes are small enough to consolidate into a single cluster. Instead, use cross-cluster replication or centralized indexing to reduce network overhead.

Production Patterns

In production, cross-cluster search is used to query data across geographic regions for global applications, to separate data by business units while enabling unified search, and to support multi-tenant architectures where clusters isolate tenant data but share search capabilities.

Connections

Distributed Systems

Cross-cluster search builds on distributed system principles of coordination and data partitioning.

Understanding distributed systems helps grasp how queries are split, sent, and results merged across independent clusters.

Federated Search

Cross-cluster search is a form of federated search where multiple independent data sources are queried together.

Knowing federated search concepts clarifies challenges like result merging, latency, and security in cross-cluster search.

Supply Chain Management

Like cross-cluster search aggregates data from multiple clusters, supply chain management integrates information from various suppliers to optimize delivery.

Seeing cross-cluster search as data supply chain coordination reveals the importance of connection, timing, and trust between independent entities.

Common Pitfalls

#1 Trying to search a remote cluster without configuring it as a remote cluster.

Wrong approach:

GET /clusterB:index/_search { "query": { "match_all": {} } }

Correct approach:

PUT /_cluster/settings { "persistent": { "search.remote.clusterB.seeds": ["hostB:9300"] } }

GET /clusterB:index/_search { "query": { "match_all": {} } }

Root cause:

Assuming remote clusters are searchable without setup causes connection failures.

#2 Using the same index name without cluster prefix, expecting cross-cluster search.

Wrong approach:

GET /index/_search { "query": { "match": { "field": "value" } } }

Correct approach:

GET /clusterA:index,clusterB:index/_search { "query": { "match": { "field": "value" } } }

Root cause:

Not specifying cluster prefixes means the query only targets local indices.

#3 Ignoring security configuration and expecting access to remote cluster data.

Wrong approach:

Configuring remote cluster without credentials and querying data.

Correct approach:

Configure remote cluster with proper credentials and roles before querying.

Root cause:

Overlooking security leads to authorization errors and data access denial.

Key Takeaways

Cross-cluster search lets you query multiple Elasticsearch clusters with one search request, combining results transparently.

You must configure remote clusters before searching them, including network addresses and security credentials.

Elasticsearch merges and sorts results from all clusters automatically, providing a unified view of distributed data.

Performance depends on network latency and query complexity; cross-cluster search is not always as fast as single-cluster search.

Security settings on remote clusters are enforced, so proper access control is essential for safe cross-cluster search.