Cluster, node, and shard architecture in Elasticsearch - Mini Project: Build & Apply

Cluster, node, and shard architecture

📖 Scenario: You are setting up a simple Elasticsearch environment to understand how clusters, nodes, and shards work together to store and manage data efficiently.

🎯 Goal: You will create a basic Elasticsearch cluster configuration with nodes and shards, then query the cluster to see shard allocation.

📋 What You'll Learn

Create a cluster configuration with two nodes

Assign 3 primary shards and 1 replica shard to the index

Query the cluster to display shard allocation

💡 Why This Matters

🌍 Real World

Understanding cluster, node, and shard architecture helps in designing scalable and fault-tolerant search systems.

💼 Career

Knowledge of Elasticsearch architecture is essential for roles like DevOps engineers, backend developers, and data engineers working with search and analytics.

1

Create cluster and node settings

Create a dictionary called cluster_settings with the following keys and values: "cluster.name" set to "my_cluster", and "node.name" set to "node_1".

Elasticsearch

# Create cluster_settings dictionary with cluster.name and node.name
# Your code here

Need a hint?

Use a Python dictionary with keys as strings and values as strings.

2

Add index shard configuration

Create a dictionary called index_settings with "number_of_shards" set to 3 and "number_of_replicas" set to 1.

Elasticsearch

cluster_settings = {
    "cluster.name": "my_cluster",
    "node.name": "node_1"
}
# Create index_settings dictionary with number_of_shards and number_of_replicas
# Your code here

Need a hint?

Use integers for shard and replica counts.

3

Simulate shard allocation

Create a list called shard_allocation that contains dictionaries for each shard. Each dictionary should have keys "shard_id" (values 0 to 2) and "node" set to "node_1" for primary shards, and one dictionary with "shard_id" 0 and "node" set to "node_2" for the replica shard.

Elasticsearch

cluster_settings = {
    "cluster.name": "my_cluster",
    "node.name": "node_1"
}

index_settings = {
    "number_of_shards": 3,
    "number_of_replicas": 1
}
# Create shard_allocation list with primary and replica shards
# Your code here

Need a hint?

Use a list of dictionaries to represent shards and their assigned nodes.

4

Display shard allocation

Write a print statement to display the shard_allocation list.

Elasticsearch

cluster_settings = {
    "cluster.name": "my_cluster",
    "node.name": "node_1"
}

index_settings = {
    "number_of_shards": 3,
    "number_of_replicas": 1
}

shard_allocation = [
    {"shard_id": 0, "node": "node_1"},
    {"shard_id": 1, "node": "node_1"},
    {"shard_id": 2, "node": "node_1"},
    {"shard_id": 0, "node": "node_2"}
]
# Print the shard_allocation list
# Your code here

Need a hint?

Use print(shard_allocation) to show the list.