Adding and removing nodes in Redis - Deep Dive

Overview - Adding and removing nodes

What is it?

Adding and removing nodes in Redis means changing the number of servers (nodes) in a Redis cluster. Nodes are individual Redis instances that work together to store and manage data. Adding nodes helps the cluster grow and handle more data or traffic. Removing nodes shrinks the cluster, often for maintenance or cost-saving.

Why it matters

Without the ability to add or remove nodes, a Redis cluster would be stuck with a fixed size. This limits how much data it can store and how well it can handle many users at once. Being able to change the number of nodes lets Redis adapt to real-world needs, like growing a website or fixing broken servers without downtime.

Where it fits

Before learning this, you should understand what a Redis cluster is and how data is split across nodes (sharding). After this, you can learn about cluster rebalancing and failover to keep the cluster healthy and efficient.

Mental Model

Core Idea

Adding or removing nodes changes the Redis cluster size and requires moving data to keep everything balanced and available.

Think of it like...

Imagine a team of delivery drivers (nodes) sharing packages (data). Adding a driver means some packages get reassigned to the new driver to share the work. Removing a driver means their packages must be given to others so no package is lost.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│   Node 1     │◄──────│   Node 2     │──────►│   Node 3     │
│  (Redis)    │       │  (Redis)    │       │  (Redis)    │
└───────────────┘       └───────────────┘       └───────────────┘
       ▲                      ▲                      ▲
       │                      │                      │
   Data slots             Data slots             Data slots
   assigned               assigned               assigned

Adding a node inserts a new box and redistributes data slots.
Removing a node removes a box and moves its data slots to others.

Build-Up - 7 Steps

1

Foundation

What is a Redis node?

Concept: A Redis node is a single Redis server instance that stores part of the data in a cluster.

A Redis cluster is made of multiple nodes. Each node holds a subset of the total data, called hash slots. The cluster splits 16,384 hash slots among all nodes. Each node is responsible for some slots and the keys that map to them.

Result

You understand that a node is a building block of a Redis cluster, holding some data.

Knowing what a node is helps you see why adding or removing nodes affects data distribution.

2

Foundation

How data is distributed in Redis cluster

Concept: Redis uses hash slots to split data evenly across nodes.

Redis assigns each key to one of 16,384 hash slots using a hash function. These slots are divided among nodes. When you add or remove nodes, the slots must be reassigned to keep data balanced.

Result

You see that data is not randomly placed but carefully split by hash slots.

Understanding hash slots is key to grasping why adding/removing nodes requires moving data.

3

Intermediate

Adding a node to a Redis cluster

🤔Before reading on: do you think adding a node automatically moves data, or does it require manual steps? Commit to your answer.

Concept: Adding a node means assigning it some hash slots and moving keys from existing nodes to it.

To add a node, you start a new Redis instance and join it to the cluster. Then you use the 'reshard' process to move some hash slots from existing nodes to the new one. This moves the keys for those slots to the new node, balancing the cluster.

Result

The cluster grows, and data is redistributed so the new node holds part of the data.

Knowing that adding nodes involves moving data helps you plan for the time and resources needed during scaling.

4

Intermediate

Removing a node from a Redis cluster

🤔Before reading on: do you think removing a node deletes its data or moves it elsewhere? Commit to your answer.

Concept: Removing a node requires moving its hash slots and keys to other nodes before safely removing it.

To remove a node, you first migrate its hash slots to other nodes using resharding. Once all data is moved, you can safely remove the node from the cluster. This prevents data loss and keeps the cluster balanced.

Result

The cluster shrinks, but all data remains available on remaining nodes.

Understanding that data must be moved before removal prevents accidental data loss.

5

Intermediate

Resharding: moving hash slots between nodes

🤔Before reading on: do you think resharding can happen automatically or only manually? Commit to your answer.

Concept: Resharding is the process of moving hash slots and their keys between nodes to balance data.

Redis provides commands and tools to reshard data. You specify which slots to move and to which node. Redis then migrates keys for those slots, updating cluster state. This is needed when adding or removing nodes.

Result

Data is moved smoothly between nodes without downtime.

Knowing how resharding works helps you manage cluster size changes safely.

6

Advanced

Handling cluster state during node changes

🤔Before reading on: do you think cluster state updates happen instantly or require coordination? Commit to your answer.

Concept: The cluster must update its internal state to reflect new node assignments and maintain consistency.

When nodes are added or removed, Redis nodes communicate to update slot assignments and cluster topology. This coordination ensures clients can find keys correctly. The cluster uses gossip protocol and configuration epochs to manage changes.

Result

Clients see a consistent view of the cluster and can route commands properly.

Understanding cluster state coordination explains how Redis avoids errors during node changes.

7

Expert

Surprises in node removal and failover

🤔Before reading on: do you think removing a node with replicas affects failover? Commit to your answer.

Concept: Removing nodes with replicas requires careful handling to avoid losing redundancy and causing failover issues.

If a node has replicas, you must promote a replica before removal or reassign replicas to other masters. Removing a master node without handling replicas can cause data unavailability or split-brain scenarios. Tools like redis-trib or redis-cli help manage this safely.

Result

Cluster remains highly available and consistent after node removal.

Knowing replica handling during node removal prevents costly downtime and data loss in production.

Under the Hood

Redis cluster divides data into 16,384 hash slots. Each node owns some slots. Adding or removing nodes triggers slot migration, where keys are copied from one node to another. Nodes communicate using a gossip protocol to share cluster state and slot ownership. Configuration epochs track changes to avoid conflicts. During migration, keys are copied and clients are redirected to the new owner.

Why designed this way?

Redis was designed for high availability and scalability. Splitting data into hash slots allows easy distribution and rebalancing. Gossip protocol enables decentralized cluster state sharing without a single point of failure. This design balances simplicity, speed, and fault tolerance. Alternatives like centralized management were rejected to avoid bottlenecks.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│   Node A     │◄──────│   Node B     │──────►│   Node C     │
│ Slots 0-5000 │       │ Slots 5001-10000│     │ Slots 10001-16383│
└───────────────┘       └───────────────┘       └───────────────┘
       │                      │                      │
       │  Migrate slots 4000-5000 to new node D       │
       ▼                      ▼                      ▼
┌───────────────┐
│   Node D     │
│ Slots 4000-5000│
└───────────────┘

Nodes update cluster state via gossip to reflect new slot ownership.

Myth Busters - 4 Common Misconceptions

Quick: Does removing a node delete its data permanently? Commit yes or no.

Common Belief: Removing a node deletes all its data immediately.

Reality: Data is migrated to other nodes before removal, so no data is lost.

Why it matters: Believing data is lost can cause panic or improper removal steps risking data loss.

Quick: Can you add a node and have it start serving data instantly without resharding? Commit yes or no.

Common Belief: Adding a node automatically balances data without manual intervention.

Reality: You must manually reshard to move hash slots and data to the new node.

Why it matters: Assuming automatic balancing leads to unbalanced clusters and performance issues.

Quick: Does removing a master node with replicas require special steps? Commit yes or no.

Common Belief: You can remove any node anytime without extra steps.

Reality: You must handle replicas carefully to avoid failover problems.

Why it matters: Ignoring replicas can cause downtime or data inconsistency.

Quick: Is cluster state updated instantly and globally when nodes change? Commit yes or no.

Common Belief: Cluster state changes propagate instantly to all nodes.

Reality: State updates use gossip protocol and take time to reach all nodes.

Why it matters: Expecting instant updates can cause confusion and errors during transitions.

Expert Zone

1

When resharding, migrating slots in small batches reduces impact on cluster performance and client latency.

2

Replica promotion before master removal must consider replication lag to avoid data loss.

3

Cluster configuration epochs prevent conflicting slot ownership but can cause temporary split-brain if mismanaged.

When NOT to use

Avoid adding or removing nodes frequently in very small clusters; instead, plan capacity ahead. For very large clusters, consider automated cluster management tools or managed Redis services that handle scaling transparently.

Production Patterns

In production, teams use rolling resharding during low traffic periods to add nodes. They automate replica promotion before master removal. Monitoring tools track slot distribution and cluster health to trigger scaling events.

Connections

Distributed Hash Table (DHT)

Redis cluster's hash slot distribution is a form of DHT used in distributed systems.

Understanding DHTs from peer-to-peer networks helps grasp how Redis distributes and locates data efficiently.

Load Balancing

Adding/removing nodes and resharding is similar to load balancing across servers.

Knowing load balancing principles clarifies why data must be moved to keep workload even.

Teamwork and Task Sharing

Adding/removing nodes is like changing team members and redistributing tasks.

This connection helps appreciate the importance of smooth transitions to avoid dropped work or confusion.

Common Pitfalls

#1 Removing a node without migrating its slots first.

Wrong approach:

redis-cli --cluster del-node 127.0.0.1:7001

Correct approach:

Use redis-cli --cluster reshard to move slots from node, then del-node after migration.

Root cause:

Misunderstanding that data must be moved before node removal to avoid data loss.

#2 Adding a node but not resharding slots to it.

Wrong approach:

Start new Redis node and join cluster, then do nothing else.

Correct approach:

After joining, run redis-cli --cluster reshard to assign slots to the new node.

Root cause:

Assuming new nodes automatically get data without manual slot migration.

#3 Removing a master node with replicas without promoting replicas.

Wrong approach:

Directly delete master node without handling replicas.

Correct approach:

Promote a replica to master or reassign replicas before removing the master node.

Root cause:

Ignoring replica roles and failover requirements.

Key Takeaways

Redis clusters split data into hash slots distributed across nodes to scale storage and performance.

Adding or removing nodes requires moving hash slots and keys to keep data balanced and available.

Resharding is the process of migrating hash slots between nodes and must be done carefully to avoid downtime.

Cluster state updates use gossip protocol and configuration epochs to maintain consistency during changes.

Handling replicas properly during node removal is critical to prevent data loss and maintain high availability.

Practice

1. What is the primary purpose of adding nodes to a Redis cluster?

easy

A. To increase the cluster's capacity and distribute data

B. To delete all data from the cluster

C. To backup the cluster data

D. To change the Redis version on all nodes

Solution

1. Step 1: Understand what adding nodes means

   Adding nodes means increasing the number of servers in the cluster to share the data load.

2. Step 2: Identify the effect of adding nodes

   More nodes mean more capacity and better data distribution across the cluster.

3. Final Answer:

   To increase the cluster's capacity and distribute data -> Option A

4. Quick Check:

   Adding nodes = More capacity and distribution [OK]

Hint: Adding nodes grows cluster capacity and data spread [OK]

Common Mistakes:

• Thinking adding nodes deletes data
• Confusing adding nodes with backup
• Assuming adding nodes upgrades Redis version

2. Which of the following is the correct syntax to add a node to a Redis cluster using redis-cli?

easy

A. redis-cli --cluster add-node :<port> :<port>

B. redis-cli --cluster remove-node :<port>

C. redis-cli --cluster create-node :<port>

D. redis-cli --cluster add :<port>

Solution

1. Step 1: Recall the command to add a node

   The correct command uses --cluster add-node followed by the new node and an existing node in the cluster.

2. Step 2: Match the syntax

   redis-cli --cluster add-node :<port> :<port> matches the correct syntax: redis-cli --cluster add-node new-node existing-node.

3. Final Answer:

   redis-cli --cluster add-node <new-node-ip>:<port> <existing-node-ip>:<port> -> Option A

4. Quick Check:

   Correct add-node syntax = redis-cli --cluster add-node :<port> :<port> [OK]

Hint: Add node command uses --cluster add-node with new and existing nodes [OK]

Common Mistakes:

• Using remove-node instead of add-node
• Using create-node which does not exist
• Omitting existing node IP in command

3. Given the command redis-cli --cluster del-node 192.168.1.5:7000 07c37dfeb2352e03a5a1e1f3a2f8b9a1f2e3d4c5, what will happen?

medium

A. A new node will be added at 192.168.1.5:7000

B. The node at 192.168.1.5:7000 will be removed from the cluster

C. The cluster will be created with the given node ID

D. The command will fail due to wrong syntax

Solution

1. Step 1: Understand the del-node command

   The command redis-cli --cluster del-node removes a node from the cluster using its IP and node ID.

2. Step 2: Analyze the given command

   The command specifies the node IP and the node ID, so it will remove that node from the cluster.

3. Final Answer:

   The node at 192.168.1.5:7000 will be removed from the cluster -> Option B

4. Quick Check:

   del-node removes specified node [OK]

Hint: del-node removes node by IP and node ID [OK]

Common Mistakes:

• Confusing del-node with add-node
• Thinking it adds a node
• Assuming command syntax is incorrect

4. You run redis-cli --cluster add-node 192.168.1.10:7003 192.168.1.11:7000 but get an error. What is the most likely cause?

medium

A. The new node is already part of the cluster

B. The existing node IP is incorrect

C. The command should be --cluster del-node instead

D. Redis server is not running on the new node

Solution

1. Step 1: Check prerequisites for adding a node

   The new node must be running Redis and reachable for the add-node command to succeed.

2. Step 2: Identify common causes of add-node failure

   If Redis is not running on the new node, the command will fail because it cannot connect.

3. Final Answer:

   Redis server is not running on the new node -> Option D

4. Quick Check:

   New node must run Redis to add [OK]

Hint: New node must be running Redis before adding [OK]

Common Mistakes:

• Assuming IP typo without checking server status
• Confusing add-node with del-node command
• Ignoring that Redis must run on new node

5. You want to remove a node from a Redis cluster but keep its data safe. Which sequence of commands should you use?

hard

A. Use redis-cli --cluster del-node then backup data manually

B. Directly stop the Redis server on the node and remove it

C. Use redis-cli --cluster reshard to move slots, then redis-cli --cluster del-node

D. Use redis-cli --cluster add-node to replace the node

Solution

1. Step 1: Understand data safety when removing nodes

   Removing a node without moving its data causes data loss.

2. Step 2: Use reshard to move data slots

   Resharding moves the data slots from the node to be removed to other nodes safely.

3. Step 3: Remove the node after resharding

   After data is moved, del-node safely removes the node without data loss.

4. Final Answer:

   Use redis-cli --cluster reshard to move slots, then redis-cli --cluster del-node -> Option C

5. Quick Check:

   Reshard then del-node = safe removal [OK]

Hint: Reshard slots before removing node to keep data safe [OK]

Common Mistakes:

• Removing node without moving data first
• Stopping server directly causing data loss
• Confusing add-node with removal process