Hash slots distribution in Redis - Deep Dive

Overview - Hash slots distribution

What is it?

Hash slots distribution is a method Redis uses to split data across multiple servers in a cluster. It divides all possible keys into 16,384 slots, and each server in the cluster is responsible for a subset of these slots. This way, Redis can spread data evenly and handle more requests by sharing the load.

Why it matters

Without hash slots distribution, Redis would store all data on a single server, limiting speed and capacity. This method allows Redis to scale horizontally, meaning it can grow by adding more servers. It also helps keep data balanced and makes the system more reliable by distributing the workload.

Where it fits

Before learning hash slots distribution, you should understand basic Redis concepts like keys, values, and clustering. After this, you can explore how Redis handles failover, replication, and advanced cluster management.

Mental Model

Core Idea

Hash slots distribution assigns each key to one of 16,384 fixed slots, and each server in a Redis cluster manages a range of these slots to evenly spread data and requests.

Think of it like...

Imagine a large library with 16,384 numbered shelves (slots). Each librarian (server) is responsible for a group of shelves. When a new book (key) arrives, it goes to the shelf number determined by a simple rule, so librarians know exactly where to find or store it.

┌─────────────────────────────┐
│ Redis Cluster with Servers  │
├─────────────┬───────────────┤
│ Server A    │ Slots 0-5460  │
│ Server B    │ Slots 5461-10922 │
│ Server C    │ Slots 10923-16383 │
└─────────────┴───────────────┘

Keys hashed → slot number → stored on server managing that slot range

Build-Up - 7 Steps

1

Foundation

Understanding Redis Keys and Hashing

Concept: Learn what Redis keys are and how hashing maps keys to numbers.

Redis stores data as key-value pairs. To distribute keys, Redis uses a hash function that converts each key into a number between 0 and 16,383. This number is called the hash slot. The hash function ensures that the same key always maps to the same slot.

Result

Every key has a unique slot number between 0 and 16,383.

Understanding that keys map to fixed slots is the foundation for how Redis spreads data across servers.

2

Foundation

What Are Hash Slots in Redis Cluster

Concept: Introduce the concept of hash slots as fixed buckets for keys.

Redis divides all possible keys into 16,384 hash slots. Each slot is like a bucket that can hold many keys. The cluster assigns these slots to different servers, so each server manages a subset of slots and the keys inside them.

Result

The entire key space is split into 16,384 slots ready to be distributed.

Knowing the fixed number of slots helps understand how Redis balances data and scales.

3

Intermediate

Assigning Slots to Servers in a Cluster

Concept: Learn how Redis assigns ranges of slots to different servers.

In a Redis cluster, each server is responsible for one or more continuous ranges of hash slots. For example, one server might handle slots 0 to 5460, another 5461 to 10922, and so on. This division allows the cluster to spread data evenly and handle more requests.

Result

Each server knows exactly which slots it owns and which keys it should store.

Understanding slot ranges per server clarifies how Redis achieves load balancing and fault tolerance.

4

Intermediate

How Redis Routes Commands Using Slots

🤔Before reading on: do you think Redis sends commands to all servers or just one? Commit to your answer.

Concept: Discover how Redis uses hash slots to route commands to the correct server.

When a client sends a command with a key, Redis calculates the key's hash slot. It then sends the command only to the server responsible for that slot. This avoids unnecessary communication and speeds up operations.

Result

Commands reach only the server that holds the relevant data, improving efficiency.

Knowing that commands route by slot explains how Redis keeps cluster operations fast and simple.

5

Intermediate

Handling Slot Migration During Resharding

🤔Before reading on: do you think slots move instantly or gradually between servers? Commit to your answer.

Concept: Learn how Redis moves slots between servers when scaling or rebalancing.

When adding or removing servers, Redis moves hash slots from one server to another. This process is called resharding. Redis migrates keys in the slot gradually to avoid downtime, updating the cluster state so clients know the new slot owners.

Result

Slots and their keys move smoothly between servers without stopping the cluster.

Understanding gradual slot migration reveals how Redis maintains availability during scaling.

6

Advanced

Why 16,384 Slots? The Design Choice

🤔Before reading on: do you think the number of slots is arbitrary or carefully chosen? Commit to your answer.

Concept: Explore why Redis fixed the number of hash slots at 16,384.

Redis chose 16,384 slots as a balance between granularity and performance. More slots mean finer distribution but more overhead in managing slot assignments. Fewer slots reduce overhead but limit flexibility. 16,384 is a power of two, which simplifies hashing and slot calculation.

Result

Redis achieves efficient, balanced distribution with manageable cluster metadata.

Knowing the slot count rationale helps appreciate Redis's balance between speed and scalability.

7

Expert

Surprises in Slot Distribution and Client Behavior

🤔Before reading on: do you think clients always know the correct slot mapping instantly? Commit to your answer.

Concept: Understand subtle behaviors in slot distribution and client interaction.

Clients cache slot-to-server mappings to reduce cluster queries. However, during slot migration or failover, mappings can become outdated, causing MOVED or ASK errors. Clients must handle these errors by updating their cache. Also, keys with the same hash tag (part of the key in {}) always map to the same slot, enabling atomic operations on related keys.

Result

Clients efficiently interact with the cluster but must handle dynamic slot changes gracefully.

Recognizing client caching and error handling nuances prevents common cluster communication bugs.

Under the Hood

Redis uses a CRC16 hash function on the key (or the substring inside curly braces if present) to compute a number between 0 and 16,383. This number is the hash slot. Each cluster node maintains a mapping of which slots it owns. When a command arrives, Redis routes it to the node owning the slot. During resharding, Redis migrates keys slot by slot, updating cluster state and informing clients of changes via special error replies.

Why designed this way?

Redis needed a simple, fast way to distribute keys evenly and allow cluster scaling without complex coordination. Fixing the number of slots simplifies cluster metadata and slot assignment. Using a hash function ensures uniform key distribution. The design balances performance, scalability, and operational simplicity, avoiding the overhead of dynamic slot counts or complex consistent hashing.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│   Client      │──────▶│  Redis Node A │──────▶│ Slots 0-5460  │
└───────────────┘       └───────────────┘       └───────────────┘
                             │
                             │
                             ▼
                      ┌───────────────┐
                      │  Redis Node B │
                      │ Slots 5461-10922│
                      └───────────────┘
                             │
                             ▼
                      ┌───────────────┐
                      │  Redis Node C │
                      │ Slots 10923-16383│
                      └───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Do you think Redis can assign keys to any number of slots dynamically? Commit yes or no.

Common Belief: Redis can assign keys to any number of slots dynamically based on cluster size.

Reality: Redis uses a fixed number of 16,384 hash slots regardless of cluster size.

Why it matters: Believing slots are dynamic can lead to confusion about cluster scaling and slot management, causing misconfiguration.

Quick: Do you think all keys with the same prefix always go to the same slot? Commit yes or no.

Common Belief: Keys with the same prefix always map to the same hash slot.

Reality: Only keys with the same hash tag (the substring inside {}) map to the same slot; prefixes alone do not affect slot assignment.

Why it matters: Assuming prefixes control slot placement can cause unexpected key distribution and break multi-key operations.

Quick: Do you think clients always know the correct slot mapping instantly? Commit yes or no.

Common Belief: Clients always have up-to-date slot-to-node mappings and never need to retry commands.

Reality: Clients cache slot mappings but must handle MOVED or ASK errors when slots migrate or cluster topology changes.

Why it matters: Ignoring this leads to failed commands and poor cluster reliability.

Quick: Do you think slot migration happens instantly without affecting availability? Commit yes or no.

Common Belief: Slot migration between nodes happens instantly without any impact on cluster availability.

Reality: Slot migration is gradual to avoid downtime, and clients may temporarily receive errors during migration.

Why it matters: Expecting instant migration can cause surprises during scaling or failover.

Expert Zone

1

Redis clients use cached slot maps to reduce cluster queries but must implement error handling for MOVED and ASK replies to maintain correctness.

2

Hash tags allow grouping related keys into the same slot, enabling atomic multi-key operations within a cluster, a subtle but powerful feature.

3

The fixed slot count simplifies cluster metadata but limits the maximum number of nodes to 16,384, influencing cluster design decisions.

When NOT to use

Hash slots distribution is not suitable for workloads requiring complex multi-key transactions across many keys in different slots. In such cases, consider using Redis in single-node mode or external sharding solutions that support cross-shard transactions.

Production Patterns

In production, Redis clusters use hash slots to balance load and enable horizontal scaling. Operators monitor slot distribution to avoid hotspots and perform resharding during scaling. Clients implement retry logic for MOVED and ASK errors to handle topology changes smoothly.

Connections

Consistent Hashing

Both distribute keys across servers but Redis uses fixed slots while consistent hashing uses a ring structure.

Understanding consistent hashing helps appreciate Redis's simpler fixed-slot approach and its tradeoffs in flexibility versus performance.

Load Balancing in Web Servers

Hash slots distribution is like load balancing requests to servers based on a hash of the request data.

Knowing load balancing concepts clarifies how Redis spreads data and requests evenly to avoid overload.

Library Book Shelving Systems

Hash slots distribution is similar to assigning books to shelves by a numbering system for easy retrieval.

This connection shows how organizing data into fixed buckets simplifies finding and storing items efficiently.

Common Pitfalls

#1 Trying to store related keys without hash tags expecting them to be in the same slot.

Wrong approach:

SET user:1:name "Alice"
SET user:1:email "[e-mail]"

Correct approach:

SET user:{1}:name "Alice"
SET user:{1}:email "[e-mail]"

Root cause:

Misunderstanding that only keys with the same substring inside {} share the same slot.

#2 Ignoring MOVED errors and not updating client slot cache.

Wrong approach:

Client retries commands blindly without handling MOVED replies.

Correct approach:

Client catches MOVED errors, updates slot-to-node mapping, and retries commands accordingly.

Root cause:

Not realizing cluster topology changes require client awareness and cache updates.

#3 Assuming slot migration is instant and causes no temporary errors.

Wrong approach:

Performing resharding without preparing clients for temporary ASK errors.

Correct approach:

Clients handle ASK errors during migration by redirecting commands to the correct node temporarily.

Root cause:

Overlooking the gradual nature of slot migration and its impact on client commands.

Key Takeaways

Redis divides all keys into 16,384 fixed hash slots to distribute data evenly across cluster nodes.

Each server in a Redis cluster manages a continuous range of slots, enabling load balancing and scalability.

Clients route commands to the correct server by computing the key's hash slot, improving efficiency.

Hash tags allow grouping related keys into the same slot for atomic multi-key operations.

Clients must handle cluster changes by updating slot mappings and managing errors during slot migration.