SCAN for safe key iteration in Redis - Deep Dive

Overview - SCAN for safe key iteration

What is it?

SCAN is a Redis command that lets you safely go through all keys in the database without blocking the server. Unlike commands that return all keys at once, SCAN returns small batches of keys each time you call it. This way, you can iterate over keys even in large databases without causing delays or crashes.

Why it matters

Without SCAN, fetching all keys at once can freeze Redis, making your app slow or unresponsive. SCAN solves this by spreading the work into small steps, so Redis stays fast and responsive. This is important for real-world apps that store lots of data and need to find keys safely.

Where it fits

Before learning SCAN, you should know basic Redis commands and understand what keys and databases are. After SCAN, you can learn about other cursor-based commands like SSCAN, HSCAN, and ZSCAN for iterating sets, hashes, and sorted sets.

Mental Model

Core Idea

SCAN lets you walk through Redis keys step-by-step using a cursor, avoiding heavy work all at once.

Think of it like...

Imagine searching for a book in a huge library by checking a few shelves at a time instead of trying to see every book at once.

┌─────────────┐
│ Start cursor│
└──────┬──────┘
       │ SCAN returns batch of keys + new cursor
       ▼
┌─────────────┐
│ Process keys│
└──────┬──────┘
       │ Use new cursor to continue
       ▼
┌─────────────┐
│ Repeat until│
│ cursor = 0  │
└─────────────┘

Build-Up - 7 Steps

1

Foundation

What is SCAN command in Redis

Concept: Introducing SCAN as a command to iterate keys safely.

In Redis, the SCAN command lets you get keys in small groups instead of all at once. You start with cursor 0, and Redis returns some keys plus a new cursor. You keep calling SCAN with the new cursor until it returns 0, meaning no more keys.

Result

You get keys in batches, avoiding big delays or memory spikes.

Understanding SCAN as a cursor-based iterator helps avoid blocking Redis when listing keys.

2

Foundation

Why not use KEYS command for iteration

Concept: Explaining the problem with KEYS command for large data.

The KEYS command returns all matching keys at once. If the database is big, this can freeze Redis and slow down your app. KEYS is simple but unsafe for production use.

Result

Using KEYS on large databases causes delays and can crash Redis.

Knowing KEYS is unsafe motivates using SCAN for safe iteration.

3

Intermediate

How SCAN cursor works step-by-step

🤔Before reading on: do you think SCAN returns keys in a fixed order or can the order change? Commit to your answer.

Concept: Understanding the cursor mechanism and unordered results.

SCAN returns a cursor and a batch of keys. The cursor is a position marker inside Redis's internal data structure. Each call continues from the last cursor. The order of keys is not guaranteed and can change if keys are added or removed during iteration.

Result

You get partial key lists each time, and must keep calling SCAN until cursor is 0.

Knowing SCAN's unordered and incremental nature helps write code that handles partial and changing results.

4

Intermediate

Using MATCH and COUNT options with SCAN

🤔Before reading on: do you think MATCH filters keys before or after SCAN returns them? Commit to your answer.

Concept: Adding filtering and batch size control to SCAN.

MATCH lets you filter keys by pattern, like 'user:*'. COUNT suggests how many keys to return per call, but Redis may return more or fewer. These options help focus iteration and control workload.

Result

SCAN returns keys matching the pattern in batches roughly the size of COUNT.

Understanding MATCH and COUNT lets you customize SCAN for efficient and targeted key iteration.

5

Intermediate

Handling SCAN results safely in code

🤔Before reading on: do you think you must collect all keys in memory before processing or can you process them as they come? Commit to your answer.

Concept: Best practices for processing SCAN results incrementally.

Because SCAN returns partial results, you should process keys as you receive them instead of storing all keys at once. This keeps memory use low and avoids blocking. Also, be ready for keys to appear or disappear during iteration.

Result

Your code safely iterates keys without freezing or running out of memory.

Knowing to process keys incrementally prevents common memory and performance problems.

6

Advanced

Why SCAN is not a snapshot iterator

🤔Before reading on: do you think SCAN guarantees to return every key exactly once during iteration? Commit to your answer.

Concept: Understanding SCAN's limitations with concurrent changes.

SCAN does not guarantee a perfect snapshot. Keys added or deleted during iteration may be missed or returned multiple times. This is because SCAN scans Redis's internal hash table incrementally without locking the database.

Result

You may see some keys twice or miss some keys if the database changes during SCAN.

Knowing SCAN's eventual consistency helps design robust applications that tolerate duplicates or missing keys.

7

Expert

Internal hashing and cursor mechanics of SCAN

🤔Before reading on: do you think SCAN's cursor is a simple index or something more complex? Commit to your answer.

Concept: How SCAN uses Redis's hash table and bit-reversal cursor internally.

Redis stores keys in a hash table. SCAN's cursor is a 64-bit integer representing a position in this table. It uses a bit-reversal algorithm to visit all hash buckets efficiently. This design avoids scanning the entire table at once and supports incremental iteration.

Result

SCAN efficiently visits all keys without blocking, but order is unpredictable.

Understanding SCAN's internal cursor and hashing explains why iteration order is not fixed and why SCAN is fast and safe.

Under the Hood

SCAN works by iterating over Redis's internal hash table that stores keys. It uses a cursor that points to a position in the hash table. Each SCAN call returns keys from the current cursor position and a new cursor for the next call. The cursor uses a bit-reversal pattern to cover all buckets without scanning the entire table at once. This incremental approach avoids blocking Redis and keeps memory use low.

Why designed this way?

Before SCAN, commands like KEYS blocked Redis by returning all keys at once, causing performance problems. SCAN was designed to allow safe, incremental iteration without locking or blocking. The bit-reversal cursor was chosen to efficiently cover the hash table in small steps. This design balances speed, safety, and simplicity.

┌───────────────┐
│ Redis Hash    │
│ Table buckets │
│ 0 1 2 3 ... N │
└──────┬────────┘
       │ SCAN cursor points here
       ▼
┌─────────────────────────────┐
│ SCAN call:                  │
│ - Reads keys at cursor pos  │
│ - Returns keys + new cursor │
└─────────────┬───────────────┘
              │
              ▼
┌─────────────────────────────┐
│ Next SCAN call uses new cursor│
│ to continue scanning         │
└─────────────────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does SCAN guarantee to return all keys exactly once? Commit to yes or no.

Common Belief: SCAN returns every key exactly once in a fixed order.

Reality: SCAN may return some keys multiple times or miss keys if the database changes during iteration.

Why it matters: Assuming perfect iteration can cause bugs if your code does not handle duplicates or missing keys.

Quick: Is SCAN faster than KEYS for small databases? Commit to yes or no.

Common Belief: SCAN is always faster than KEYS, no matter the database size.

Reality: For very small databases, KEYS can be faster because it returns all keys at once without multiple calls.

Why it matters: Using SCAN unnecessarily on small datasets can add complexity without benefit.

Quick: Does MATCH option in SCAN filter keys before scanning or after? Commit to your answer.

Common Belief: MATCH filters keys before scanning, so SCAN only visits matching keys internally.

Reality: MATCH filters keys after scanning; SCAN still scans all keys internally and then filters results.

Why it matters: Expecting MATCH to reduce internal scanning can lead to wrong performance assumptions.

Quick: Can you rely on SCAN to get a consistent snapshot of keys at one moment? Commit to yes or no.

Common Belief: SCAN provides a consistent snapshot of keys at the time iteration starts.

Reality: SCAN does not provide a snapshot; keys added or removed during iteration may affect results.

Why it matters: Relying on SCAN for consistent snapshots can cause data inconsistency in your application.

Expert Zone

1

SCAN's cursor is a 64-bit integer that encodes the position in the hash table using bit-reversal, which is non-intuitive but critical for efficient iteration.

2

The COUNT option is a hint, not a guarantee; Redis may return more or fewer keys per call depending on internal state.

3

SCAN is safe for production but requires client code to handle duplicates and missing keys gracefully, which many beginners overlook.

When NOT to use

Do not use SCAN when you need a perfect snapshot of keys or guaranteed order. Instead, consider Redis keyspace notifications or maintaining your own index. For small datasets, KEYS is simpler and faster.

Production Patterns

In production, SCAN is used for background tasks like cache cleanup, monitoring, or migrating keys. It is combined with incremental processing and deduplication logic. Often, SCAN is wrapped in scripts or libraries that handle cursor management and retries.

Connections

Cursor-based pagination

SCAN uses a cursor to paginate through keys, similar to how databases paginate query results.

Understanding cursor pagination in databases helps grasp SCAN's incremental iteration and why it avoids loading everything at once.

Hash tables

SCAN iterates over Redis's internal hash table buckets using a bit-reversal cursor.

Knowing how hash tables store data explains why SCAN's order is unpredictable and why it uses a special cursor.

Library book search

Like scanning shelves in small batches to find books, SCAN checks keys in small groups to avoid overload.

This real-world process shows why breaking big tasks into small steps keeps systems responsive.

Common Pitfalls

#1 Trying to get all keys with SCAN but ignoring the cursor and calling SCAN only once.

Wrong approach:

SCAN 0 MATCH * COUNT 100

Correct approach:

cursor = 0
repeat {
result = SCAN cursor MATCH * COUNT 100
cursor = result.cursor
process(result.keys)
} while (cursor != 0)

Root cause:

Misunderstanding that SCAN returns partial results and requires repeated calls with updated cursor.

#2 Assuming SCAN returns keys in a fixed order and relying on that order in application logic.

Wrong approach:

keys = []
cursor = 0
while cursor != 0:
cursor, batch = SCAN cursor
keys.extend(batch)
# Then expecting keys to be sorted or stable

Correct approach:

Process keys as they come without assuming order; if order is needed, sort keys after collection.

Root cause:

Not knowing SCAN's iteration order is unpredictable and can change between calls.

#3 Using MATCH to reduce SCAN's internal workload, expecting it to speed up scanning.

Wrong approach:

SCAN 0 MATCH user:* COUNT 100

Correct approach:

Use MATCH to filter results, but understand SCAN still scans all keys internally; optimize by reducing keyspace size if needed.

Root cause:

Believing MATCH filters keys before scanning rather than after.

Key Takeaways

SCAN is a safe way to iterate Redis keys incrementally without blocking the server.

It uses a cursor to remember position and returns keys in batches until the cursor is zero.

SCAN does not guarantee order or perfect snapshots; keys may appear multiple times or be missed if changed during iteration.

MATCH and COUNT options help filter and control batch size but have limitations.

Proper use of SCAN requires handling partial results, duplicates, and incremental processing in your code.

Practice

1. What is the main advantage of using SCAN over KEYS in Redis for iterating keys?

easy

A. SCAN returns all keys at once for faster access.

B. SCAN only works with sorted sets.

C. SCAN deletes keys while iterating.

D. SCAN does not block Redis and returns keys in batches.

Solution

1. Step 1: Understand KEYS behavior

   KEYS returns all matching keys at once and blocks Redis during execution.

2. Step 2: Understand SCAN behavior

   SCAN returns keys in small batches without blocking Redis, allowing safe iteration.

3. Final Answer:

   SCAN does not block Redis and returns keys in batches. -> Option D

4. Quick Check:

   Safe iteration = SCAN non-blocking [OK]

Hint: Remember: SCAN is non-blocking, KEYS blocks [OK]

Common Mistakes:

• Thinking SCAN returns all keys at once
• Confusing SCAN with key deletion
• Assuming SCAN only works on sorted sets

2. Which of the following is the correct syntax to start a SCAN iteration from the beginning in Redis CLI?

easy

A. SCAN 1

B. SCAN 0

C. SCAN *

D. SCAN start

Solution

1. Step 1: Identify the starting cursor for SCAN

   The SCAN command requires a cursor argument; starting cursor is always 0.

2. Step 2: Check options for correct cursor

   SCAN 0 starts iteration from the beginning; other options are invalid.

3. Final Answer:

   SCAN 0 -> Option B

4. Quick Check:

   Start cursor = 0 [OK]

Hint: Always start SCAN with cursor 0 [OK]

Common Mistakes:

• Starting with cursor 1 instead of 0
• Using wildcard * as cursor
• Using invalid strings like 'start'

3. Given the following Redis commands executed sequentially, what will be the output of the last SCAN command?

SET user:1 Alice
SET user:2 Bob
SET admin:1 Carol
SCAN 0 MATCH user:*

medium

A. [cursor, ["user:1", "user:2"]]

B. [0, ["user:1", "user:2"]]

C. [cursor, ["user:1", "user:2", "admin:1"]]

D. [0, ["admin:1"]]

Solution

1. Step 1: Understand SCAN with MATCH

   The MATCH user:* filters keys starting with 'user:'. So keys 'user:1' and 'user:2' match; 'admin:1' does not.

2. Step 2: Understand SCAN output format

   SCAN returns a two-element array: a cursor (string) and a list of keys matching the pattern in this batch. Cursor is not necessarily 0 unless iteration is complete.

3. Final Answer:

   [cursor, ["user:1", "user:2"]] -> Option A

4. Quick Check:

   Filtered keys = user keys only [OK]

Hint: MATCH filters keys; cursor is a string, not always 0 [OK]

Common Mistakes:

• Expecting cursor 0 on first SCAN call
• Including keys not matching pattern
• Confusing cursor with count

4. You wrote this Redis command to iterate keys safely:

SCAN 0 MATCH user:*

But you notice it returns only a partial list of keys matching the pattern. What is the likely mistake?

medium

A. Not repeating SCAN with the returned cursor until it returns 0.

B. Using MATCH with an incorrect pattern.

C. Starting SCAN with cursor 1 instead of 0.

D. Using COUNT option with a value too high.

Solution

1. Step 1: Understand SCAN iteration process

   SCAN returns a cursor and a batch of keys; to get all keys, you must repeat SCAN with the new cursor until it returns 0.

2. Step 2: Identify the mistake

   Running SCAN 0 MATCH user:* only once returns partial keys; not looping causes incomplete results.

3. Final Answer:

   Not repeating SCAN with the returned cursor until it returns 0. -> Option A

4. Quick Check:

   Repeat SCAN until cursor 0 [OK]

Hint: Always loop SCAN calls until cursor returns 0 [OK]

Common Mistakes:

• Expecting all keys in one SCAN call
• Misusing MATCH pattern syntax
• Starting with wrong cursor

5. You want to safely iterate all keys starting with session: in Redis but avoid blocking the server. You also want to limit each batch to about 10 keys. Which command sequence correctly achieves this?

hard

A. Use SCAN 0 COUNT 10 without MATCH.

B. Run KEYS session:* once to get all keys.

C. Repeat SCAN cursor MATCH session:* COUNT 10 until cursor returns 0.

D. Use SCAN 0 MATCH session:* once without COUNT.

Solution

1. Step 1: Use SCAN with MATCH and COUNT

   To filter keys starting with 'session:' and limit batch size, use MATCH session:* and COUNT 10.

2. Step 2: Repeat SCAN calls until cursor is 0

   Iterate by passing the returned cursor each time until it returns 0 to get all keys safely without blocking.

3. Final Answer:

   Repeat SCAN cursor MATCH session:* COUNT 10 until cursor returns 0. -> Option C

4. Quick Check:

   Safe iteration + filter + batch size = Repeat SCAN cursor MATCH session:* COUNT 10 until cursor returns 0. [OK]

Hint: Combine MATCH and COUNT, loop until cursor 0 [OK]

Common Mistakes:

• Using KEYS which blocks Redis
• Not looping SCAN calls
• Omitting MATCH or COUNT when needed