WATCH for optimistic locking in Redis - Deep Dive

Overview - WATCH for optimistic locking

What is it?

WATCH is a Redis command used to implement optimistic locking, a way to safely handle changes to data when multiple clients might try to update it at the same time. It monitors one or more keys for changes before a transaction runs. If any watched key changes, the transaction is aborted to avoid conflicts. This helps keep data consistent without locking resources for long.

Why it matters

Without optimistic locking, multiple clients could overwrite each other's changes, causing data loss or corruption. WATCH helps prevent this by detecting conflicts early and stopping unsafe updates. This is important in real-time apps like chat, games, or financial systems where many users update shared data simultaneously.

Where it fits

Before learning WATCH, you should understand basic Redis commands and transactions (MULTI/EXEC). After WATCH, you can explore more advanced concurrency controls and Lua scripting in Redis for atomic operations.

Mental Model

Core Idea

WATCH lets Redis watch keys for changes and cancels a transaction if those keys are modified, ensuring safe concurrent updates without locking.

Think of it like...

Imagine you put a sticky note on a shared whiteboard section saying 'I'm about to write here.' If someone erases or changes that section before you finish, you stop and try again later to avoid messing up their work.

┌───────────────┐
│ Client calls  │
│ WATCH key(s)  │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Redis monitors │
│ key changes   │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Client starts │
│ MULTI/EXEC    │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ If keys changed│
│ → transaction │
│   aborts      │
│ Else → commit │
└───────────────┘

Build-Up - 7 Steps

1

Foundation

Basic Redis Transactions

Concept: Introduce Redis transactions using MULTI and EXEC commands.

Redis transactions group multiple commands to run sequentially without interruption. You start with MULTI, queue commands, then run EXEC to execute all at once. However, transactions don't check for data changes by default.

Result

Commands inside MULTI/EXEC run atomically but without conflict detection.

Understanding basic transactions is key because WATCH builds on this to add safety when multiple clients update data.

2

Foundation

Concurrent Data Update Problem

Concept: Explain the problem of multiple clients updating the same key simultaneously.

If two clients read a key, modify it, and write back without coordination, one client's update can overwrite the other's. This causes lost updates and inconsistent data.

Result

Data can become incorrect or outdated when clients don't coordinate updates.

Recognizing this problem shows why we need a mechanism like WATCH to detect conflicts.

3

Intermediate

Using WATCH to Monitor Keys

🤔Before reading on: do you think WATCH locks the key to prevent changes, or just watches for changes? Commit to your answer.

Concept: Introduce WATCH command that monitors keys for changes without locking them.

WATCH marks keys to be watched. If any watched key changes before EXEC, the transaction aborts. This is optimistic locking because it assumes no conflict but checks before committing.

Result

If keys change, EXEC returns null and transaction is discarded; otherwise, it commits.

Knowing WATCH doesn't lock keys but detects changes helps understand how Redis supports high concurrency without blocking.

4

Intermediate

Typical WATCH Transaction Flow

🤔Before reading on: do you think commands inside MULTI after WATCH run immediately or only after EXEC? Commit to your answer.

Concept: Explain the sequence of WATCH, MULTI, commands, and EXEC in a transaction.

Client calls WATCH on keys, then MULTI to start queuing commands. Commands are queued but not executed until EXEC. If any watched key changed, EXEC aborts the transaction.

Result

Commands execute atomically only if watched keys remain unchanged.

Understanding this flow clarifies how Redis ensures safe updates without locking during command queuing.

5

Intermediate

Handling Transaction Aborts

🤔Before reading on: do you think Redis automatically retries aborted transactions? Commit to your answer.

Concept: Describe what happens when a transaction aborts due to key changes and how clients should handle it.

If EXEC returns null, the transaction was aborted because a watched key changed. Clients must retry the whole process: WATCH, read data, MULTI, queue commands, EXEC again.

Result

Clients detect conflicts and retry updates to ensure data consistency.

Knowing that clients must retry aborted transactions prevents silent data loss and ensures robust concurrency handling.

6

Advanced

WATCH with Multiple Keys and Performance

🤔Before reading on: do you think watching many keys impacts Redis performance significantly? Commit to your answer.

Concept: Explain how WATCH works with multiple keys and its performance implications.

WATCH can monitor multiple keys at once. Redis tracks changes efficiently, but watching many keys increases overhead. Use WATCH on only necessary keys to balance safety and performance.

Result

Safe concurrency with minimal performance cost when used wisely.

Understanding the tradeoff helps design efficient Redis transactions in real applications.

7

Expert

WATCH Internals and Edge Cases

🤔Before reading on: do you think WATCH detects changes only from other clients or also from the same client? Commit to your answer.

Concept: Reveal how Redis internally tracks watched keys and subtle behaviors like same-client changes and transaction aborts.

Redis tracks watched keys per connection. If any watched key changes by any client, including the same one outside the transaction, the transaction aborts. Also, commands like DISCARD or EXEC clear watched keys. WATCH is connection-specific and resets after EXEC or DISCARD.

Result

Transactions abort precisely on any conflicting change, ensuring strict consistency.

Knowing these internals prevents subtle bugs and helps write correct, efficient Redis code using WATCH.

Under the Hood

WATCH sets a watch flag on specified keys for the client's connection. Redis keeps track of the version or modification count of these keys. When EXEC is called, Redis compares the current version of each watched key to the version at WATCH time. If any key changed, EXEC returns null and discards the transaction. This check is done atomically to prevent race conditions.

Why designed this way?

WATCH was designed to provide optimistic locking without blocking other clients. Blocking locks reduce concurrency and performance. By watching keys and aborting on conflicts, Redis allows many clients to work concurrently with minimal overhead. Alternatives like pessimistic locking were rejected because they hurt Redis's speed and simplicity.

Client Connection
  │
  ├─ WATCH keys → Redis marks keys with version snapshot
  │
  ├─ MULTI → commands queued
  │
  ├─ EXEC → Redis compares current key versions with snapshot
  │       ├─ If unchanged → execute queued commands atomically
  │       └─ If changed → abort transaction, return null
  │
  └─ DISCARD or EXEC clears watched keys

Myth Busters - 4 Common Misconceptions

Quick: Does WATCH lock keys to prevent other clients from changing them? Commit yes or no.

Common Belief: WATCH locks keys so no other client can modify them during the transaction.

Reality: WATCH does not lock keys; it only monitors them for changes and aborts the transaction if changes occur.

Why it matters: Believing WATCH locks keys leads to incorrect assumptions about concurrency and can cause inefficient designs or bugs.

Quick: If a watched key changes by the same client before EXEC, does the transaction abort? Commit yes or no.

Common Belief: Only changes from other clients cause transaction aborts.

Reality: Any change to a watched key, even by the same client outside the transaction, causes the transaction to abort.

Why it matters: Ignoring this can cause unexpected transaction failures and confusion during debugging.

Quick: Does Redis automatically retry aborted transactions when a watched key changes? Commit yes or no.

Common Belief: Redis retries transactions automatically after aborts caused by watched key changes.

Reality: Redis does not retry automatically; clients must detect aborts and retry manually.

Why it matters: Assuming automatic retries can cause lost updates or inconsistent data if clients don't handle aborts properly.

Quick: Does WATCH guarantee no conflicts if you watch only one key in a multi-key update? Commit yes or no.

Common Belief: Watching one key is enough to prevent conflicts in multi-key updates.

Reality: WATCH only monitors specified keys; changes to un-watched keys can still cause conflicts.

Why it matters: Partial watching can lead to subtle data inconsistencies if all relevant keys are not watched.

Expert Zone

1

WATCH is connection-specific; multiple connections watching the same keys do not interfere with each other.

2

WATCH resets after EXEC or DISCARD, so clients must re-watch keys for each transaction attempt.

3

Using WATCH with Lua scripts can be tricky because scripts run atomically and bypass WATCH checks.

When NOT to use

WATCH is not suitable when you need guaranteed locking or blocking behavior; in such cases, use Redis locks (SETNX with expiration) or Lua scripts for atomic operations.

Production Patterns

In production, WATCH is often combined with retry loops in clients to handle transaction aborts gracefully. It's used in counters, inventory systems, and session management where concurrent updates are common but conflicts are rare.

Connections

Optimistic Concurrency Control (OCC)

WATCH implements OCC pattern in Redis.

Understanding WATCH helps grasp OCC, a common concurrency control method in databases and distributed systems.

Database Transactions

WATCH extends Redis transactions with conflict detection.

Knowing how WATCH works deepens understanding of transaction isolation and consistency in databases.

Version Control Systems

WATCH's conflict detection is similar to how version control detects changes before merging.

Recognizing this connection helps appreciate conflict detection as a universal pattern beyond databases.

Common Pitfalls

#1 Assuming WATCH locks keys and prevents other clients from changing them.

Wrong approach:

WATCH mykey
MULTI
SET mykey newvalue
EXEC

Correct approach:

WATCH mykey
GET mykey
MULTI
SET mykey newvalue
EXEC
if EXEC returns null then retry

Root cause:

Misunderstanding that WATCH only monitors changes but does not block other clients.

#2 Not retrying the transaction after EXEC returns null due to key changes.

Wrong approach:

WATCH mykey
MULTI
INCR mykey
EXEC
// no retry on null result

Correct approach:

do {
WATCH mykey
val = GET mykey
MULTI
INCR mykey
result = EXEC
} while (result == null)

Root cause:

Ignoring that transactions can abort and must be retried to ensure success.

#3 Watching only some keys involved in a multi-key update.

Wrong approach:

WATCH key1
MULTI
SET key1 val1
SET key2 val2
EXEC

Correct approach:

WATCH key1 key2
MULTI
SET key1 val1
SET key2 val2
EXEC

Root cause:

Not watching all keys that can cause conflicts leads to undetected data races.

Key Takeaways

WATCH in Redis provides optimistic locking by monitoring keys for changes before committing transactions.

It does not lock keys but aborts transactions if watched keys change, allowing safe concurrent updates.

Clients must handle transaction aborts by retrying the entire WATCH-MULTI-EXEC sequence.

WATCH is connection-specific and resets after each transaction attempt.

Using WATCH effectively requires watching all relevant keys and understanding its limitations compared to locks or atomic scripts.

Practice

1. What is the main purpose of the WATCH command in Redis optimistic locking?

easy

A. To monitor keys for changes before executing a transaction

B. To lock keys permanently for exclusive access

C. To delete keys automatically after a timeout

D. To create backups of keys before modification

Solution

1. Step 1: Understand the role of WATCH in Redis

   WATCH is used to monitor keys for any changes before a transaction starts.

2. Step 2: Identify what happens if keys change

   If any watched key changes before EXEC, the transaction is canceled to avoid conflicts.

3. Final Answer:

   To monitor keys for changes before executing a transaction -> Option A

4. Quick Check:

   WATCH detects key changes before transaction [OK]

Hint: WATCH checks keys before transaction to avoid conflicts [OK]

Common Mistakes:

• Thinking WATCH locks keys permanently
• Assuming WATCH deletes keys
• Believing WATCH creates backups

2. Which of the following is the correct sequence of Redis commands to safely update a key counter using optimistic locking?

easy

A. MULTI, WATCH counter, INCR counter, EXEC

B. WATCH counter, MULTI, INCR counter, EXEC

C. WATCH counter, INCR counter, MULTI, EXEC

D. INCR counter, WATCH counter, MULTI, EXEC

Solution

1. Step 1: Identify correct order for optimistic locking

   You must first WATCH the key to monitor changes, then start MULTI for transaction.

2. Step 2: Confirm command order

   The correct order is WATCH counter, then MULTI, then commands like INCR, finally EXEC.

3. Final Answer:

   WATCH counter, MULTI, INCR counter, EXEC -> Option B

4. Quick Check:

   WATCH before MULTI for optimistic locking [OK]

Hint: Always WATCH before MULTI to start transaction safely [OK]

Common Mistakes:

• Starting MULTI before WATCH
• Running commands before MULTI
• Not using EXEC at the end

3. Consider this Redis transaction sequence:

WATCH balance
MULTI
INCR balance
EXEC

If another client modifies balance after WATCH but before EXEC, what will be the result of EXEC?

medium

A. EXEC will fail and return null

B. EXEC will succeed and increment balance

C. EXEC will throw a syntax error

D. EXEC will retry automatically

Solution

1. Step 1: Understand WATCH behavior on key change

   If a watched key changes before EXEC, the transaction is aborted to prevent conflicts.

2. Step 2: Identify EXEC result when aborting

   When aborted, EXEC returns null (or nil) indicating the transaction did not run.

3. Final Answer:

   EXEC will fail and return null -> Option A

4. Quick Check:

   Key changed after WATCH causes EXEC to return null [OK]

Hint: If watched key changes, EXEC returns null, no update done [OK]

Common Mistakes:

• Assuming EXEC still increments key
• Expecting syntax error on EXEC
• Thinking EXEC retries automatically

4. You wrote this Redis code:

WATCH stock
MULTI
DECR stock
EXEC

But sometimes the stock count is not decremented even though no errors occur. What is the most likely cause?

medium

A. EXEC command was missing

B. The DECR command syntax is incorrect

C. The stock key was modified by another client after WATCH

D. WATCH does not support DECR command

Solution

1. Step 1: Analyze why stock is not decremented

   If another client changes the watched key after WATCH, EXEC aborts the transaction silently.

2. Step 2: Confirm command correctness

   DECR syntax is correct and EXEC is present, so no syntax errors.

3. Final Answer:

   The stock key was modified by another client after WATCH -> Option C

4. Quick Check:

   Concurrent key change aborts transaction silently [OK]

Hint: Concurrent changes abort EXEC silently, no decrement done [OK]

Common Mistakes:

• Blaming DECR syntax
• Forgetting EXEC command
• Thinking WATCH blocks commands

5. You want to safely increment a shared counter only if it is less than 10 using Redis optimistic locking. Which code snippet correctly implements this logic?

hard

A. WATCH counter
if val < 10:
val = GET counter
MULTI
INCR counter
EXEC

B. MULTI
WATCH counter
val = GET counter
if val < 10:
INCR counter
EXEC

C. WATCH counter
MULTI
val = GET counter
if val < 10:
INCR counter
EXEC

D. WATCH counter
val = GET counter
MULTI
if val < 10:
INCR counter
EXEC

Solution

1. Step 1: Understand optimistic locking with conditional logic

   You must WATCH the key, then GET its value before starting MULTI to ensure value is current.

2. Step 2: Confirm correct command order

   WATCH first, then GET value, then MULTI, then conditionally INCR, finally EXEC.

3. Final Answer:

   WATCH counter val = GET counter MULTI if val < 10: INCR counter EXEC -> Option D

4. Quick Check:

   GET before MULTI after WATCH ensures correct conditional update [OK]

Hint: GET value after WATCH but before MULTI for safe conditional update [OK]

Common Mistakes:

• Starting MULTI before GET
• Getting value after MULTI
• Not watching key before GET