WATCH for optimistic locking in Redis - Deep Dive

Overview - WATCH for optimistic locking

What is it?

WATCH is a Redis command used to implement optimistic locking, a way to safely handle changes to data when multiple clients might try to update it at the same time. It monitors one or more keys for changes before a transaction runs. If any watched key changes, the transaction is aborted to avoid conflicts. This helps keep data consistent without locking resources for long.

Why it matters

Without optimistic locking, multiple clients could overwrite each other's changes, causing data loss or corruption. WATCH helps prevent this by detecting conflicts early and stopping unsafe updates. This is important in real-time apps like chat, games, or financial systems where many users update shared data simultaneously.

Where it fits

Before learning WATCH, you should understand basic Redis commands and transactions (MULTI/EXEC). After WATCH, you can explore more advanced concurrency controls and Lua scripting in Redis for atomic operations.

Mental Model

Core Idea

WATCH lets Redis watch keys for changes and cancels a transaction if those keys are modified, ensuring safe concurrent updates without locking.

Think of it like...

Imagine you put a sticky note on a shared whiteboard section saying 'I'm about to write here.' If someone erases or changes that section before you finish, you stop and try again later to avoid messing up their work.

┌───────────────┐
│ Client calls  │
│ WATCH key(s)  │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Redis monitors │
│ key changes   │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Client starts │
│ MULTI/EXEC    │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ If keys changed│
│ → transaction │
│   aborts      │
│ Else → commit │
└───────────────┘

Build-Up - 7 Steps

1

Foundation

Basic Redis Transactions

Concept: Introduce Redis transactions using MULTI and EXEC commands.

Redis transactions group multiple commands to run sequentially without interruption. You start with MULTI, queue commands, then run EXEC to execute all at once. However, transactions don't check for data changes by default.

Result

Commands inside MULTI/EXEC run atomically but without conflict detection.

Understanding basic transactions is key because WATCH builds on this to add safety when multiple clients update data.

2

Foundation

Concurrent Data Update Problem

Concept: Explain the problem of multiple clients updating the same key simultaneously.

If two clients read a key, modify it, and write back without coordination, one client's update can overwrite the other's. This causes lost updates and inconsistent data.

Result

Data can become incorrect or outdated when clients don't coordinate updates.

Recognizing this problem shows why we need a mechanism like WATCH to detect conflicts.

3

Intermediate

Using WATCH to Monitor Keys

🤔Before reading on: do you think WATCH locks the key to prevent changes, or just watches for changes? Commit to your answer.

Concept: Introduce WATCH command that monitors keys for changes without locking them.

WATCH marks keys to be watched. If any watched key changes before EXEC, the transaction aborts. This is optimistic locking because it assumes no conflict but checks before committing.

Result

If keys change, EXEC returns null and transaction is discarded; otherwise, it commits.

Knowing WATCH doesn't lock keys but detects changes helps understand how Redis supports high concurrency without blocking.

4

Intermediate

Typical WATCH Transaction Flow

🤔Before reading on: do you think commands inside MULTI after WATCH run immediately or only after EXEC? Commit to your answer.

Concept: Explain the sequence of WATCH, MULTI, commands, and EXEC in a transaction.

Client calls WATCH on keys, then MULTI to start queuing commands. Commands are queued but not executed until EXEC. If any watched key changed, EXEC aborts the transaction.

Result

Commands execute atomically only if watched keys remain unchanged.

Understanding this flow clarifies how Redis ensures safe updates without locking during command queuing.

5

Intermediate

Handling Transaction Aborts

🤔Before reading on: do you think Redis automatically retries aborted transactions? Commit to your answer.

Concept: Describe what happens when a transaction aborts due to key changes and how clients should handle it.

If EXEC returns null, the transaction was aborted because a watched key changed. Clients must retry the whole process: WATCH, read data, MULTI, queue commands, EXEC again.

Result

Clients detect conflicts and retry updates to ensure data consistency.

Knowing that clients must retry aborted transactions prevents silent data loss and ensures robust concurrency handling.

6

Advanced

WATCH with Multiple Keys and Performance

🤔Before reading on: do you think watching many keys impacts Redis performance significantly? Commit to your answer.

Concept: Explain how WATCH works with multiple keys and its performance implications.

WATCH can monitor multiple keys at once. Redis tracks changes efficiently, but watching many keys increases overhead. Use WATCH on only necessary keys to balance safety and performance.

Result

Safe concurrency with minimal performance cost when used wisely.

Understanding the tradeoff helps design efficient Redis transactions in real applications.

7

Expert

WATCH Internals and Edge Cases

🤔Before reading on: do you think WATCH detects changes only from other clients or also from the same client? Commit to your answer.

Concept: Reveal how Redis internally tracks watched keys and subtle behaviors like same-client changes and transaction aborts.

Redis tracks watched keys per connection. If any watched key changes by any client, including the same one outside the transaction, the transaction aborts. Also, commands like DISCARD or EXEC clear watched keys. WATCH is connection-specific and resets after EXEC or DISCARD.

Result

Transactions abort precisely on any conflicting change, ensuring strict consistency.

Knowing these internals prevents subtle bugs and helps write correct, efficient Redis code using WATCH.

Under the Hood

WATCH sets a watch flag on specified keys for the client's connection. Redis keeps track of the version or modification count of these keys. When EXEC is called, Redis compares the current version of each watched key to the version at WATCH time. If any key changed, EXEC returns null and discards the transaction. This check is done atomically to prevent race conditions.

Why designed this way?

WATCH was designed to provide optimistic locking without blocking other clients. Blocking locks reduce concurrency and performance. By watching keys and aborting on conflicts, Redis allows many clients to work concurrently with minimal overhead. Alternatives like pessimistic locking were rejected because they hurt Redis's speed and simplicity.

Client Connection
  │
  ├─ WATCH keys → Redis marks keys with version snapshot
  │
  ├─ MULTI → commands queued
  │
  ├─ EXEC → Redis compares current key versions with snapshot
  │       ├─ If unchanged → execute queued commands atomically
  │       └─ If changed → abort transaction, return null
  │
  └─ DISCARD or EXEC clears watched keys

Myth Busters - 4 Common Misconceptions

Quick: Does WATCH lock keys to prevent other clients from changing them? Commit yes or no.

Common Belief: WATCH locks keys so no other client can modify them during the transaction.

Reality: WATCH does not lock keys; it only monitors them for changes and aborts the transaction if changes occur.

Why it matters: Believing WATCH locks keys leads to incorrect assumptions about concurrency and can cause inefficient designs or bugs.

Quick: If a watched key changes by the same client before EXEC, does the transaction abort? Commit yes or no.

Common Belief: Only changes from other clients cause transaction aborts.

Reality: Any change to a watched key, even by the same client outside the transaction, causes the transaction to abort.

Why it matters: Ignoring this can cause unexpected transaction failures and confusion during debugging.

Quick: Does Redis automatically retry aborted transactions when a watched key changes? Commit yes or no.

Common Belief: Redis retries transactions automatically after aborts caused by watched key changes.

Reality: Redis does not retry automatically; clients must detect aborts and retry manually.

Why it matters: Assuming automatic retries can cause lost updates or inconsistent data if clients don't handle aborts properly.

Quick: Does WATCH guarantee no conflicts if you watch only one key in a multi-key update? Commit yes or no.

Common Belief: Watching one key is enough to prevent conflicts in multi-key updates.

Reality: WATCH only monitors specified keys; changes to un-watched keys can still cause conflicts.

Why it matters: Partial watching can lead to subtle data inconsistencies if all relevant keys are not watched.

Expert Zone

1

WATCH is connection-specific; multiple connections watching the same keys do not interfere with each other.

2

WATCH resets after EXEC or DISCARD, so clients must re-watch keys for each transaction attempt.

3

Using WATCH with Lua scripts can be tricky because scripts run atomically and bypass WATCH checks.

When NOT to use

WATCH is not suitable when you need guaranteed locking or blocking behavior; in such cases, use Redis locks (SETNX with expiration) or Lua scripts for atomic operations.

Production Patterns

In production, WATCH is often combined with retry loops in clients to handle transaction aborts gracefully. It's used in counters, inventory systems, and session management where concurrent updates are common but conflicts are rare.

Connections

Optimistic Concurrency Control (OCC)

WATCH implements OCC pattern in Redis.

Understanding WATCH helps grasp OCC, a common concurrency control method in databases and distributed systems.

Database Transactions

WATCH extends Redis transactions with conflict detection.

Knowing how WATCH works deepens understanding of transaction isolation and consistency in databases.

Version Control Systems

WATCH's conflict detection is similar to how version control detects changes before merging.

Recognizing this connection helps appreciate conflict detection as a universal pattern beyond databases.

Common Pitfalls

#1 Assuming WATCH locks keys and prevents other clients from changing them.

Wrong approach:

WATCH mykey
MULTI
SET mykey newvalue
EXEC

Correct approach:

WATCH mykey
GET mykey
MULTI
SET mykey newvalue
EXEC
if EXEC returns null then retry

Root cause:

Misunderstanding that WATCH only monitors changes but does not block other clients.

#2 Not retrying the transaction after EXEC returns null due to key changes.

Wrong approach:

WATCH mykey
MULTI
INCR mykey
EXEC
// no retry on null result

Correct approach:

do {
WATCH mykey
val = GET mykey
MULTI
INCR mykey
result = EXEC
} while (result == null)

Root cause:

Ignoring that transactions can abort and must be retried to ensure success.

#3 Watching only some keys involved in a multi-key update.

Wrong approach:

WATCH key1
MULTI
SET key1 val1
SET key2 val2
EXEC

Correct approach:

WATCH key1 key2
MULTI
SET key1 val1
SET key2 val2
EXEC

Root cause:

Not watching all keys that can cause conflicts leads to undetected data races.

Key Takeaways

WATCH in Redis provides optimistic locking by monitoring keys for changes before committing transactions.

It does not lock keys but aborts transactions if watched keys change, allowing safe concurrent updates.

Clients must handle transaction aborts by retrying the entire WATCH-MULTI-EXEC sequence.

WATCH is connection-specific and resets after each transaction attempt.

Using WATCH effectively requires watching all relevant keys and understanding its limitations compared to locks or atomic scripts.