Rate limiter with INCR and EXPIRE in Redis - Deep Dive

Overview - Rate limiter with INCR and EXPIRE

What is it?

A rate limiter controls how often an action can happen in a set time. Using Redis commands INCR and EXPIRE, we count actions and reset the count after a time window. This helps stop too many requests or actions happening too fast. It works by increasing a counter and setting a timer to reset it.

Why it matters

Without rate limiting, systems can get overwhelmed by too many requests, causing slowdowns or crashes. This can ruin user experience and waste resources. Rate limiting protects services by controlling traffic flow, ensuring fairness and stability. It helps websites, APIs, and apps stay reliable even under heavy use.

Where it fits

Before learning this, you should understand basic Redis commands and how key-value stores work. After this, you can explore more advanced rate limiting techniques like token buckets or sliding windows, and how to combine Redis with other tools for distributed systems.

Mental Model

Core Idea

A rate limiter counts actions in Redis and resets the count after a set time to control how often something happens.

Think of it like...

It's like a ticket counter at a bus stop that counts how many people get on, and after a certain time, the counter resets to zero to start counting again.

┌───────────────┐
│ User Action   │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Redis INCR    │  <-- increases count
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Check count   │
│ against limit │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Redis EXPIRE  │  <-- sets reset timer
└───────────────┘

Build-Up - 6 Steps

1

Foundation

Understanding Redis INCR command

Concept: Learn how INCR increases a numeric value stored in Redis by 1.

The INCR command adds 1 to the number stored at a key. If the key does not exist, Redis creates it with value 0 before incrementing. This is atomic, so no two increments clash. Example: INCR user:123:requests If the key 'user:123:requests' was 5, it becomes 6.

Result

The key's value increases by 1 each time INCR runs, or starts at 1 if new.

Understanding INCR is key because it lets us count actions safely and efficiently without race conditions.

2

Foundation

Using EXPIRE to set key lifetime

Concept: Learn how EXPIRE sets a time limit on a Redis key before it is deleted.

The EXPIRE command sets a countdown in seconds for a key to live. After the time ends, Redis deletes the key automatically. Example: EXPIRE user:123:requests 60 This means the key 'user:123:requests' will be removed after 60 seconds.

Result

The key will exist only for the set time, then disappear.

Knowing EXPIRE lets us reset counters automatically, which is essential for time-based rate limiting.

3

Intermediate

Combining INCR and EXPIRE for counting

🤔Before reading on: do you think EXPIRE resets the timer every time INCR runs, or only once when the key is created? Commit to your answer.

Concept: Use INCR to count actions and EXPIRE to set the time window for counting.

When a user performs an action, run INCR on their key. If the key is new (first action), set EXPIRE to start the countdown. If the key exists, do not reset EXPIRE to avoid extending the window. Example sequence: 1. INCR user:123:requests 2. If result is 1, EXPIRE user:123:requests 60 This counts actions in 60 seconds.

Result

The counter increases with each action, and resets after 60 seconds from the first action.

Understanding that EXPIRE should be set only once prevents the time window from extending endlessly, keeping the rate limit fair.

4

Intermediate

Checking limits to block excess actions

🤔Before reading on: do you think you should check the count before or after incrementing? Commit to your answer.

Concept: Compare the current count to a limit to decide if the action is allowed.

After INCR, check if the count exceeds the allowed limit. If it does, reject the action. Example: count = INCR user:123:requests if count == 1: EXPIRE user:123:requests 60 if count > 10: reject action else allow action This stops users from doing more than 10 actions per minute.

Result

Actions beyond the limit are blocked until the counter resets.

Knowing when to check the count ensures the rate limiter enforces limits correctly without blocking too early or too late.

5

Advanced

Atomic rate limiting with Lua scripts

🤔Before reading on: do you think running INCR and EXPIRE separately can cause race conditions? Commit to your answer.

Concept: Use Redis Lua scripts to run INCR and EXPIRE together atomically to avoid timing issues.

Running INCR and EXPIRE as separate commands can cause race conditions if multiple requests happen simultaneously. Lua scripts run commands atomically in Redis. Example script: local current = redis.call('INCR', KEYS[1]) if current == 1 then redis.call('EXPIRE', KEYS[1], ARGV[1]) end return current This script increments and sets expiry in one step.

Result

The count and expiry are set safely without conflicts, ensuring accurate rate limiting.

Understanding atomic operations prevents subtle bugs in high-traffic systems where commands can overlap.

6

Expert

Limitations and edge cases of INCR+EXPIRE rate limiter

🤔Before reading on: do you think this rate limiter perfectly handles bursts and sliding windows? Commit to your answer.

Concept: Explore where this simple rate limiter can fail or behave unexpectedly in real systems.

This method counts actions in fixed windows, so bursts at window edges can bypass limits (e.g., 10 actions at end of one window and 10 at start of next). It also does not smooth traffic over time. For more precise control, advanced algorithms like sliding window or token bucket are used. Also, clock skew or Redis failures can affect accuracy.

Result

The simple rate limiter works well for many cases but can allow bursts and has timing limitations.

Knowing the limits helps choose the right rate limiting strategy for your system's needs.

Under the Hood

Redis stores keys with values and expiration times. INCR atomically increases the integer value of a key, creating it if missing. EXPIRE sets a TTL (time to live) on the key, after which Redis deletes it. When combined, INCR counts actions, and EXPIRE ensures the count resets after a time window. Redis handles these operations in memory with high speed and atomicity, making it ideal for rate limiting.

Why designed this way?

Redis was designed for speed and atomic operations to support real-time applications. INCR and EXPIRE commands are simple but powerful primitives that can be combined flexibly. This design avoids complex locking or external timers, reducing latency and errors. Alternatives like external counters or databases would be slower or more complex.

┌───────────────┐
│ Client sends  │
│ action request│
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Redis INCR    │  <-- atomic increment
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Check if count│
│ == 1          │
└──────┬────────┘
       │ yes
       ▼
┌───────────────┐
│ Redis EXPIRE  │  <-- set TTL
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Compare count │
│ to limit      │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Allow or block│
│ action        │
└───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does EXPIRE reset every time INCR runs? Commit yes or no.

Common Belief: EXPIRE resets the timer every time INCR runs, so the window moves forward.

Reality: EXPIRE should be set only once when the key is created; otherwise, the window extends indefinitely.

Why it matters: Resetting EXPIRE on every increment breaks the fixed window, allowing users to avoid limits by spreading actions.

Quick: Is running INCR and EXPIRE separately always safe? Commit yes or no.

Common Belief: Running INCR and EXPIRE as separate commands is safe and race-free.

Reality: Separating them can cause race conditions where expiry is not set correctly if multiple requests happen simultaneously.

Why it matters: Race conditions can cause counters to never expire or expire too early, breaking rate limiting.

Quick: Does this rate limiter perfectly smooth out request bursts? Commit yes or no.

Common Belief: This method prevents all bursts by evenly spacing requests.

Reality: It uses fixed windows, so bursts at window edges can exceed limits temporarily.

Why it matters: Unexpected bursts can overload systems or cause unfair blocking.

Quick: Can Redis INCR handle non-numeric values? Commit yes or no.

Common Belief: INCR can increment any value stored at a key.

Reality: INCR only works on keys holding integers or keys that don't exist; otherwise, it errors.

Why it matters: Using INCR on wrong data types causes errors and breaks rate limiting.

Expert Zone

1

The EXPIRE timer starts from the first increment, not from each action, which means the window is fixed, not sliding.

2

Using Lua scripts for atomic INCR and EXPIRE prevents subtle race conditions in high concurrency environments.

3

Redis eviction policies and memory limits can affect keys with EXPIRE, potentially removing counters early under memory pressure.

When NOT to use

Avoid this simple rate limiter when you need smooth rate limiting or sliding windows. Use token bucket or leaky bucket algorithms instead, or specialized libraries like Redis Cell or external rate limiting services.

Production Patterns

In production, this pattern is often wrapped in Lua scripts for atomicity and combined with user identifiers or IP addresses as keys. It's used in APIs, login throttling, and spam prevention. Monitoring and fallback mechanisms handle Redis failures gracefully.

Connections

Token Bucket Algorithm

Builds on and improves fixed window counting by smoothing request rates over time.

Understanding fixed window limits helps grasp why token buckets provide better control for bursty traffic.

Distributed Systems Consistency

Shares challenges with atomic operations and race conditions in distributed environments.

Knowing how Redis commands ensure atomicity aids understanding of consistency models in distributed systems.

Traffic Shaping in Networking

Both control flow rates to prevent overload, using counters and timers.

Seeing rate limiting as traffic shaping connects database commands to network engineering principles.

Common Pitfalls

#1 Resetting EXPIRE on every INCR call, extending the time window.

Wrong approach:

INCR user:123:requests
EXPIRE user:123:requests 60
INCR user:123:requests
EXPIRE user:123:requests 60

Correct approach:

count = INCR user:123:requests
if count == 1 then
EXPIRE user:123:requests 60
end

Root cause:

Misunderstanding that EXPIRE should be set only once to keep a fixed time window.

#2 Running INCR and EXPIRE as separate commands without atomicity.

Wrong approach:

INCR user:123:requests
EXPIRE user:123:requests 60

Correct approach:

Use Lua script:
local current = redis.call('INCR', KEYS[1])
if current == 1 then
redis.call('EXPIRE', KEYS[1], ARGV[1])
end
return current

Root cause:

Not accounting for race conditions when multiple requests happen simultaneously.

#3 Using INCR on keys with non-integer values causing errors.

Wrong approach:

SET user:123:requests 'hello'
INCR user:123:requests

Correct approach:

Ensure key is new or holds an integer before INCR, or delete/reset key if needed.

Root cause:

Assuming INCR works on any data type without checking.

Key Takeaways

Rate limiting with INCR and EXPIRE uses Redis to count actions and reset counts after a time window.

INCR atomically increases counters, and EXPIRE sets how long counters live before resetting.

Setting EXPIRE only once per window keeps the time fixed and prevents unfair extensions.

Atomic operations via Lua scripts avoid race conditions in concurrent environments.

This simple method works well but has limits like burstiness and fixed windows; advanced algorithms may be needed for complex cases.