Script loading and caching in Redis - Deep Dive

Overview - Script loading and caching

What is it?

Script loading and caching in Redis means saving Lua scripts on the server so they can run faster and more efficiently. Instead of sending the whole script every time, Redis stores it once and runs it by a short identifier. This helps reduce network traffic and speeds up repeated script execution.

Why it matters

Without script loading and caching, every time you want to run a Lua script on Redis, you must send the entire script over the network. This wastes time and bandwidth, especially for large or frequently used scripts. By caching scripts, Redis makes operations faster and reduces delays, improving the performance of applications that rely on it.

Where it fits

Before learning script loading and caching, you should understand basic Redis commands and Lua scripting in Redis. After this, you can explore advanced scripting techniques, error handling in scripts, and optimizing Redis performance with scripts.

Mental Model

Core Idea

Redis stores Lua scripts once and runs them by a short ID to save time and network resources.

Think of it like...

It's like memorizing a phone number instead of dialing the full address every time you want to call someone.

┌───────────────┐       ┌───────────────┐
│ Client sends  │       │ Redis server  │
│ full Lua      │──────▶│ stores script │
│ script once   │       │ and returns   │
└───────────────┘       │ script SHA1   │
                        └──────┬────────┘
                               │
                               ▼
                        ┌───────────────┐
                        │ Client sends  │
                        │ script SHA1   │
                        │ to execute    │
                        └───────────────┘

Build-Up - 7 Steps

1

Foundation

What is Redis Lua scripting

Concept: Introducing Lua scripting in Redis as a way to run custom commands atomically.

Redis allows running Lua scripts to perform multiple commands in one step. This helps keep operations safe and fast because the script runs fully or not at all. You send the whole Lua script to Redis, and it executes it.

Result

You can run complex operations in Redis with one command.

Understanding Lua scripting is the base for knowing why caching scripts matters.

2

Foundation

How scripts run without caching

Concept: Explaining that without caching, the full script is sent every time to Redis.

Each time you want to run a Lua script, you send the entire script text to Redis. Redis parses and runs it immediately. This means more data sent over the network and more time spent parsing.

Result

Every script execution involves sending the full script and parsing it again.

Knowing this shows why sending full scripts repeatedly is inefficient.

3

Intermediate

Script loading with SCRIPT LOAD command

Concept: Introducing SCRIPT LOAD to save a script on Redis and get a SHA1 hash as its ID.

You can send SCRIPT LOAD with your Lua script once. Redis stores it and returns a SHA1 hash. This hash is a short ID representing your script.

Result

Redis stores the script and gives you a short identifier (SHA1).

Understanding SCRIPT LOAD is key to using script caching effectively.

4

Intermediate

Running cached scripts with EVALSHA

🤔Before reading on: do you think EVALSHA requires the full script or just the SHA1 hash? Commit to your answer.

Concept: Using EVALSHA to run a cached script by its SHA1 hash instead of sending the full script.

After loading a script, you run it by sending EVALSHA with the SHA1 hash and any needed arguments. Redis finds the script by hash and runs it without needing the full script text.

Result

Scripts run faster because only the short hash and arguments are sent.

Knowing EVALSHA reduces network load and speeds up repeated script execution.

5

Intermediate

Handling missing cached scripts

🤔Before reading on: if you run EVALSHA with a wrong hash, do you think Redis runs the script or returns an error? Commit to your answer.

Concept: What happens if Redis does not have the script for the given SHA1 hash.

If you call EVALSHA with a hash Redis doesn't know, it returns a NOSCRIPT error. Then you must send the full script again with EVAL or SCRIPT LOAD to cache it.

Result

You get an error if the script is not cached, so you must reload it.

Understanding this helps avoid errors and manage script caching properly.

6

Advanced

Benefits of script caching in production

🤔Before reading on: do you think caching scripts only saves bandwidth or also improves CPU usage? Commit to your answer.

Concept: Explaining how caching scripts reduces parsing overhead and network traffic in real systems.

In production, caching scripts means Redis parses the script once, saving CPU time on repeated runs. It also reduces network data sent, improving latency and throughput for busy applications.

Result

Better performance and lower resource use in real-world Redis deployments.

Knowing these benefits guides when to use script caching for efficiency.

7

Expert

Script caching internals and eviction

🤔Before reading on: do you think Redis automatically removes cached scripts over time? Commit to your answer.

Concept: How Redis stores cached scripts internally and what happens if scripts are evicted or Redis restarts.

Redis stores cached scripts in memory keyed by SHA1. Scripts stay cached until Redis restarts or SCRIPT FLUSH is called. Redis does not automatically evict scripts, so clients must handle reloading if needed.

Result

Cached scripts persist during runtime but not across restarts unless reloaded.

Understanding script lifecycle prevents unexpected NOSCRIPT errors in production.

Under the Hood

When SCRIPT LOAD is called, Redis computes the SHA1 hash of the Lua script and stores the script text in an internal dictionary keyed by this hash. When EVALSHA is called, Redis looks up the script by hash and executes it directly without parsing. If the script is missing, Redis returns an error. Scripts remain in memory until explicitly flushed or Redis restarts.

Why designed this way?

This design balances speed and memory use. Using SHA1 hashes avoids sending large scripts repeatedly, saving bandwidth and CPU. Redis does not auto-evict scripts to avoid complexity and unexpected failures. This approach keeps script execution fast and predictable.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│ Client sends  │──────▶│ Redis computes │──────▶│ Stores script │
│ SCRIPT LOAD   │       │ SHA1 hash     │       │ keyed by SHA1 │
└───────────────┘       └───────────────┘       └──────┬────────┘
                                                       │
                                                       ▼
┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│ Client sends  │──────▶│ Redis looks   │──────▶│ Executes Lua  │
│ EVALSHA with  │       │ up script by  │       │ script from   │
│ SHA1 hash     │       │ SHA1 hash     │       │ cache         │
└───────────────┘       └───────────────┘       └───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does EVALSHA run the script even if Redis never saw it before? Commit yes or no.

Common Belief: EVALSHA will run the script even if Redis has not cached it yet.

Reality: EVALSHA returns a NOSCRIPT error if the script is not cached on Redis.

Why it matters: Assuming EVALSHA always works can cause unexpected errors and failures in applications.

Quick: Does Redis automatically remove cached scripts when memory is low? Commit yes or no.

Common Belief: Redis automatically evicts cached scripts when memory is tight.

Reality: Redis does not evict cached scripts automatically; they stay until flushed or restart.

Why it matters: Believing in automatic eviction can lead to missing scripts and errors if clients don't reload them.

Quick: Does caching scripts only save network bandwidth? Commit yes or no.

Common Belief: Script caching only reduces the amount of data sent over the network.

Reality: Caching scripts also saves CPU time by avoiding repeated parsing of the same script.

Why it matters: Ignoring CPU savings misses a key performance benefit of script caching.

Quick: Can you modify a cached script by sending a new SCRIPT LOAD with the same SHA1? Commit yes or no.

Common Belief: Sending SCRIPT LOAD again with the same script updates the cached script.

Reality: Scripts are identified by SHA1; if the script text changes, the SHA1 changes, so Redis stores it as a new script.

Why it matters: Thinking you can update scripts by reloading with the same SHA1 causes confusion and stale script use.

Expert Zone

1

Cached scripts are stored in Redis memory and do not persist across restarts, so clients must handle reloading scripts after a restart.

2

Using SCRIPT FLUSH removes all cached scripts, which can cause NOSCRIPT errors if clients are not prepared to reload scripts.

3

SHA1 hashes are collision-resistant but not impossible to collide; however, Redis assumes uniqueness for script identification.

When NOT to use

Script caching is not suitable when scripts change frequently or are unique per execution. In such cases, sending full scripts with EVAL is better. Also, for very simple commands, scripting overhead may not be justified.

Production Patterns

In production, scripts are loaded once at application startup using SCRIPT LOAD, then executed repeatedly with EVALSHA. Applications handle NOSCRIPT errors by reloading scripts automatically. Monitoring SCRIPT FLUSH commands and Redis restarts is important to maintain script availability.

Connections

Content Delivery Networks (CDNs)

Both cache content to reduce repeated data transfer and improve speed.

Understanding script caching in Redis is similar to how CDNs cache web content to reduce bandwidth and latency.

Function Memoization in Programming

Script caching stores results (scripts) to avoid recomputation (parsing).

Knowing memoization helps grasp why caching scripts avoids repeated parsing and speeds execution.

Compiler Caching in Software Build Systems

Both cache compiled code or scripts to avoid recompiling on every run.

Recognizing this pattern across domains shows how caching improves efficiency by reusing work.

Common Pitfalls

#1 Running EVALSHA without loading the script first.

Wrong approach:

EVALSHA 0

Correct approach:

SCRIPT LOAD