Handling worker crashes and restart in Node.js - Step-by-Step Execution

Concept Flow - Handling worker crashes and restart

Start Master Process

↓

Fork Worker Process

↓

Worker Runs

↓

Worker Crashes?

No→Continue Running

Yes↓

Master Detects Crash

↓

Master Forks New Worker

↓

Worker Runs

The master process starts and forks a worker. If the worker crashes, the master detects it and restarts a new worker, keeping the system running.

Execution Sample

Node.js

import cluster from 'cluster';
import os from 'os';

if (cluster.isPrimary) {
  cluster.fork();
  cluster.on('exit', (worker) => {
    console.log(`Worker ${worker.process.pid} died, restarting...`);
    cluster.fork();
  });
} else {
  // Worker code
  setTimeout(() => { process.exit(1); }, 1000); // Simulate crash
}

This code forks a worker that crashes after 1 second; the master detects the crash and restarts the worker.

Execution Table

Step	Action	Worker PID	Master Detects Crash	Master Action	Output
1	Master starts and forks worker	12345	No	None	Worker 12345 started
2	Worker runs normally	12345	No	None	Worker 12345 running
3	Worker crashes (exit code 1)	12345	Yes	Fork new worker	Worker 12345 died, restarting...
4	Master forks new worker	12346	No	None	Worker 12346 started
5	New worker runs	12346	No	None	Worker 12346 running
6	No more crashes	-	No	None	System stable
Exit	No crash detected	-	No	None	Execution stops

💡 No crash detected, system runs stable

Variable Tracker

Variable	Start	After Step 1	After Step 3	After Step 4	Final
worker PID	none	12345	12345 (crashed)	12346	12346
master detects crash	false	false	true	false	false

Key Moments - 2 Insights

Why does the master fork a new worker after a crash?

Because the execution_table row 3 shows the master detects the worker crash and immediately forks a new worker to keep the system running.

Does the master fork multiple workers at start?

No, the master forks only one worker at start as shown in execution_table row 1; additional forks happen only after crashes.

Visual Quiz - 3 Questions

Look at the execution_table at step 3, what does the master do when the worker crashes?

A Forks a new worker

B Does nothing

C Shuts down the system

D Logs but does not restart

💡 Hint

See execution_table row 3 under 'Master Action' and 'Output'

At which step does the new worker start running after the crash?

A Step 2

B Step 4

C Step 5

D Step 6

💡 Hint

Check execution_table rows 4 and 5 for worker start and running

If the worker never crashes, what happens to the master’s restart logic?

A It runs repeatedly

B It forks multiple workers at start

C It never runs

D It shuts down the master

💡 Hint

Look at variable_tracker 'master detects crash' staying false and execution_table exit note

Concept Snapshot

Node.js cluster master forks worker(s).
If a worker crashes, master detects 'exit' event.
Master forks a new worker to replace crashed one.
This keeps the app running continuously.
Use cluster.on('exit', ...) to handle crashes.
Restart logic runs only on worker crash.

Full Transcript

In Node.js cluster, the master process starts and forks a worker process. The worker runs its code. If the worker crashes, the master detects this via the 'exit' event. Upon detection, the master immediately forks a new worker to replace the crashed one. This cycle ensures the application keeps running even if workers fail. The example code forks one worker that crashes after one second. The master logs the crash and restarts the worker. Variables like worker PID and crash detection status change as the program runs. The master only forks new workers after crashes, not multiple at start. This pattern is essential for building resilient Node.js applications using cluster.

Practice

1. What is the main purpose of listening to the exit event on a worker in Node.js cluster module?

easy

A. To log the worker's CPU usage

B. To start a new worker automatically

C. To send messages between workers

D. To detect when a worker crashes or stops running

Solution

1. Step 1: Understand the exit event role

   The exit event is triggered when a worker process stops, either normally or due to a crash.

2. Step 2: Identify the purpose of listening to exit

   Listening to exit helps detect unexpected worker crashes so the master can respond.

3. Final Answer:

   To detect when a worker crashes or stops running -> Option D

4. Quick Check:

   exit event = detect crash [OK]

Hint: Remember: exit event means worker stopped or crashed [OK]

Common Mistakes:

• Confusing exit event with message passing
• Thinking exit event starts new workers automatically
• Assuming exit event logs CPU usage

2. Which of the following is the correct way to listen for a worker's exit event in Node.js cluster?

easy

A. cluster.on('exit', worker => { /* handle exit */ });

B. worker.on('exit', () => { /* handle exit */ });

C. worker.listen('exit', () => { /* handle exit */ });

D. process.on('workerExit', () => { /* handle exit */ });

Solution

1. Step 1: Recall event listener syntax on worker

   In Node.js cluster, each worker is an EventEmitter and uses on to listen to events.

2. Step 2: Match correct event and method

   The correct event is exit and the method is on, so worker.on('exit', ...) is correct.

3. Final Answer:

   worker.on('exit', () => { /* handle exit */ }); -> Option B

4. Quick Check:

   Use on with exit on worker [OK]

Hint: Use worker.on('exit', callback) to catch exit events [OK]

Common Mistakes:

• Using .listen instead of .on
• Listening on cluster instead of worker
• Using wrong event name like 'workerExit'

3. Given the code below, what will be logged when a worker crashes?

const cluster = require('cluster');
if (cluster.isMaster) {
  const worker = cluster.fork();
  worker.on('exit', (code, signal) => {
    console.log(`Worker exited with code ${code} and signal ${signal}`);
  });
} else {
  process.exit(1); // Simulate crash
}

medium

A. Worker exited with code null and signal SIGTERM

B. Worker exited with code 0 and signal null

C. Worker exited with code 1 and signal null

D. No output because exit event is not triggered

Solution

1. Step 1: Understand process.exit(1) effect

   Calling process.exit(1) ends the worker with exit code 1, indicating an error.

2. Step 2: Check exit event parameters

   The exit event callback receives the exit code and signal; here signal is null because no signal caused the exit.

3. Final Answer:

   Worker exited with code 1 and signal null -> Option C

4. Quick Check:

   Exit code 1 means crash, signal null if no signal [OK]

Hint: Exit code 1 means crash, signal null if no signal sent [OK]

Common Mistakes:

• Assuming exit code 0 means crash
• Confusing signal with exit code
• Thinking exit event won't fire on crash

4. Identify the error in this code snippet that tries to restart a worker after it crashes:

const cluster = require('cluster');
if (cluster.isMaster) {
  cluster.fork();
  cluster.on('exit', (worker) => {
    console.log('Worker crashed, restarting...');
    cluster.fork();
  });
}

medium

A. The 'exit' event should be listened on 'cluster' but the callback parameter should be (worker, code, signal)

B. The 'exit' event should be listened on 'cluster.workers', not 'cluster'

C. The 'exit' event should be listened on 'cluster' but the callback parameters are wrong

D. The 'exit' event callback parameters are incorrect; it should receive (code, signal)

Solution

1. Step 1: Check where to listen for worker exit

   The 'exit' event is emitted by the cluster module, and the callback receives (worker, code, signal).

2. Step 2: Identify callback parameter mismatch

   The code uses only one parameter (worker), but the event provides three parameters; this can cause confusion or errors.

3. Final Answer:

   The 'exit' event should be listened on 'cluster' but the callback parameter should be (worker, code, signal) -> Option A

4. Quick Check:

   cluster.on('exit', (worker, code, signal)) is correct [OK]

Hint: cluster.on('exit') callback needs (worker, code, signal) parameters [OK]

Common Mistakes:

• Listening on cluster.workers instead of cluster
• Using wrong callback parameters
• Ignoring code and signal parameters

5. You want to ensure your Node.js app automatically restarts a worker if it crashes, but only up to 3 restarts per minute to avoid infinite loops. Which approach best implements this behavior?

hard

A. Use a counter and timestamp in the master process to track restarts; restart only if under limit

B. Restart workers immediately on every exit event without limits

C. Use a setTimeout to delay restarts by 1 minute after each crash

D. Restart workers only if exit code is 0, ignore other exit codes

Solution

1. Step 1: Understand the need to limit restarts

   Unlimited restarts can cause infinite loops if the worker crashes repeatedly.

2. Step 2: Implement a counter and timestamp logic

   Track how many times workers restart within a time window (e.g., 3 restarts per minute) and only restart if under the limit.

3. Final Answer:

   Use a counter and timestamp in the master process to track restarts; restart only if under limit -> Option A

4. Quick Check:

   Limit restarts with counter and time check [OK]

Hint: Count restarts with time checks to avoid infinite loops [OK]

Common Mistakes:

• Restarting without limits causing infinite loops
• Delaying restarts but not counting attempts
• Restarting only on exit code 0 (normal exit)