Worker pool pattern in Node.js - Step-by-Step Execution

Concept Flow - Worker pool pattern

Start main thread

↓

Create worker pool

↓

Assign tasks to workers

↓

Worker executes task

↓

Worker sends result back

↓

Main thread collects results

↓

Check if more tasks

↓

Assign next

The main thread creates a pool of workers, assigns tasks to them, collects results, and repeats until all tasks are done, then shuts down the pool.

Execution Sample

Node.js

import { Worker } from 'worker_threads';

const pool = [];
for(let i=0; i<2; i++) pool.push(new Worker('./worker.js'));

pool[0].postMessage('task1');
pool[1].postMessage('task2');

This code creates two workers and sends each a task message to start processing.

Execution Table

Step	Action	Worker State	Main Thread State	Message Passing
1	Create Worker 0	Idle	Pool: [Worker0]	No message
2	Create Worker 1	Idle	Pool: [Worker0, Worker1]	No message
3	Send 'task1' to Worker 0	Busy with task1	Pool unchanged	'task1' sent to Worker0
4	Send 'task2' to Worker 1	Busy with task2	Pool unchanged	'task2' sent to Worker1
5	Worker 0 processes task1	Processing task1	Waiting for result	No message
6	Worker 1 processes task2	Processing task2	Waiting for result	No message
7	Worker 0 sends result	Idle	Received result1	'result1' received from Worker0
8	Worker 1 sends result	Idle	Received result2	'result2' received from Worker1
9	Check for more tasks	Idle	No more tasks, shutdown pool	No message
10	Shutdown workers	Terminated	Pool empty	No message

💡 All tasks processed and results collected, workers shut down.

Variable Tracker

Variable	Start	After Step 3	After Step 5	After Step 7	Final
pool	[]	[Worker0, Worker1]	[Worker0, Worker1]	[Worker0, Worker1]	[]
Worker0 state	Idle	Busy with task1	Processing task1	Idle	Terminated
Worker1 state	Idle	Busy with task2	Processing task2	Idle	Terminated
Main thread results	[]	[]	[]	[result1]	[result1, result2]

Key Moments - 3 Insights

Why does the main thread wait after sending tasks?

Because workers run tasks asynchronously, the main thread waits to receive results before proceeding, as shown in steps 5 and 6 where workers process tasks while the main thread waits.

What happens if a worker finishes a task before others?

The worker sends its result back and becomes idle, ready for new tasks. This is shown in step 7 where Worker0 finishes and sends result while Worker1 is still processing.

Why do we shut down workers at the end?

To free system resources once all tasks are done, as shown in step 10 where workers terminate after no more tasks remain.

Visual Quiz - 3 Questions

Look at the execution table, what is Worker0's state at step 6?

A Processing task1

B Idle

C Busy with task2

D Terminated

💡 Hint

Check the 'Worker State' column at step 6 in the execution_table.

At which step does the main thread receive the first result?

A Step 5

B Step 8

C Step 7

D Step 10

💡 Hint

Look at the 'Main Thread State' and 'Message Passing' columns for when 'result1' is received.

If we add a third worker, how would the pool variable change after creation?

A pool would have 1 worker

B pool would have 3 workers

C pool would have 2 workers

D pool would be empty

💡 Hint

Refer to the variable_tracker 'pool' row after worker creation steps.

Concept Snapshot

Worker Pool Pattern in Node.js:
- Create multiple Worker threads in a pool.
- Assign tasks to idle workers asynchronously.
- Workers process tasks and send results back.
- Main thread collects results and assigns new tasks.
- Shutdown workers when all tasks complete.

Full Transcript

The Worker Pool Pattern in Node.js helps run multiple tasks in parallel using worker threads. First, the main thread creates a pool of workers. Then, it sends tasks to each worker. Workers run tasks independently and send results back. The main thread waits for these results and can assign more tasks if needed. When all tasks are done, the main thread shuts down the workers to save resources. This pattern improves performance by using multiple threads efficiently.