Worker pool pattern in Node.js - Mini Project: Build & Apply

Worker Pool Pattern in Node.js

📖 Scenario: You are building a Node.js application that needs to process multiple CPU-intensive tasks efficiently. To avoid blocking the main thread, you will use the worker pool pattern to distribute tasks across multiple worker threads.

🎯 Goal: Create a worker pool that manages a fixed number of worker threads. The pool will accept tasks and assign them to available workers, ensuring tasks run concurrently without blocking the main thread.

📋 What You'll Learn

Create a list of tasks to process

Set a maximum number of worker threads in the pool

Implement logic to assign tasks to workers using the worker pool pattern

Complete the worker pool setup to handle task execution and results

💡 Why This Matters

🌍 Real World

Worker pools help Node.js applications perform CPU-heavy tasks without blocking the main event loop, improving responsiveness and throughput.

💼 Career

Understanding worker pools is important for backend developers building scalable and efficient Node.js services that handle parallel processing.

1

Create the list of tasks

Create a constant array called tasks with these exact numbers: 100000000, 200000000, 300000000, 400000000, 500000000.

Node.js

# Create the tasks array with the specified numbers
# Your code here

Need a hint?

Use const tasks = [100000000, 200000000, 300000000, 400000000, 500000000]; to create the array.

2

Set the maximum number of workers

Create a constant called maxWorkers and set it to 2 to limit the number of worker threads in the pool.

Node.js

const tasks = [100000000, 200000000, 300000000, 400000000, 500000000];
# Define maxWorkers to 2
# Your code here

Need a hint?

Use const maxWorkers = 2; to set the worker limit.

3

Implement the worker pool logic

Import Worker from 'worker_threads'. Create a function called runTask that takes a number task and returns a Promise. Inside, create a new Worker using new Worker('./worker.js', { workerData: task }). Resolve the Promise when the worker sends a message, and reject if the worker emits an error.

Node.js

const tasks = [100000000, 200000000, 300000000, 400000000, 500000000];
const maxWorkers = 2;

// Import Worker and create runTask function
# Your code here

Need a hint?

Use require('worker_threads') to import Worker. Wrap the worker creation in a Promise and handle message and error events.

4

Complete the worker pool to process all tasks

Create an async function called runWorkerPool. Inside, create an empty array results. Use a while loop to process tasks while tasks.length > 0. In each loop, create an array promises by slicing tasks to get up to maxWorkers tasks, then map each to runTask. Use await Promise.all(promises) to get results and push them into results. Remove processed tasks from tasks. Finally, return results. Then call runWorkerPool().

Node.js

const { Worker } = require('worker_threads');
const tasks = [100000000, 200000000, 300000000, 400000000, 500000000];
const maxWorkers = 2;

function runTask(task) {
    return new Promise((resolve, reject) => {
        const worker = new Worker('./worker.js', { workerData: task });
        worker.on('message', resolve);
        worker.on('error', reject);
        worker.on('exit', (code) => {
            if (code !== 0) reject(new Error(`Worker stopped with exit code ${code}`));
        });
    });
}

// Create runWorkerPool async function to process all tasks
# Your code here

Need a hint?

Use a while loop to process tasks in batches of maxWorkers. Use Promise.all to wait for all workers in the batch to finish.