Queue consumers (processors) in NestJS

Introduction

Queue consumers (processors) help your app handle tasks one by one or in groups without blocking other work. They listen to a queue and process jobs as they come.

When you want to send emails without making users wait.

When you need to resize images after upload without slowing down the app.

When processing orders or payments in the background.

When handling notifications that can be delayed.

When you want to retry failed tasks automatically.

Syntax

NestJS

import { Processor, Process } from '@nestjs/bull';
import { Job } from 'bull';

@Processor('queueName')
export class QueueConsumer {
  @Process('jobName')
  async handleJob(job: Job<any>) {
    // Your job processing logic here
  }
}

@Processor decorator marks the class as a queue consumer for a specific queue.

@Process decorator marks the method that will handle jobs with a specific name.

Examples

Processes 'sendEmail' jobs from 'emailQueue'. Logs email details.

NestJS

import { Processor, Process } from '@nestjs/bull';
import { Job } from 'bull';

@Processor('emailQueue')
export class EmailConsumer {
  @Process('sendEmail')
  async sendEmailJob(job: Job<{ to: string; subject: string }>) {
    console.log(`Sending email to ${job.data.to} with subject ${job.data.subject}`);
  }
}

Handles 'resizeImage' jobs and checks if image path is provided before processing.

NestJS

import { Processor, Process } from '@nestjs/bull';
import { Job } from 'bull';

@Processor('imageQueue')
export class ImageConsumer {
  @Process('resizeImage')
  async resizeImageJob(job: Job<{ imagePath: string }>) {
    if (!job.data.imagePath) {
      console.log('No image path provided');
      return;
    }
    console.log(`Resizing image at ${job.data.imagePath}`);
  }
}

Example of a consumer with no jobs added yet. Shows what happens if queue is empty.

NestJS

import { Processor, Process } from '@nestjs/bull';
import { Job } from 'bull';

@Processor('emptyQueue')
export class EmptyConsumer {
  @Process('noJob')
  async noJobHandler(job: Job) {
    console.log('This will never run because no jobs are added');
  }
}

Sample Program

This NestJS app sets up a queue named 'taskQueue' and a consumer that processes 'printTask' jobs. It adds one job with a message, which the consumer prints.

NestJS

import { Injectable, Module } from '@nestjs/common';
import { BullModule } from '@nestjs/bull';
import { InjectQueue, Processor, Process } from '@nestjs/bull';
import { Job, Queue } from 'bull';
import { NestFactory } from '@nestjs/core';

@Processor('taskQueue')
export class TaskConsumer {
  @Process('printTask')
  async handlePrintTask(job: Job<{ message: string }>) {
    console.log(`Processing task: ${job.data.message}`);
  }
}

@Module({
  imports: [
    BullModule.forRoot({
      redis: {
        host: 'localhost',
        port: 6379
      }
    }),
    BullModule.registerQueue({ name: 'taskQueue' }),
  ],
  providers: [TaskConsumer, JobProducer],
})
export class AppModule {}

@Injectable()
class JobProducer {
  constructor(@InjectQueue('taskQueue') private taskQueue: Queue) {}

  async addJob() {
    await this.taskQueue.add('printTask', { message: 'Hello from queue!' });
    console.log('Job added to queue');
  }
}

async function bootstrap() {
  const app = await NestFactory.createApplicationContext(AppModule);
  const producer = app.get(JobProducer);
  await producer.addJob();
  setTimeout(() => app.close(), 2000);
}

bootstrap();

Important Notes

Processing a job usually takes O(1) time per job, but depends on job complexity.

Memory use depends on job data size and number of concurrent jobs.

Common mistake: forgetting to register the queue or consumer in the module.

Use queue consumers when tasks can be done later without blocking user actions.

Summary

Queue consumers listen to queues and process jobs asynchronously.

Use @Processor to mark a consumer class and @Process to mark job handlers.

This helps keep your app responsive by handling tasks in the background.

Practice

1. What is the main purpose of a queue consumer (processor) in a NestJS application?

easy

A. To send HTTP requests to external APIs

B. To render UI components on the client side

C. To listen to a queue and process jobs asynchronously in the background

D. To manage database migrations synchronously

Solution

1. Step 1: Understand the role of queue consumers

   Queue consumers listen to queues and handle jobs asynchronously, allowing background processing.

2. Step 2: Compare with other options

   Sending HTTP requests, rendering UI, and managing migrations are unrelated to queue consumers.

3. Final Answer:

   To listen to a queue and process jobs asynchronously in the background -> Option C

4. Quick Check:

   Queue consumer = background job handler [OK]

Hint: Queue consumers handle background jobs, not UI or HTTP [OK]

Common Mistakes:

• Confusing queue consumers with HTTP controllers
• Thinking queue consumers run synchronously
• Assuming queue consumers manage UI rendering

2. Which decorator is used to mark a class as a queue consumer (processor) in NestJS?

easy

A. @Processor

B. @Queue

C. @Process

D. @Consumer

Solution

1. Step 1: Identify the decorator for queue consumers

   The @Processor decorator marks a class as a queue consumer in NestJS.

2. Step 2: Differentiate from other decorators

   @Process marks methods, @Queue and @Consumer are not valid NestJS decorators for this purpose.

3. Final Answer:

   @Processor -> Option A

4. Quick Check:

   Class decorator for consumers = @Processor [OK]

Hint: Class decorator for consumers is @Processor, method decorator is @Process [OK]

Common Mistakes:

• Using @Process on the class instead of methods
• Confusing @Processor with @Queue
• Assuming @Consumer is a valid decorator

3. Given the following NestJS queue consumer code, what will be logged when a job with data { name: 'Alice' } is processed?

import { Processor, Process } from '@nestjs/bull';
import { Job } from 'bull';

@Processor('email')
export class EmailProcessor {
  @Process('send')
  async handleSend(job: Job) {
    console.log(`Sending email to ${job.data.name}`);
  }
}

medium

A. No output because handleSend is not called

B. Sending email to Alice

C. Error: job.data is not defined

D. Sending email to undefined

Solution

1. Step 1: Understand job data usage

   The job object contains data with a name property 'Alice'. The method logs this value.

2. Step 2: Confirm method is called on 'send' jobs

   The @Process('send') decorator ensures handleSend runs for jobs named 'send' in the 'email' queue.

3. Final Answer:

   Sending email to Alice -> Option B

4. Quick Check:

   job.data.name = 'Alice' logged correctly [OK]

Hint: Job data is accessed via job.data inside @Process methods [OK]

Common Mistakes:

• Assuming job.data is undefined
• Thinking method won't run without explicit call
• Confusing queue name with job name

4. Identify the error in this NestJS queue consumer code snippet:

import { Processor, Process } from '@nestjs/bull';
import { Job } from 'bull';

@Processor('notifications')
export class NotificationProcessor {
  @Process()
  async handleNotification(job: Job) {
    console.log('Notification:', job.data.message);
  }
}

medium

A. Job parameter type should be any, not Job

B. The class should not have @Processor decorator

C. Console.log syntax is incorrect

D. @Process decorator is missing the job name argument

Solution

1. Step 1: Check @Process decorator usage

   @Process requires a job name string to specify which jobs to handle; it's missing here.

2. Step 2: Validate other parts

   @Processor on class is correct, Job type is valid, console.log syntax is correct.

3. Final Answer:

   @Process decorator is missing the job name argument -> Option D

4. Quick Check:

   @Process needs job name argument [OK]

Hint: @Process must specify job name string inside parentheses [OK]

Common Mistakes:

• Omitting job name in @Process decorator
• Removing @Processor from class
• Changing Job type unnecessarily

5. You want to create a NestJS queue consumer that processes two different job types: 'email' and 'sms' in the same queue 'notifications'. Which is the correct way to define the processor class and methods?

hard

A. Use @Processor('notifications') on class, and two methods with @Process('email') and @Process('sms') decorators

B. Use two separate classes each with @Processor('notifications') and one @Process method

C. Use @Processor('email') and @Processor('sms') on the same class with one @Process method

D. Use @Process('notifications') on methods without @Processor on class

Solution

1. Step 1: Understand queue and job type separation

   The queue is 'notifications'. The job types are 'email' and 'sms'. One class can handle multiple job types with separate @Process decorators.

2. Step 2: Validate correct decorator usage

   @Processor('notifications') marks the queue. Methods use @Process('email') and @Process('sms') to handle each job type.

3. Final Answer:

   Use @Processor('notifications') on class, and two methods with @Process('email') and @Process('sms') decorators -> Option A

4. Quick Check:

   One class, multiple @Process methods for job types [OK]

Hint: One @Processor class can have many @Process methods for different jobs [OK]

Common Mistakes:

• Creating multiple classes for same queue unnecessarily
• Using multiple @Processor decorators on one class
• Omitting @Processor on class and only using @Process