Task queue concept in Flask

Introduction

A task queue helps your Flask app do long jobs in the background. This keeps the app fast and responsive for users.

When sending emails after a user signs up without making them wait.

When processing uploaded files like images or videos after upload.

When running reports or data analysis that takes time.

When calling slow external services without blocking the user.

When scheduling tasks to run later or repeatedly.

Syntax

Flask

from flask import Flask
from flask_sqlalchemy import SQLAlchemy
from rq import Queue
from redis import Redis

app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///app.db'
db = SQLAlchemy(app)

redis_connection = Redis()
task_queue = Queue('default', connection=redis_connection)

def background_task(n):
    """Example task that runs in the background."""
    print(f"Working on task {n}")
    return f"Task {n} done"

You need Redis installed and running for the queue to work.

Use rq library to create and manage task queues in Flask.

Examples

This adds a task to the queue to run background_task(5) later.

Flask

from rq import Queue
from redis import Redis

redis_connection = Redis()
task_queue = Queue('default', connection=redis_connection)

# Add a task to the queue
job = task_queue.enqueue(background_task, 5)
print(job.get_id())

Check if Redis is running before using the queue to avoid errors.

Flask

# What if Redis is not running?
try:
    redis_connection = Redis()
    redis_connection.ping()
except Exception as error:
    print("Redis server is not available.")

Handle the case when no tasks are waiting in the queue.

Flask

# What if the task queue is empty?
if task_queue.is_empty():
    print("No tasks to process right now.")

This schedules background_task(10) to run 10 minutes later.

Flask

# What if you want to schedule a task for later?
import datetime
from rq_scheduler import Scheduler

scheduler = Scheduler(connection=redis_connection)
scheduler.enqueue_at(datetime.datetime.utcnow() + datetime.timedelta(minutes=10), background_task, 10)

Sample Program

This Flask app has an endpoint /start-task that starts a background task to sleep for a given number of seconds. The task runs in the background, so the app stays responsive.

Flask

from flask import Flask, request, jsonify
from rq import Queue
from redis import Redis
import time

app = Flask(__name__)
redis_connection = Redis()
task_queue = Queue('default', connection=redis_connection)

def background_task(seconds):
    """Simulate a long task by sleeping."""
    print(f"Starting task for {seconds} seconds")
    time.sleep(seconds)
    print(f"Finished task for {seconds} seconds")
    return f"Slept for {seconds} seconds"

@app.route('/start-task', methods=['POST'])
def start_task():
    data = request.get_json() or {}
    seconds = data.get('seconds', 5)
    job = task_queue.enqueue(background_task, seconds)
    return jsonify({'message': 'Task started', 'job_id': job.get_id()})

if __name__ == '__main__':
    app.run(debug=True)

Important Notes

Task queues let your app handle slow jobs without making users wait.

Time complexity depends on the task itself, but queue operations are usually fast (O(1)).

Common mistake: forgetting to run a worker process that actually executes queued tasks.

Use task queues when you want to do work later or in parallel, not for instant responses.

Summary

Task queues run slow jobs in the background so your Flask app stays fast.

Redis and rq are popular tools to create task queues in Flask.

Always run a worker to process tasks from the queue.

Practice

1. What is the main purpose of using a task queue in a Flask application?

easy

A. To serve static files faster

B. To run slow or long tasks in the background without blocking the main app

C. To handle user authentication securely

D. To speed up the Flask app by caching all responses

Solution

1. Step 1: Understand task queue role

   Task queues let slow jobs run separately from the main app, so the app stays responsive.

2. Step 2: Identify the correct purpose

   Running slow tasks in the background matches the main purpose of task queues.

3. Final Answer:

   To run slow or long tasks in the background without blocking the main app -> Option B

4. Quick Check:

   Task queue purpose = background slow jobs [OK]

Hint: Task queues handle slow jobs outside main app flow [OK]

Common Mistakes:

• Confusing task queues with caching
• Thinking task queues handle authentication
• Assuming task queues speed up static file serving

2. Which of the following is the correct way to enqueue a task using rq in Flask?

easy

A. queue.enqueue(my_task, arg1, arg2)

B. queue.run(my_task, arg1, arg2)

C. queue.start(my_task, arg1, arg2)

D. queue.execute(my_task, arg1, arg2)

Solution

1. Step 1: Recall rq enqueue method

   In rq, tasks are added to the queue using the enqueue() method.

2. Step 2: Match method to options

   Only queue.enqueue(my_task, arg1, arg2) uses enqueue() correctly with task and arguments.

3. Final Answer:

   queue.enqueue(my_task, arg1, arg2) -> Option A

4. Quick Check:

   rq enqueue method = enqueue() [OK]

Hint: Use enqueue() to add tasks to rq queue [OK]

Common Mistakes:

• Using run(), start(), or execute() instead of enqueue()
• Forgetting to pass task arguments
• Calling enqueue on wrong object

3. Given this Flask rq task code snippet:

from rq import Queue
from redis import Redis

redis_conn = Redis()
queue = Queue(connection=redis_conn)

def add(x, y):
    return x + y

job = queue.enqueue(add, 3, 4)
print(job.result)

What will be printed immediately after running this code?

medium

A. None

B. 7

C. Error: job.result not available

D. 3 + 4

Solution

1. Step 1: Understand rq job processing

   When a job is enqueued, it runs asynchronously. The result is not ready immediately.

2. Step 2: Check job.result value right after enqueue

   Immediately after enqueue, job.result is None because the worker hasn't processed it yet.

3. Final Answer:

   None -> Option A

4. Quick Check:

   job.result before worker runs = None [OK]

Hint: job.result is None until worker finishes task [OK]

Common Mistakes:

• Assuming job.result is ready immediately
• Expecting synchronous return value
• Confusing job.result with function return

4. You wrote this code to enqueue a task in Flask with rq:

queue.enqueue(my_task, arg1, arg2)

But the task never runs. What is the most likely reason?

medium

A. Redis server is not installed

B. The task function my_task is missing

C. You forgot to start the rq worker process

D. You used enqueue instead of run

Solution

1. Step 1: Check rq worker role

   rq worker must be running to process tasks from the queue.

2. Step 2: Identify missing worker

   If tasks never run, the worker process is likely not started.

3. Final Answer:

   You forgot to start the rq worker process -> Option C

4. Quick Check:

   Worker must run to process tasks [OK]

Hint: Always run rq worker to process queued tasks [OK]

Common Mistakes:

• Assuming enqueue runs task immediately
• Not starting rq worker process
• Confusing Redis installation with worker running

5. You want to run a slow image processing function in the background using Flask and rq. Which combination correctly sets up the task queue and worker?

hard

A. Install Redis, but only enqueue tasks; worker runs automatically without command

B. Create Flask app, call task function directly; no need for Redis or worker

C. Use Flask's built-in threading to run task; no Redis or rq needed

D. Create Redis connection, create rq Queue with it, enqueue task; run rq worker in terminal

Solution

1. Step 1: Setup Redis and rq Queue

   You must create a Redis connection and pass it to rq Queue to enqueue tasks properly.

2. Step 2: Run rq worker separately

   The rq worker must be started manually in the terminal to process queued tasks.

3. Final Answer:

   Create Redis connection, create rq Queue with it, enqueue task; run rq worker in terminal -> Option D

4. Quick Check:

   Redis + rq Queue + worker process = correct setup [OK]

Hint: Redis + rq Queue + manual worker run = background tasks [OK]

Common Mistakes:

• Calling task function directly instead of enqueueing
• Assuming worker runs automatically
• Using Flask threading instead of task queue