How to Use Multiprocessing Module in Python for Parallel Tasks

Use the multiprocessing module in Python to run functions in parallel by creating Process objects and starting them with start(). You can also use Pool to manage multiple worker processes easily for tasks like mapping functions over data.

Syntax

The multiprocessing module lets you create separate processes to run code in parallel. The main parts are:

• Process(target=function, args=(args,)): Creates a new process to run function with given arguments.
• start(): Starts the process.
• join(): Waits for the process to finish.
• Pool(processes=n): Creates a pool of worker processes to run tasks concurrently.
• map(function, iterable): Runs function on each item in iterable using the pool.

python

from multiprocessing import Process, Pool

def worker(num):
    print(f'Worker {num} is running')

if __name__ == '__main__':
    # Using Process
    p = Process(target=worker, args=(1,))
    p.start()
    p.join()

    # Using Pool
    with Pool(processes=2) as pool:
        pool.map(worker, [2, 3])

Output

Worker 1 is running
Worker 2 is running
Worker 3 is running

Example

This example shows how to run a function in parallel using Process and Pool. It prints messages from different worker processes.

python

from multiprocessing import Process, Pool
import time

def worker(num):
    print(f'Worker {num} started')
    time.sleep(1)
    print(f'Worker {num} finished')

if __name__ == '__main__':
    # Run a single process
    p = Process(target=worker, args=(1,))
    p.start()
    p.join()

    # Run multiple processes with Pool
    with Pool(processes=3) as pool:
        pool.map(worker, [2, 3, 4])

Output

Worker 1 started
Worker 1 finished
Worker 2 started
Worker 3 started
Worker 4 started
Worker 2 finished
Worker 3 finished
Worker 4 finished

Common Pitfalls

Common mistakes when using multiprocessing include:

• Not protecting the entry point with if __name__ == '__main__': which causes infinite process spawning on Windows.
• Trying to share state between processes without using special shared objects or queues.
• Forgetting to call join() to wait for processes to finish.

Always use the main guard and communicate between processes properly.

python

from multiprocessing import Process

def worker():
    print('Worker running')

# Wrong: Missing main guard
# p = Process(target=worker)
# p.start()
# p.join()

# Right:
if __name__ == '__main__':
    p = Process(target=worker)
    p.start()
    p.join()

Quick Reference

Feature	Description
Process(target, args)	Create a new process to run a function with arguments
start()	Start the process
join()	Wait for the process to finish
Pool(processes=n)	Create a pool of worker processes
map(function, iterable)	Run function on each item in iterable using the pool
if __name__ == '__main__':	Protect entry point to avoid recursive process spawning

Key Takeaways

Use Process to run functions in separate processes for parallelism.

Always protect your code with if __name__ == '__main__': when using multiprocessing.

Use Pool to manage multiple worker processes easily for batch tasks.

Call start() to begin a process and join() to wait for it to finish.

Processes do not share memory; use special objects or communication methods to share data.