How to Create Process Pool in Python: Simple Guide

To create a process pool in Python, use the multiprocessing.Pool class which manages multiple worker processes. You create a pool by calling Pool(processes=n), where n is the number of worker processes you want to run in parallel.

Syntax

The basic syntax to create a process pool is:

• Pool(processes=None): Creates a pool of worker processes.
• processes: Number of worker processes to use. If None, it uses the number of CPU cores.
• Use pool.map(function, iterable) to apply a function to each item in an iterable in parallel.
• Always close the pool with pool.close() and wait for workers to finish with pool.join().

python

from multiprocessing import Pool

# Create a pool with 4 worker processes
pool = Pool(processes=4)

# Use pool.map to apply a function to a list
results = pool.map(str, [1, 2, 3, 4])

# Close the pool and wait for tasks to complete
pool.close()
pool.join()

Example

This example shows how to create a process pool to square numbers in parallel. It demonstrates creating the pool, running tasks, and collecting results.

python

from multiprocessing import Pool

def square(x):
    return x * x

if __name__ == '__main__':
    with Pool(processes=3) as pool:
        numbers = [1, 2, 3, 4, 5]
        results = pool.map(square, numbers)
        print(results)

Output

[1, 4, 9, 16, 25]

Common Pitfalls

• Not protecting the entry point with if __name__ == '__main__' causes errors on Windows.
• Forgetting to close and join the pool can cause the program to hang.
• Passing non-picklable objects to worker functions will raise errors.
• Using too many processes can slow down due to overhead.

python

from multiprocessing import Pool

def f(x):
    return x * 2

# Wrong: Missing if __name__ == '__main__'
pool = Pool(2)
print(pool.map(f, [1, 2, 3]))
pool.close()
pool.join()

# Right way:
if __name__ == '__main__':
    with Pool(2) as pool:
        print(pool.map(f, [1, 2, 3]))

Quick Reference

Summary tips for using process pools in Python:

• Use Pool(processes=n) to create a pool with n workers.
• Use pool.map() to run a function on many inputs in parallel.
• Always protect your code with if __name__ == '__main__' on Windows.
• Close the pool with pool.close() and wait with pool.join().
• Use with Pool() as pool for automatic cleanup.

Key Takeaways

Use multiprocessing.Pool to create a pool of worker processes for parallel tasks.

Always protect your main code with if __name__ == '__main__' to avoid errors on Windows.

Use pool.map() to apply functions to lists in parallel easily.

Close and join the pool to clean up worker processes properly.

Using 'with Pool() as pool' is a clean way to manage process pools.