How to Share Data Between Processes in Python Easily

In Python, you can share data between processes using the multiprocessing module's shared objects like Value and Array, or use Queue and Manager for more complex data. These tools let separate processes communicate safely by sharing memory or passing messages.

Syntax

The multiprocessing module provides several ways to share data between processes:

• Shared memory objects: Value(typecode, initial_value) and Array(typecode, size_or_initializer) create shared memory for simple data.
• Queue: multiprocessing.Queue() allows sending data safely between processes.
• Manager: multiprocessing.Manager() creates a server process that holds Python objects and allows other processes to manipulate them.

Each method helps processes communicate without conflicts.

python

from multiprocessing import Value, Array, Queue, Manager

# Shared value example
shared_val = Value('i', 0)  # 'i' means integer

# Shared array example
shared_arr = Array('d', 5)  # 'd' means double (float), size 5

# Queue example
q = Queue()

# Manager example
manager = Manager()
shared_dict = manager.dict()

Example

This example shows two processes sharing a number using Value and communicating via Queue. One process increments the shared number and sends messages through the queue.

python

from multiprocessing import Process, Value, Queue
import time

def worker(num, q):
    for i in range(5):
        with num.get_lock():  # Lock to safely update shared value
            num.value += 1
        q.put(f"Incremented to {num.value}")
        time.sleep(0.1)

if __name__ == '__main__':
    shared_num = Value('i', 0)
    q = Queue()

    p = Process(target=worker, args=(shared_num, q))
    p.start()

    for _ in range(5):
        msg = q.get()  # Receive messages from worker
        print(msg)

    p.join()
    print(f"Final shared number: {shared_num.value}")

Output

Incremented to 1
Incremented to 2
Incremented to 3
Incremented to 4
Incremented to 5
Final shared number: 5

Common Pitfalls

Common mistakes when sharing data between processes include:

• Not using locks with shared memory objects, causing race conditions.
• Trying to share normal Python objects without using Manager, which leads to errors.
• Assuming global variables are shared across processes—they are not.
• Using Queue without proper synchronization, which can cause deadlocks if not handled carefully.

Always use the provided synchronization tools and share data explicitly.

python

from multiprocessing import Process, Value

def unsafe_worker(num):
    num.value += 1  # No lock, unsafe update

if __name__ == '__main__':
    shared_num = Value('i', 0)
    processes = [Process(target=unsafe_worker, args=(shared_num,)) for _ in range(100)]

    for p in processes:
        p.start()
    for p in processes:
        p.join()

    print(f"Unsafe final value: {shared_num.value}")  # May be less than 100 due to race conditions

# Correct way with lock
from multiprocessing import Lock

def safe_worker(num, lock):
    with lock:
        num.value += 1

if __name__ == '__main__':
    shared_num = Value('i', 0)
    lock = Lock()
    processes = [Process(target=safe_worker, args=(shared_num, lock)) for _ in range(100)]

    for p in processes:
        p.start()
    for p in processes:
        p.join()

    print(f"Safe final value: {shared_num.value}")  # Should be 100

Output

Unsafe final value: 87
Safe final value: 100

Quick Reference

Summary tips for sharing data between processes in Python:

• Use Value and Array for simple shared memory with primitive types.
• Use Queue to send messages or data safely between processes.
• Use Manager to share complex Python objects like lists or dictionaries.
• Always use locks or synchronization when modifying shared memory to avoid race conditions.
• Remember that global variables are not shared between processes.

Key Takeaways

Use multiprocessing shared objects like Value and Array for simple shared data.

Use Queue or Manager for safe communication and sharing complex objects between processes.

Always protect shared memory updates with locks to avoid race conditions.

Global variables are not shared across processes; explicit sharing is required.

Choose the right sharing method based on data complexity and communication needs.