The multiprocessing module runs Python code in separate operating-system processes. Each process has its own interpreter and memory space, which makes it possible to use several CPU cores for CPU-bound work. The module also provides queues, pipes, locks, shared memory, pools, managers, and synchronization primitives with an interface inspired by threading.
Processes are not simply faster threads. They cost more to start, use additional memory, and require serialization when objects cross process boundaries. Connections, sockets, files, locks, and external resources need explicit ownership rules. A robust design starts by choosing an appropriate start method and reducing communication among workers.
Protect the program entry point
On platforms that start workers by importing the main module, process creation must be guarded.
from multiprocessing import Process
def work(value):
print(value * 2)
if __name__ == "__main__":
process = Process(target=work, args=(21,))
process.start()
process.join()
Without the guard, each imported child may attempt to create more children, leading to recursion, startup errors, or a process storm.
Process, start, and join
Process represents a child. start() requests creation, join() waits, and exitcode reports how the child ended.
process.start()
process.join(timeout=10)
if process.is_alive():
process.terminate()
process.join()
print(process.exitcode)
A join() timeout does not kill the process. The coordinator must decide whether to keep waiting, request cooperative cancellation, or use force as a last resort.
Start methods
The main methods are spawn, fork, and forkserver, with availability and defaults varying by platform. spawn launches a fresh interpreter, fork copies the current process, and forkserver asks a dedicated server to create children.
Select the method explicitly when it is part of the architecture.
import multiprocessing as mp
if __name__ == "__main__":
context = mp.get_context("spawn")
process = context.Process(target=work, args=(10,))
process.start()
process.join()
Fork hazards
fork inherits memory, descriptors, and library state. In a multithreaded program, locks held by threads that disappear can remain locked in the child. Database clients, TLS libraries, loggers, and native runtimes may also become inconsistent.
See Python posix for the Unix process layer. In complex applications, spawn is often more predictable despite higher startup cost.
Serialization
Arguments sent to workers, queue messages, and pool results are commonly serialized. Worker functions should usually be defined at module level. Lambdas, closures, active generators, locks, and open connections may not be transferable.
Never deserialize multiprocessing payloads received from an untrusted source. Python object serialization can execute code during reconstruction.
Queue
Queue offers process-safe message passing with an interface similar to a thread queue.
from multiprocessing import Process, Queue
def producer(queue):
for number in range(5):
queue.put(number * number)
queue.put(None)
if __name__ == "__main__":
queue = Queue(maxsize=10)
process = Process(target=producer, args=(queue,))
process.start()
while True:
item = queue.get()
if item is None:
break
print(item)
process.join()
A bounded queue applies backpressure. Define an explicit completion protocol using a sentinel, event, close operation, or message type.
Avoid join-and-queue deadlocks
A process that has written a large amount to a queue may wait for background buffers to be consumed before it can exit. If the parent calls join() before reading, both sides can block.
Consume messages while the worker is active and follow the documented close and join order for queues.
Pipe
Pipe() creates two connection objects and is useful for point-to-point communication.
from multiprocessing import Pipe, Process
def child(connection):
connection.send({"status": "ok"})
connection.close()
if __name__ == "__main__":
parent, child_end = Pipe(duplex=False)
process = Process(target=child, args=(child_end,))
process.start()
child_end.close()
print(parent.recv())
process.join()
Close unused ends. Extra descriptors kept open can prevent EOF detection.
Pool
Pool keeps a fixed set of workers for many tasks.
from multiprocessing import Pool
def cube(number):
return number ** 3
if __name__ == "__main__":
with Pool(processes=4) as pool:
print(pool.map(cube, range(10)))
For a smaller uniform API across threads and processes, see Python concurrent.futures.
map, imap, and unordered results
map() collects ordered results and may retain substantial data. imap() streams ordered results gradually, while imap_unordered() yields completed items without preserving input order.
Use chunksize to group inputs. Tiny chunks increase scheduling overhead; huge chunks reduce load balancing and delay early results.
apply_async and callbacks
apply_async() returns an asynchronous result and can receive success and error callbacks.
with Pool(4) as pool:
result = pool.apply_async(cube, (5,))
print(result.get(timeout=5))
Callbacks run in the coordinating process and should be short. Collect errors explicitly rather than allowing failed tasks to disappear.
Close pools correctly
close() prevents new work, join() waits for workers, and terminate() stops them abruptly. A context manager simplifies cleanup, but understand its behavior when the block exits because of an exception.
Do not rely on garbage collection to finalize global pools.
Events and cooperative cancellation
A shared Event lets the coordinator request a stop.
from multiprocessing import Event
def worker(stop_event):
while not stop_event.is_set():
execute_chunk()
Workers must check the signal at reasonable intervals and use timeouts in blocking operations.
terminate and kill
Forced termination can leave acquired locks, corrupted queues, incomplete files, and open external transactions. Use it only after cooperative shutdown has failed and with a recovery plan.
Never terminate a worker in the middle of a critical update unless the storage operation is transactional or recoverable.
Locks and semaphores
Lock, RLock, Semaphore, BoundedSemaphore, Condition, and Barrier coordinate processes. Choose the simplest primitive and keep critical sections short.
from multiprocessing import Lock, Value
lock = Lock()
counter = Value("i", 0)
with lock:
counter.value += 1
Using the lock as a context manager guarantees release when an exception occurs.
Value and Array
Value and Array place C-compatible values in shared memory. Access can be synchronized automatically or explicitly.
They are useful for small state, but complex mutable structures become difficult to reason about. Immutable messages through a queue are often safer.
shared_memory
The shared-memory API allows processes to access a byte block without copying large arrays through serialization. Every participant must agree on layout, element type, shape, size, and lifetime.
The creator must close and unlink the block at the correct time. Crashes can leave orphaned resources that require cleanup.
Managers
A Manager starts a server process that exposes proxy objects such as lists, dictionaries, locks, and namespaces. It is convenient, but every proxy operation requires communication and serialization.
Do not use a managed dictionary in a high-frequency inner loop. Batch changes or redesign the flow around messages.
Share less
The most reliable architecture usually sends a self-contained task to a worker and receives a self-contained result. Shared mutable state increases coupling, synchronization, and deadlock risk.
Partition data so each process owns a region whenever possible.
Worker initialization
Pools can call an initializer inside every worker. This is the right place to open one database connection or load read-only resources per process. Connections created by the parent should not be reused blindly by children.
connection = None
def initialize_worker():
global connection
connection = open_connection()
Close resources at worker exit and never write secrets to logs.
Memory and copy-on-write
On systems using fork, pages may initially be shared until a process modifies them. This can make startup efficient, but writes increase memory. Python object management and garbage collection may touch pages unexpectedly.
Measure the total resident memory of every process. Python resource explains several Unix limits and measurements.
Exception handling
A worker exception must be observed by the parent. Pools re-raise failures when the caller invokes get() or consumes results. A program that never collects a result can miss the error.
Attach a task identifier and preserve the traceback for diagnostics.
Logging
Several processes writing directly to one file can interleave records. Use a QueueHandler and a central listener process or thread.
Include PID, process name, task ID, attempt, and duration.
Signals and shutdown
Make the coordinator responsible for operating-system signals and distribute cancellation to workers. Signal handlers should remain minimal. See Python signal.
During shutdown, stop producing work, send sentinels, wait for a deadline, terminate only remaining workers, and close queues and shared memory.
Frozen executables
Packaged applications may require freeze_support(). Test the actual executable, not only the development script.
Security
Do not expose managers or multiprocessing connections over a network without authentication and isolation. Never accept serialized object payloads from untrusted clients.
Cap process counts, memory, queue depth, and task sizes to prevent local resource exhaustion.
Testing
Test different start methods, small and large pools, exceptions, worker crashes, full queues, Ctrl+C, shutdown, stuck tasks, large data, systems without fork, and packaged executables.
Avoid exact sleep-based timing assertions. Coordinate with events and generous deadlines.
Common mistakes
Common failures include missing the main-module guard, submitting lambdas, sharing inherited connections, calling join() before consuming queues, using managers for everything, terminating workers during writes, ignoring task exceptions, creating too many workers, and leaking shared-memory blocks.
Conclusion
multiprocessing enables real parallelism and rich process communication, but it requires discipline around serialization, lifecycle, ownership, and cleanup. Prefer independent tasks, limited communication, cooperative cancellation, and controlled pool sizes.
Consult the official multiprocessing documentation and compare it with Python concurrent.futures for a more compact task API.
