The concurrent.futures module provides a high-level interface for running functions concurrently. Instead of creating and coordinating threads or processes manually, an application submits calls to an executor and receives Future objects that represent results that may not be ready yet. The same programming model works with thread-based and process-based executors.
The API simplifies queues, result collection, exception propagation, timeouts, cancellation, and shutdown. It does not remove the hard parts of concurrency: you still need to choose the correct executor, limit pending work, avoid deadlocks, protect shared state, and decide how failures affect the rest of the operation.
Executor and Future
An Executor accepts work. submit() returns a Future. The future lets the caller inspect state, wait, retrieve a value, observe an exception, or cancel work that has not started.
from concurrent.futures import ThreadPoolExecutor
def square(number):
return number * number
with ThreadPoolExecutor(max_workers=4) as executor:
future = executor.submit(square, 12)
print(future.result())
The with statement calls shutdown() when the block ends and applies the executor’s normal waiting policy. This prevents abandoned worker pools.
ThreadPoolExecutor
ThreadPoolExecutor runs threads inside one process. It is commonly suitable for operations that spend substantial time waiting for I/O: HTTP requests, files, databases, subprocesses, message brokers, and external APIs.
Threads share memory, so passing objects is easy, but mutable shared state creates race conditions. Use queues, locks, immutable data, or ownership rules rather than allowing every worker to modify the same structures.
CPU-bound Python work
For functions that continuously execute Python bytecode, multiple threads usually do not provide full multicore parallelism on traditional interpreter builds. A process pool can be a better fit.
Native extensions that release the interpreter lock may scale with threads. Measure the actual workload rather than applying a universal rule.
ProcessPoolExecutor
ProcessPoolExecutor uses separate processes and can execute CPU work on multiple cores. Arguments and results must be serializable, and submitted functions must be importable by workers.
from concurrent.futures import ProcessPoolExecutor
def calculate(number):
return sum(i * i for i in range(number))
if __name__ == "__main__":
with ProcessPoolExecutor() as executor:
futures = [executor.submit(calculate, n) for n in (10000, 20000, 30000)]
print([future.result() for future in futures])
The if __name__ == "__main__" guard is essential on platforms that start workers by importing the main module again.
Choose worker counts deliberately
More workers do not automatically mean more throughput. Too many threads increase context switching, open connections, and pressure on external services. Too many processes increase memory, serialization, startup cost, and CPU contention.
Base the limit on the most constrained resource: CPU cores, database pool size, rate limits, bandwidth, memory, file descriptors, or downstream capacity.
Submit individual tasks
submit(function, *args, **kwargs) provides control over each task.
with ThreadPoolExecutor(max_workers=8) as executor:
futures = {
executor.submit(download, url, timeout=10): url
for url in urls
}
Mapping futures back to inputs makes failures easier to diagnose and retry selectively.
Use map for ordered transformations
executor.map() applies a function to several inputs and returns results in input order.
with ThreadPoolExecutor(max_workers=4) as executor:
results = list(executor.map(process, items))
Ordered delivery is convenient, but one slow task can delay later completed results. Use as_completed() when completion order is preferable.
Consume results with as_completed
as_completed() yields futures as they finish.
from concurrent.futures import as_completed
for future in as_completed(futures):
source = futures[future]
try:
result = future.result()
except Exception as error:
record_failure(source, error)
else:
save(result)
An exception raised inside the worker is re-raised by result() in the collecting thread or process, preserving a normal error-handling flow.
Wait for groups
wait() can wait for all tasks, the first completion, or the first exception. It returns sets of completed and pending futures.
from concurrent.futures import wait, FIRST_EXCEPTION
done, pending = wait(futures, return_when=FIRST_EXCEPTION)
After a failure, explicitly decide whether pending work should continue, be cancelled, or finish to preserve consistency.
Timeouts
future.result(timeout=...), wait(), and as_completed() can limit how long the caller waits. A waiting timeout does not automatically stop the function that is already running.
from concurrent.futures import TimeoutError
try:
value = future.result(timeout=5)
except TimeoutError:
log_warning("the task exceeded the caller's wait limit")
The task itself must use timeouts for sockets, databases, subprocesses, and external APIs so it can eventually finish.
Cancellation
cancel() succeeds only before a task starts. cancelled() reports successful cancellation, and running() reports that execution has begun.
Stopping work that is already running requires cooperation through an Event, deadline, cancellation token, or another shared signal.
def process(items, stop_event):
for item in items:
if stop_event.is_set():
return None
execute_step(item)
Shutdown
shutdown(wait=True) closes the executor and normally waits for submitted tasks. Cancelling pending futures can be useful during failure or application shutdown.
Do not leave the lifetime of a global executor undefined. Establish who creates it, who closes it, and what happens to incomplete operations.
Thread-pool deadlocks
A deadlock can occur when one task waits for another task from the same pool while every worker is already occupied.
def task_a():
return future_b.result()
Avoid blocking dependencies among tasks in the same executor. Coordinate sequences outside the pool, use callbacks, or combine results in the submitting thread.
The single-worker trap
With one worker, a task that submits another task to the same executor and calls result() can never release the worker needed to execute the second call.
Increasing the worker count only hides the structural problem. Remove the dependency cycle.
Backpressure
Submitting millions of tasks at once consumes memory for arguments, futures, and internal queues. Produce work in batches or maintain a bounded window of pending tasks.
def run_batches(executor, items, size=100):
batch = []
for item in items:
batch.append(executor.submit(process, item))
if len(batch) == size:
for future in batch:
yield future.result()
batch.clear()
A sliding window can submit one replacement whenever a future completes, keeping memory and downstream load stable.
Worker initialization
Executors may configure each worker through an initializer. If initialization fails, the pool can become unusable and pending futures may receive an executor-broken error.
Keep initialization short, deterministic, and independent of fragile network services whenever possible.
Global state and processes
Processes do not share normal Python objects. Each worker has its own imported modules and globals. A change to a worker’s global variable does not automatically return to the parent.
Pass inputs explicitly and return outputs. For the operating-system process layer, see Python posix and use multiprocessing abstractions for richer communication.
Serialization
Local functions, lambdas, open generators, locks, sockets, and active database connections generally cannot be submitted to a process pool. Define worker functions at module level and pass simple data.
Large objects also increase copying cost. It can be more efficient for each worker to open its own resource from an identifier.
Completion callbacks
add_done_callback() registers a function that runs after completion.
def on_done(future):
try:
future.result()
except Exception:
metrics.failure()
else:
metrics.success()
future.add_done_callback(on_done)
Callbacks should remain short and nonblocking. Publish an event to a queue for larger follow-up work.
Context and logging
Threads share process logging, but request context may not propagate automatically. Processes need their own logging configuration or a queue-based logging design.
Include a task identifier so logs can connect input, attempts, duration, worker, and exception.
Retries
Executors do not retry failed calls automatically. Retry only failures that are likely to be transient, with a limit, delay, jitter, and an idempotent operation.
Validation errors do not improve with repetition. A partially completed write may create duplicates when no idempotency key is used.
Security
Never submit functions or serialized payloads supplied by untrusted users to a process pool. Python serialization is not a safe interchange format for hostile input.
Limit concurrency so the application cannot create a denial of service against its own database, filesystem, network, or third-party API.
Testing
Test success, worker exceptions, timeout, cancellation before start, shutdown, broken pools, unordered completion, empty input, very slow tasks, and process interruption. Test with a small worker count to expose hidden dependencies.
Avoid assertions based on exact timing. Synchronize tests with events and observable state.
Common mistakes
Common failures include using threads for CPU work without measuring, creating process pools without the main-module guard, calling result() from tasks in the same pool, submitting unlimited work, assuming a timeout kills execution, ignoring future exceptions, passing nonserializable objects, and forgetting to close the executor.
Conclusion
concurrent.futures provides a uniform task-oriented layer for concurrency. Use threads for waiting-heavy I/O, processes for CPU work when appropriate, cap workers, apply backpressure, and treat every future as an operation that can fail, time out, or be cancelled.
Consult the official concurrent.futures documentation and Python signal for coordinated shutdown.
