The select module waits until sockets, pipes, and other file descriptors are ready for reading, writing, or exceptional conditions. This technique, called I/O multiplexing, lets one thread coordinate many connections without blocking on only one of them.
Different primitives are available by operating system. select() is widely available, poll() exists on Unix, epoll() is Linux-specific, kqueue() is available on BSD and macOS, and /dev/poll belongs to Solaris. For portable applications, selectors is normally preferred because it chooses an efficient backend automatically.
Readiness is not completion
A readable socket has some condition to process: data, EOF, or an error. It does not guarantee that a complete application message has arrived. A writable socket can accept at least some output, but it may accept only part of a pending buffer.
Protocols need explicit input and output buffers. Never assume one recv() returns one message or that one send() transmits every byte.
A basic select() server
import select
import socket
server = socket.socket()
server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
server.bind(("127.0.0.1", 9000))
server.listen()
server.setblocking(False)
readers = [server]
while True:
readable, _, exceptional = select.select(readers, [], readers, 1.0)
for sock in readable:
if sock is server:
client, address = server.accept()
client.setblocking(False)
readers.append(client)
else:
data = sock.recv(4096)
if data:
print(data)
else:
readers.remove(sock)
sock.close()
for sock in exceptional:
if sock in readers:
readers.remove(sock)
sock.close()
This is only a teaching example. A production server requires framing, bounded buffers, output queues, authentication, timeouts, and robust exception handling. See Python socketserver for a higher-level server framework.
The three select lists
The first iterable contains objects watched for readability. The second contains objects watched for writability. The third represents platform-defined exceptional conditions. Each entry can be an integer file descriptor or an object with a parameterless fileno() method.
The return value contains three ready subsets. A timeout of None waits indefinitely, zero performs a non-blocking poll, and a positive float limits the wait in seconds.
Windows differences
On Windows, select() works only with WinSock sockets. Regular files, standard input, and ordinary pipe handles cannot be monitored the same way as on Unix. Portable programs may need threads, asyncio, or Windows-specific APIs for non-socket sources.
The module is unavailable on WASI, so WebAssembly environments require different abstractions.
Why selectors is usually preferable
selectors.DefaultSelector chooses epoll, kqueue, poll, or select according to the platform. Its registration model can attach application data to every file object and hides many bitmask differences.
Use select directly for kernel-specific flags, edge-triggered behavior, kqueue process and vnode events, or integration with existing low-level code.
Partial reads and message framing
TCP provides a byte stream rather than message boundaries. Applications can define a delimiter, length prefix, fixed record size, or structured framing format.
buffers = {}
def receive(sock):
chunk = sock.recv(4096)
if not chunk:
return False
buffer = buffers.setdefault(sock, bytearray())
buffer.extend(chunk)
while b"\n" in buffer:
line, _, remaining = buffer.partition(b"\n")
buffer[:] = remaining
process_line(line)
return True
Set a maximum buffer size. A peer that never sends a delimiter must not consume memory without limit.
Partial writes and backpressure
Monitor a socket for writing only while output is pending. Most connected sockets are writable much of the time, so registering every socket permanently can create a busy loop.
outputs = {}
def send_pending(sock):
buffer = outputs.get(sock)
if not buffer:
return False
sent = sock.send(buffer)
del buffer[:sent]
return bool(buffer)
Bound each output queue. If a client reads slowly, pause producers, reject more data, drop messages according to policy, or close the connection. This pressure control prevents one client from exhausting server memory.
EOF, half-close, and errors
recv() returning b"" means the peer closed its sending side. The application may still have output to send, depending on the protocol. poll() and epoll() expose flags such as POLLHUP, EPOLLHUP, and EPOLLRDHUP.
Handle errors and hangups even when readability is also reported. Final buffered data may remain before the connection fully closes.
poll()
poll() registers descriptors with masks such as POLLIN, POLLOUT, POLLERR, POLLHUP, and POLLNVAL.
import select
poller = select.poll()
poller.register(sock, select.POLLIN)
for fd, event in poller.poll(1000):
if event & (select.POLLERR | select.POLLHUP | select.POLLNVAL):
close_fd(fd)
elif event & select.POLLIN:
read_fd(fd)
The timeout is measured in milliseconds. Maintain a mapping from descriptor numbers to connection objects and their state.
epoll on Linux
epoll() scales well for large descriptor sets and supports level-triggered and edge-triggered operation. Level-triggered mode continues reporting an event while the condition remains true. With EPOLLET, notifications describe state changes.
Edge-triggered code must use non-blocking sockets and read or write until EAGAIN or EWOULDBLOCK. Stopping early can leave unread data without another notification.
import errno
import select
with select.epoll() as ep:
ep.register(sock.fileno(), select.EPOLLIN | select.EPOLLET)
for fd, event in ep.poll(1.0):
if event & select.EPOLLIN:
while True:
try:
chunk = descriptors[fd].recv(65536)
except BlockingIOError as exc:
if exc.errno in {errno.EAGAIN, errno.EWOULDBLOCK}:
break
raise
if not chunk:
close_fd(fd)
break
process(chunk)
Python errno explains these non-blocking error codes.
EPOLLONESHOT and EPOLLEXCLUSIVE
EPOLLONESHOT disables a descriptor after one reported event until the program rearms it with modify(). This can prevent several workers from handling the same connection simultaneously. EPOLLEXCLUSIVE can reduce the thundering-herd problem when several epoll instances watch the same descriptor.
These options are Linux-specific and require tests against the real kernel versions used in production.
kqueue
kqueue() can monitor I/O, processes, signals, timers, and filesystem vnode events. Changes are represented by kevent objects. Flags such as KQ_EV_ADD, KQ_EV_DELETE, KQ_EV_ONESHOT, and KQ_EV_CLEAR modify registration behavior.
BSD and macOS can report file rename, deletion, write, and attribute changes through vnode filters. This mechanism is not portable to Linux or Windows.
Integrate signals with a wakeup descriptor
The previous guide, Python signal, introduced set_wakeup_fd(). A non-blocking pipe or socketpair can wake the selector when SIGTERM or SIGINT arrives.
import os
import select
import signal
read_fd, write_fd = os.pipe()
os.set_blocking(read_fd, False)
os.set_blocking(write_fd, False)
signal.set_wakeup_fd(write_fd)
signal.signal(signal.SIGTERM, lambda signum, frame: None)
while True:
readable, _, _ = select.select([read_fd, server], [], [], None)
if read_fd in readable:
os.read(read_fd, 4096)
break
Drain the descriptor, restore the previous wakeup configuration, and close both ends during cleanup. The signal handler remains minimal while normal loop code performs shutdown.
Timeouts and monotonic time
Selector timeouts can drive idle expiration, periodic maintenance, and shutdown deadlines. Calculate deadlines with a monotonic clock rather than wall-clock time.
import time
deadline = time.monotonic() + 30
while True:
remaining = max(0, deadline - time.monotonic())
readable, _, _ = select.select(readers, [], [], remaining)
if time.monotonic() >= deadline:
expire_connections()
Closed and reused descriptors
The operating system can reuse descriptor numbers. Unregister a descriptor before closing it and remove all associated application state. A stale event for a reused number must not be applied to a new connection.
On current Python versions, epoll unregister() can expose EBADF after premature closure. Centralize cleanup order and make it idempotent.
PIPE_BUF
On Unix, select.PIPE_BUF gives the minimum number of bytes that POSIX guarantees can be written atomically to a pipe after it has been reported writable. This guarantee does not apply to sockets and does not mean arbitrary output sizes will be accepted.
Fairness
A constantly active descriptor can monopolize the loop. Limit bytes or messages processed per event and return to the multiplexer so other connections can advance.
Move CPU-heavy work away from the I/O loop into workers, processes, or an executor. The selector thread must stay responsive.
Security and resource limits
Limit simultaneous connections, bytes per client, frame size, output queue length, idle time, request rate, and CPU work. Supporting many sockets does not create unlimited memory or processor capacity.
Non-blocking TLS requires careful state handling because handshakes and reads may request either readability or writability. See Python ssl.
Recommended tests
Test fragmented input, partial output, slow readers, close-during-send, half-close, timeouts, signals during waits, invalid descriptors, many sockets, backpressure, undrained edge-triggered reads, and Windows-versus-Unix behavior.
Use socket.socketpair() for local tests and small socket buffers to force partial operations. Assert that loops do not spin and memory remains bounded.
Common mistakes
Common failures include treating readiness as a complete message, always monitoring writability, ignoring partial sends, allowing unlimited buffers, using edge-trigger without draining until EAGAIN, closing before unregistering, assuming regular files work on Windows, and performing CPU-heavy work in the I/O loop.
Conclusion
select gives direct access to operating-system I/O multiplexing. It can coordinate many descriptors with few threads, provided the application manages buffers, backpressure, closing, and platform differences.
Prefer selectors for portability, use low-level primitives only when their capabilities are required, and keep every watched descriptor non-blocking. Consult the official select documentation and the official selectors documentation.
