The io module defines Python’s core input and output interfaces. Files opened with open(), in-memory buffers, text wrappers, and many objects returned by sockets, compression modules, and external libraries follow contracts built on this hierarchy.
There are three main categories: text I/O, buffered binary I/O, and raw I/O. Understanding the difference prevents type errors, encoding-related data loss, partial writes, excessive memory usage, and platform-specific surprises.
Streams and file-like objects
A stream is a sequence of data accessed through methods such as read(), write(), seek(), and close(). It can represent a filesystem file, memory, a pipe, a socket, an HTTP response, compressed data, or a custom implementation.
Not every stream supports every operation. Use readable(), writable(), and seekable() to inspect capabilities. Unsupported operations may raise io.UnsupportedOperation.
Text and bytes are separate contracts
Text streams consume and produce str. Binary streams consume bytes-like objects and produce bytes. Mixing the types raises TypeError.
with open("data.txt", "w", encoding="utf-8") as file:
file.write("Hello")
with open("image.png", "wb") as file:
file.write(b"\x89PNG")
Do not use text mode for images, ZIP files, PDFs, or binary protocols. Do not use binary mode for text without an explicit encoding and decoding policy.
Specify the encoding
The default encoding of open() is locale-dependent unless UTF-8 Mode is enabled. Code that works on a UTF-8 Linux environment may fail on Windows if encoding="utf-8" is omitted.
with open("README.md", "r", encoding="utf-8") as file:
content = file.read()
Use encoding="locale" when the current locale is deliberately part of the format. For new application formats, explicit UTF-8 is usually more predictable. See Python codecs for encoding and error-handler details.
EncodingWarning
Python can warn when an API relies on the default locale encoding. Run with -X warn_default_encoding or set PYTHONWARNDEFAULTENCODING. Functions that accept encoding=None can use io.text_encoding() so the warning points to the caller.
import io
def read_text(path, encoding=None):
encoding = io.text_encoding(encoding)
with open(path, encoding=encoding) as file:
return file.read()
Newline translation
The newline argument controls line-ending translation. With None, universal newline mode recognizes \n, \r, and \r\n and returns \n. With an empty string, the endings are recognized but preserved. A specific value restricts line termination.
For formats requiring exact bytes, including signatures, hashes, and wire protocols, use binary mode or configure newline behavior explicitly.
Context managers and closing
IOBase objects support with, ensuring closure even when an exception occurs.
with open("report.txt", "w", encoding="utf-8") as file:
file.write("result\n")
Operations on a closed stream normally raise ValueError. Calling close() more than once is allowed, but the object must not be used afterward.
flush() is not fsync()
flush() pushes Python’s buffered data toward the underlying stream. It does not guarantee physical persistence. When durability is required, flush and then call os.fsync() on the descriptor, understanding the platform’s guarantees and cost.
import os
with open("state.txt", "w", encoding="utf-8") as file:
file.write("confirmed")
file.flush()
os.fsync(file.fileno())
For atomic updates, write to a temporary file on the same filesystem and replace the destination only after success. See Python tempfile.
Raw I/O
RawIOBase represents low-level byte access. FileIO is the concrete filesystem implementation. Raw operations can read fewer bytes than requested and write only part of a supplied buffer.
import io
raw = io.FileIO("data.bin", "w")
try:
remaining = memoryview(b"content")
while remaining:
count = raw.write(remaining)
if count is None:
continue
remaining = remaining[count:]
finally:
raw.close()
Most applications should use buffered streams, which retry appropriate operations and provide a more predictable contract.
BufferedReader and BufferedWriter
BufferedReader reads larger chunks from the raw layer and keeps unused bytes for later calls. BufferedWriter accumulates output and sends it to the raw stream when the buffer fills, during flush(), seek operations, or closure.
The default size is available as io.DEFAULT_BUFFER_SIZE, though open() may use the file’s block size. Measure throughput, memory, and latency before changing buffer sizes.
read(), read1(), and readinto()
read(size) may issue multiple raw reads. read1(size) uses at most one raw call. readinto(buffer) fills preallocated writable memory and reduces allocations.
buffer = bytearray(64 * 1024)
with open("large.bin", "rb") as file:
while count := file.readinto(buffer):
process(memoryview(buffer)[:count])
Do not retain that view after the buffer is reused unless the data is copied.
BytesIO
BytesIO is an in-memory binary stream.
import io
stream = io.BytesIO()
stream.write(b"header")
stream.seek(0)
print(stream.read())
getvalue() returns the complete bytes value. getbuffer() exposes a writable view without copying. While that view exists, the BytesIO object cannot be resized or closed.
StringIO
StringIO is an in-memory Unicode text stream, useful for tests, report generation, and captured output.
import io
output = io.StringIO()
print("first line", file=output)
print("second line", file=output)
text = output.getvalue()
To emulate append mode, call seek(0, io.SEEK_END). Closing discards the buffer, so retrieve the value first.
TextIOWrapper
TextIOWrapper wraps a buffered binary stream and performs encoding, decoding, and newline translation.
import io
raw = open("data.txt", "rb", buffering=0)
buffer = io.BufferedReader(raw)
text = io.TextIOWrapper(buffer, encoding="utf-8", errors="strict")
try:
print(text.readline())
finally:
text.close()
Normal open(..., encoding=...) creates these layers automatically.
Encoding error policies
errors="strict" raises and should be the default when integrity matters. ignore silently removes data and is rarely appropriate. replace inserts a marker. backslashreplace, xmlcharrefreplace, and namereplace serve specific output contexts.
Do not choose ignore merely to make a file open. Correct the source encoding or adopt a documented substitution policy.
reconfigure()
TextIOWrapper.reconfigure() changes encoding, errors, newline, line buffering, and write-through. Encoding and newline cannot be changed after data has been read because the decoder already has state.
import sys
sys.stdout.reconfigure(encoding="utf-8", errors="backslashreplace")
Changing a global standard stream affects the whole process. Do it only during initialization and respect redirected environments.
seek() and tell() on text streams
Binary positions are byte offsets. A text wrapper’s tell() returns an opaque cookie that also represents decoder state. Only pass values returned by tell() back to seek(cookie).
Do not add arbitrary numbers to text positions. For byte-based random access, operate on the binary layer and decode controlled regions.
Non-blocking streams
A non-blocking raw stream may return None when no data is available and may perform partial writes. Buffered and text layers may raise BlockingIOError.
The previous guide, Python select, shows how to wait for readiness before retrying.
detach()
detach() separates and returns the underlying stream. The outer wrapper becomes unusable afterward.
binary_buffer = text.detach()
Use it only with a clear ownership transfer. StringIO and BytesIO have no detachable lower layer.
Existing descriptors and closefd
When open() or FileIO wraps an existing integer descriptor, closefd=False prevents stream closure from closing that descriptor. This requires explicit ownership rules to avoid leaks or double close.
Custom opener functions
The opener argument controls how a filesystem descriptor is created and can support directory-relative access.
import os
root_fd = os.open("data", os.O_RDONLY)
try:
def opener(path, flags):
return os.open(path, flags, dir_fd=root_fd)
with open("file.txt", "r", encoding="utf-8", opener=opener) as file:
print(file.read())
finally:
os.close(root_fd)
Validate names and path traversal. A custom opener does not automatically create a safe path policy.
Compression and file-like objects
Many standard-library modules accept file-like objects instead of paths, allowing layers to be composed.
import gzip
import io
source = io.BytesIO(compressed_data)
with gzip.GzipFile(fileobj=source, mode="rb") as file:
content = file.read(1_000_000)
In-memory decompression still needs expansion limits. See Python gzip.
Reader and Writer protocols in Python 3.14
Python 3.14 adds io.Reader[T] and io.Writer[T] protocols for typing functions that require only read() or write().
from io import Reader, Writer
def copy_text(source: Reader[str], destination: Writer[str]) -> None:
while chunk := source.read(8192):
destination.write(chunk)
Structural typing lets ordinary files, StringIO, and custom compatible objects satisfy the same interface.
Thread safety and reentrancy
FileIO follows the safety of the underlying system calls. Buffered binary objects protect internal structures with locks and can be called by multiple threads. TextIOWrapper is not thread-safe.
Buffered objects are not reentrant. Performing I/O on the same object from a signal handler may raise RuntimeError. The guide to Python signal recommends minimal handlers without complex logging or output.
Performance
Buffered I/O gives predictable performance and is usually preferable to raw I/O. Text I/O adds codec overhead. For very large files, process chunks and avoid unbounded read().
StringIO and BytesIO are efficient for moderate data sizes but still retain everything in RAM. Use temporary files or streaming when the volume is large.
Recommended tests
Test text and bytes, Unicode, invalid encodings, newline modes, empty files, partial raw reads and writes, non-blocking behavior, seek and tell, repeated close, detach, active BytesIO views, StringIO, descriptor ownership, and memory limits.
Common mistakes
Common failures include omitting encoding, mixing str and bytes, using errors="ignore", assuming raw writes are complete, reading huge files all at once, confusing flush with durable storage, calculating text offsets as bytes, and closing a descriptor owned by another layer.
Conclusion
io provides Python’s central stream contracts. Text, bytes, raw access, buffering, and memory streams are distinct layers that can be combined explicitly.
Specify encodings, prefer buffering, check raw return values, use context managers, and enforce resource limits. Consult the official io documentation and the official open() documentation.
