The mmap module creates memory-mapped objects that let a program access file content as an addressable byte sequence. Instead of copying an entire file into a Python object, the operating system maps regions of the file into the process address space and loads pages as they are touched. This can simplify random access, binary editing, searching, inter-process sharing, and work with very large files.
Memory mapping does not make every algorithm faster. Results depend on access patterns, page faults, operating-system cache behavior, file size, and synchronization needs. The API also differs in selected details between Windows and Unix. Use mmap when position-based access, buffer-protocol integration, or shared pages provide a real advantage.
Open a file for mapping
The file mode must match the desired access. Use a descriptor opened for both reading and writing when the mapping will be modified.
from pathlib import Path
import mmap
path = Path("data.bin")
with path.open("r+b") as file:
with mmap.mmap(file.fileno(), 0) as mapping:
print(len(mapping))
print(mapping[:16])
A length of zero maps the current file size on supported platforms. Empty files normally cannot be mapped this way.
Use context managers
An mmap object owns a native resource. A with block guarantees closure even when parsing or processing fails.
Keep the original file lifecycle explicit as well. Although a mapping may remain usable after the descriptor closes on many systems, avoid relying on undocumented cross-platform behavior.
Read-only mappings
When an application only inspects data, open the file in binary read mode and request read access.
with open("index.bin", "rb") as file:
with mmap.mmap(
file.fileno(),
0,
access=mmap.ACCESS_READ,
) as mapping:
position = mapping.find(b"KEY=")
print(position)
Attempts to modify a read-only mapping fail. This restriction reduces the impact of bugs and clearly documents intent.
Indexes and slices
The object behaves much like a mutable byte sequence. One index returns an integer, while a slice returns a bytes object.
first = mapping[0]
header = mapping[0:32]
Large slices still copy data. Use memoryview for zero-copy integration with compatible APIs, and manage its lifetime carefully.
The internal cursor
Besides indexed access, mmap provides file-like methods including read(), readline(), seek(), and tell().
mapping.seek(100)
block = mapping.read(64)
print(mapping.tell())
The cursor is mutable shared state. Two parts of a program using the same object can interfere. Prefer explicit offsets or synchronization.
Search byte patterns
find() and rfind() search for byte sequences without first creating a second in-memory copy of the full file.
start = 0
while True:
position = mapping.find(b"ERROR", start)
if position == -1:
break
print(position)
start = position + 5
This works well for logs and binary formats, but text content still requires correct encoding and boundary handling.
Text and encoding
A mapping contains bytes. Decode only the required region.
line = mapping[start:end].decode("utf-8", errors="strict")
A slice may cut through a multibyte character. Find delimiters at the byte level or use an incremental decoder when processing chunks.
Modify bytes in place
A writable mapping can replace bytes without rewriting the entire file.
with open("record.bin", "r+b") as file:
with mmap.mmap(file.fileno(), 0) as mapping:
mapping[8:12] = b"DONE"
mapping.flush()
The assigned byte count must match the replaced region. An mmap object does not grow like a list when a longer slice is assigned.
Flush and durability
flush() requests that changed pages be written. Exact durability depends on the operating system, filesystem, device cache, and flags.
For critical data, combine mapping with transactional design, temporary files, atomic rename, and fsync() where appropriate. A flush does not turn several writes into one atomic transaction.
Private copy-on-write mappings
ACCESS_COPY creates a private copy-on-write view. The process observes its own changes, but the original file is not updated.
with mmap.mmap(
file.fileno(),
0,
access=mmap.ACCESS_COPY,
) as mapping:
mapping[0:4] = b"TEST"
This is useful for experimental transformations or temporary views. Modified pages may still increase process memory.
Map only one region
Very large files can be processed through smaller windows. The offset must respect the allocation granularity required by the platform.
granularity = mmap.ALLOCATIONGRANULARITY
offset = (start // granularity) * granularity
delta = start - offset
length = delta + size
with mmap.mmap(
file.fileno(),
length,
access=mmap.ACCESS_READ,
offset=offset,
) as mapping:
data = mapping[delta:delta + size]
Misaligned offsets are a common source of failures in partial mappings.
Files larger than RAM
An operating system can map a file larger than physical memory because pages are loaded on demand. However, random traversal may trigger heavy paging and thrashing.
Prefer sequential access when possible, measure resident memory and page faults, and avoid keeping many huge mappings open without a reason.
Resize carefully
resize() can change a mapping under selected platform and access conditions. Support and restrictions vary.
For portable code, close the mapping, resize the file, and create a new mapping. Recalculate every offset derived from the previous size.
Create a pre-sized file
The underlying file must contain enough bytes before writing to future regions.
size = 1024 * 1024
with open("block.bin", "w+b") as file:
file.truncate(size)
with mmap.mmap(file.fileno(), size) as mapping:
mapping[0:4] = b"DATA"
Sparse-file behavior and physical allocation depend on the filesystem. truncate() does not necessarily reserve every device block.
Anonymous mappings
On supported platforms, a mapping can be created without a file and used as a working buffer or shared-memory region.
with mmap.mmap(-1, 4096) as mapping:
mapping[:5] = b"hello"
print(mapping[:5])
Naming and sharing rules differ between Windows and Unix. Python processes may also benefit from multiprocessing.shared_memory.
Share data between processes
Processes mapping the same file region can observe shared updates, depending on access mode. Mapping does not provide synchronization.
Use locks, semaphores, version markers, checksums, and a publication protocol. A reader must not parse a structure while a writer is changing half of it.
Threads and shared state
Independent indexed reads can be simple, but cursor-based operations such as seek() and read() require coordination when an object is shared.
Protect concurrent writes or centralize modification. The absence of an exception does not imply logical consistency.
External file changes
If another process truncates a mapped file, later access may produce severe errors, operating-system signals, or invalid memory references.
Define ownership and prohibit resizing while readers exist. For complete updates, create a new file and switch it into place atomically.
Buffer protocol and memoryview
A memoryview lets compatible libraries consume mapping bytes without another copy.
with mmap.mmap(file.fileno(), 0, access=mmap.ACCESS_READ) as mapping:
view = memoryview(mapping)
try:
consume_buffer(view[100:200])
finally:
view.release()
The mapping cannot close while exported views remain active. Release them explicitly.
Integrate with struct
Fixed-layout binary formats can be decoded directly using struct.unpack_from().
import struct
version, length = struct.unpack_from("!HI", mapping, 0)
Check the minimum buffer size first, and never trust lengths read from external files.
Measure performance
Compare mmap with buffered chunk reading, readinto(), and higher-level APIs. For simple sequential scans, buffered reads may be equally fast and easier to maintain.
Measure total time, resident memory, page faults, and behavior under realistic concurrency. Benchmarks against files already in the page cache can be misleading.
Security
A mapped file is still untrusted input. Validate magic bytes, signatures, sizes, counts, offsets, and boundaries before indexing calculated positions.
Do not use file-provided values in slice arithmetic without limits. A malicious file can trigger excessive memory use or invalid access.
Error handling
Opening, mapping, flushing, and indexing can raise OSError, ValueError, TypeError, or index-related exceptions.
try:
with open(path, "rb") as file:
with mmap.mmap(file.fileno(), 0, access=mmap.ACCESS_READ) as mapping:
analyze(mapping)
except (OSError, ValueError) as error:
raise RuntimeError(f"could not map {path}") from error
Include the path and operation in diagnostics without exposing sensitive content.
Testing
Test empty files, minimum size, truncation, denied permissions, misaligned offsets, concurrent changes, invalid binary structures, boundary slices, flush behavior, Windows, and Unix.
Use real temporary files. File mocks do not reproduce alignment, page faults, or operating-system semantics.
Common mistakes
Common failures include mapping empty files, using an incompatible open mode, forgetting offset alignment, assuming slices are zero-copy, closing with an active memoryview, resizing externally, treating flush() as a transaction, and choosing random access without measuring paging cost.
Conclusion
mmap turns files and shared regions into addressable buffers. It is especially useful for large-file searches, binary formats, random access, and buffer-protocol integration.
Keep ownership explicit, validate boundaries, synchronize writers, and benchmark against buffered I/O. Consult the official mmap documentation, Python multiprocessing, and Python contextlib.
