Large files can be accessed without manually reading the entire contents into one Python bytes object. Python mmap creates a mapping between a file region and the process virtual address space. The resulting object behaves both like a file and like a mutable byte array.
This technique is useful for pattern search, binary indexes, embedded storage, interprocess communication, and fixed-region updates. It does not make every workload automatically faster, and it requires careful handling of access rights, page alignment, concurrent writers, file size, and persistence. This guide covers read, write, copy-on-write, anonymous mappings, platform differences, flushing, and resource management.
It complements our guides to fileinput, linecache, tempfile, filecmp, and weakref.
How mapping works
The operating system associates file pages with virtual-memory addresses. Pages are loaded as needed, and the kernel manages caching and eviction. Python code can then use indexes, slices, searches, or file-like methods.
The disk cost still exists. Page faults, random access, and memory pressure can dominate performance, especially when the file is much larger than available RAM.
Read-only whole-file mapping
import mmap
with open("data.bin", "rb") as file:
with mmap.mmap(
file.fileno(),
length=0,
access=mmap.ACCESS_READ,
) as memory:
print(memory[:16])length=0 requests the current file size on supported platforms. Windows cannot create this mapping for an empty file.
Read-only access
ACCESS_READ prevents mutation. Assignment raises TypeError.
memory[0] = 65 # TypeErrorPrefer read-only access for search and analysis. It reduces the chance of accidental file corruption.
Persistent write access
with open("data.bin", "r+b") as file:
with mmap.mmap(
file.fileno(),
0,
access=mmap.ACCESS_WRITE,
) as memory:
memory[0:4] = b"HEAD"
memory.flush()With ACCESS_WRITE, changes affect both the mapping and the underlying file. Slice replacement must use the same length; mmap is not a dynamically growing list.
Copy-on-write
with open("data.bin", "rb") as file:
with mmap.mmap(
file.fileno(),
0,
access=mmap.ACCESS_COPY,
) as memory:
memory[0:4] = b"TEST"Changes remain private and do not update the file. This is useful for experiments and temporary transformations, but it is not the same as making a full independent file copy in advance.
Flush buffered file writes first
If a writable Python file object contains buffered modifications, call flush() before creating the map.
file.write(b"content")
file.flush()
memory = mmap.mmap(file.fileno(), 0)This ensures that local buffered data is visible to the mapping mechanism.
Searching bytes
position = memory.find(b"ERROR")
if position != -1:
print("found at", position)find() and rfind() accept start and end ranges. The re module can also search directly through a mapped object.
Regular expressions
import re
for match in re.finditer(rb"ID:\d+", memory):
print(match.start(), match.group())Use byte patterns rather than Unicode strings. Text interpretation requires a known encoding and care around multibyte character boundaries.
File-like methods
Mmap objects maintain a current position and provide read(), readline(), seek(), tell(), and write().
memory.seek(0)
first_line = memory.readline()
position = memory.tell()Since Python 3.13, seek() returns the new absolute position.
Indexing and slicing
first = memory[0] # integer
header = memory[0:16] # bytesAssigning an index requires an integer byte value. Assigning a slice requires a bytes-like object with compatible length.
Flushing changes
flush() asks the system to write modified pages back.
memory.flush()An offset and size can be supplied, but the offset must be aligned to PAGESIZE or ALLOCATIONGRANULARITY. Handle exceptions rather than assuming persistence succeeded.
Durability is not a transaction
Flushing the mapping, syncing the file descriptor, and guaranteeing crash consistency are related but distinct topics. Transactional applications must understand fsync, write ordering, metadata updates, and atomic replacement.
Mmap does not automatically turn a file into a reliable database.
Aligned offsets
The constructor offset must be a multiple of ALLOCATIONGRANULARITY. To map an arbitrary region, align the starting offset downward and maintain an internal delta.
gran = mmap.ALLOCATIONGRANULARITY
base = start // gran * gran
delta = start - base
length = delta + requested_sizeAccess the desired bytes through memory[delta:delta+requested_size].
File size and mapping size
The mapping length defines the visible region. size() can return the underlying file size, which may exceed the mapped area. len(memory) reports the mapped length.
Do not access beyond the mapping. Concurrent file-size changes can have platform-specific and dangerous consequences.
Resizing
resize() changes the map and, where applicable, the file. Read-only, copy-on-write, or trackfd=False maps cannot be resized.
On Windows, other maps for the same file can block resizing. Coordinate the operation exclusively.
trackfd on Unix
Since Python 3.13, the Unix constructor accepts trackfd=False. The descriptor is not duplicated, reducing descriptor usage, but size() and resize() stop working.
Use it only when descriptor pressure is real and the application owns the lifecycle carefully.
Anonymous memory
with mmap.mmap(-1, 4096) as memory:
memory.write(b"temporary data")Passing -1 creates a region not backed by a regular file. It can act as a buffer and, under some process models, as shared memory.
Interprocess sharing
On Unix, an anonymous map created before fork() can be shared between parent and child. For portable application code, multiprocessing.shared_memory may expose intent more clearly.
Shared memory provides no automatic synchronization. Use locks, semaphores, or a carefully designed atomic protocol.
MAP_SHARED and MAP_PRIVATE
The low-level Unix constructor supports MAP_SHARED for mappings visible to other processes and MAP_PRIVATE for copy-on-write behavior.
Do not specify access together with explicit flags and prot; that combination is invalid.
Windows differences
The Windows constructor accepts an optional tagname for named mappings, but avoiding it improves portability. Offsets still need allocation-granularity alignment.
Mapping a length larger than the file can extend the file on Windows. Do not rely on this behavior in cross-platform code.
Kernel advice with madvise
On supported systems, madvise() can describe an expected access pattern.
if hasattr(memory, "madvise"):
memory.madvise(mmap.MADV_SEQUENTIAL)Available constants differ by operating system. Treat this as an optional optimization and benchmark it.
Sequential and random access
Mapping does not eliminate storage latency. Sequential scans benefit from prefetching, while random access to data much larger than RAM can cause many page faults.
Benchmark representative data and observe resident memory, page faults, and tail latency.
Concurrent writers
Two processes can modify the same region and violate application invariants. Memory coherence does not protect multi-step logical updates.
Define headers, versions, locks, checksums, and a commit protocol. Never assume a sequence of byte writes is atomic.
External truncation
If another process truncates a mapped file, later access can fail severely or behave differently across systems. Control ownership and the file lifecycle while mappings exist.
Closing resources
Use context managers. Closing an mmap does not close the original Python file object.
with open("data.bin", "rb") as file:
with mmap.mmap(file.fileno(), 0, access=mmap.ACCESS_READ) as memory:
process(memory)Outstanding memory views can prevent closing. Release them before leaving the context.
Security
Mapping a file does not validate its contents. Lengths, offsets, and internal records can be malicious. Verify all boundaries before using values from the file to calculate slices.
Do not map devices or special files without understanding their behavior. Creating a map emits a Python auditing event.
Testing
Test empty files, small and large files, read-only maps, copy-on-write behavior, aligned regions, unsuccessful searches, flushing, controlled concurrency, and both Windows and Unix.
Common mistakes
- Mapping an empty file on Windows.
- Opening with an incompatible file mode.
- Forgetting to flush buffered writes before mapping.
- Using an unaligned offset.
- Assuming
flush()provides a transaction. - Assigning a slice of different length.
- Resizing while other maps exist.
- Sharing memory without synchronization.
Best practices
- Default to
ACCESS_READ. - Use copy-on-write for temporary edits.
- Validate sizes and offsets.
- Close maps with
with. - Benchmark real access patterns.
- Synchronize concurrent writers.
- Test platform differences.
- Do not treat mmap as a transactional database.
Conclusion
Python mmap provides flexible, efficient access to files and memory regions. It combines byte slicing, file methods, searching, and controlled sharing.
Performance depends on the access pattern and operating system. Use minimal permissions, align offsets, validate structures, and coordinate concurrency. Consult the official mmap documentation and the Unix mmap manual for lower-level details.
