The struct module converts Python values into byte sequences and reconstructs values from binary buffers. It is used in network protocols, file headers, hardware interfaces, binary databases, shared memory, and integration with C programs. A compact format string describes field types, sizes, alignment, and byte order.
Using struct requires precision. A wrong format can interpret valid bytes as absurd values, truncate data, or trigger excessive allocation. Define a documented layout, validate sizes before unpacking, and never trust counts or offsets read from files or connections.
Your first pack and unpack
pack() receives a format and values. unpack() performs the reverse conversion.
import struct
data = struct.pack("!HI", 2, 4096)
version, length = struct.unpack("!HI", data)
print(version, length)
The ! prefix selects network byte order, which is big-endian with standardized sizes.
Format characters
Common codes include b and B for 8-bit integers, h/H for 16-bit values, i/I for integers, q/Q for 64-bit values, f and d for floating point, ? for booleans, s for fixed-size bytes, and x for padding.
Document the meaning of every field outside the compact string. An unnamed layout is difficult to review and migrate.
Byte order
Prefixes control endianness and alignment: > is big-endian, < is little-endian, ! is network order, = uses native order with standard sizes, and @ uses the complete native layout.
little = struct.pack("<I", 0x12345678)
big = struct.pack(">I", 0x12345678)
print(little.hex(), big.hex())
Persistent formats should choose an explicit order. Do not use @ for data exchanged across architectures.
Calculate the size
calcsize() reports how many bytes a format occupies.
FORMAT = "!4sBHI"
SIZE = struct.calcsize(FORMAT)
Use that value for buffer validation, offsets, and allocation instead of repeating magic numbers.
Fixed-size byte strings
The Ns specifier represents exactly N bytes.
header = struct.pack("!4sI", b"DATA", 10)
magic, length = struct.unpack("!4sI", header)
Short input is padded with null bytes and long input is truncated. Validate the length first when silent truncation would be dangerous.
Text encoding
struct does not encode Python strings. Convert text explicitly.
name = "action".encode("utf-8")
if len(name) > 32:
raise ValueError("name is too long")
block = struct.pack("!32s", name)
When decoding, remove padding only according to the protocol and use the specified encoding.
unpack_from
unpack_from() reads fields from an existing buffer at an offset without creating a slice.
version, flags = struct.unpack_from("!BB", buffer, 4)
Check that offset + calcsize(format) fits inside the buffer.
pack_into
pack_into() writes into a mutable buffer such as bytearray, memoryview, or mmap.
buffer = bytearray(16)
struct.pack_into("!I", buffer, 0, 123)
This can reduce allocations in loops, but offsets and concurrent access must be controlled carefully.
iter_unpack
iter_unpack() walks through fixed-size records.
FORMAT = "!Ih"
for identifier, value in struct.iter_unpack(FORMAT, data):
process(identifier, value)
The total buffer length must be a multiple of the record size.
Variable-length records
For variable payloads, use a fixed header that declares the body length.
HEADER = "!I"
length = struct.unpack(HEADER, receive_exact(4))[0]
if length > 1_000_000:
raise ValueError("payload is too large")
payload = receive_exact(length)
Apply the limit before allocating memory or reading the body.
Signed and unsigned integers
Lowercase integer codes generally represent signed values, while uppercase codes represent unsigned values. Out-of-range input raises struct.error.
Application-level validation is still necessary. A technically representable number may violate the protocol.
Floating-point fields
f represents single precision and d double precision. Values may include rounding, infinity, and NaN.
Do not use binary floats for money or exact counters. Prefer scaled integers or a defined decimal representation.
Booleans
The ? format serializes a boolean. On unpacking, any nonzero byte is interpreted as true.
If the protocol allows only 0 or 1, validate the raw byte or use B and apply the rule explicitly.
Padding and alignment
Native mode @ may insert padding to reproduce a local C structure. The layout changes with architecture and compiler ABI.
For files and network traffic, prefer <, >, or !. Use native layout only for a local ABI integration that is tested on every target.
Integrate with mmap
unpack_from() can decode directly from a memory-mapped file.
import mmap
with mmap.mmap(file.fileno(), 0, access=mmap.ACCESS_READ) as mapping:
magic, version = struct.unpack_from("!4sH", mapping, 0)
See Python mmap for lifecycle, alignment, and synchronization.
Integrate with sockets
TCP is a stream, so one recv() call may return fewer bytes than the header requires. Receive the exact number of bytes before unpacking.
See Python socket for framing, timeouts, and disconnect handling.
The Struct class
When a format is reused, compile it with struct.Struct.
HEADER = struct.Struct("!4sBHI")
block = HEADER.pack(b"DATA", 1, 0, 128)
fields = HEADER.unpack(block)
This centralizes the layout and can reduce repeated parsing overhead.
Versioned layouts
Include magic bytes and a version near the beginning. The parser can select a known layout and reject unsupported versions.
Add fields compatibly or create a new version. Never silently change the meaning of existing bytes.
Checksums and authentication
A checksum detects accidental corruption but does not prove authenticity against an attacker.
Use a MAC or digital signature when security requires origin and integrity, and cover both header and payload.
Offsets and logical overflow
Python integers do not overflow easily, but offset arithmetic can exceed the buffer or request huge allocations.
end = offset + count * item_size
if count > LIMIT or end > len(buffer):
raise ValueError("invalid structure")
Validate before creating slices, lists, or nested objects.
Untrusted binary data
unpack() does not execute code, but an unsafe parser can exhaust CPU and memory or access invalid boundaries.
Limit depth, counts, lengths, and processing time. Use process isolation when the format or threat model justifies it.
Exceptions
Format, size, and range errors raise struct.error.
try:
fields = struct.unpack(FORMAT, block)
except struct.error as error:
raise ValueError("invalid binary record") from error
Avoid logging complete binary payloads.
Testing
Test minimum and maximum values, zero, negative numbers, both byte orders, NaN, padding, short buffers, extra data, unknown versions, and truncated files.
Use known vectors generated by another language to verify interoperability.
Common mistakes
Common failures include using native layout for portable files, forgetting calcsize(), silently truncating strings, calling unpack() with the wrong size, trusting external lengths, using floats for exact values, and assuming one socket read returns a complete record.
Conclusion
struct is the bridge between Python values and compact binary layouts. Choose explicit endianness, centralize formats, validate every buffer, and version persistent protocols.
Consult the official struct documentation, Python mmap, and Python socket.
