Python zlib: Compress Data Safely

Python zlib provides compression and decompression compatible with the zlib library and the DEFLATE algorithm. It works with bytes, supports complete in-memory buffers and incremental streams, and also exposes Adler-32 and CRC-32 checksums.

Use zlib for protocols, caches, internal storage blocks, and formats that require zlib streams, raw DEFLATE, or basic gzip framing. For ordinary .gz files with file-like behavior, timestamps, and complete gzip headers, the gzip module is usually more convenient.

Compress an in-memory buffer

zlib.compress() receives bytes and returns compressed bytes.

import zlib

source = ("Python data compression. " * 100).encode("utf-8")
compressed = zlib.compress(source)

print(len(source), len(compressed))

Repetitive data compresses well. Already compressed, encrypted, or random-looking data may not shrink and can become slightly larger because of framing overhead.

Decompress the result

restored = zlib.decompress(compressed)
assert restored == source

Invalid framing, a bad checksum, an incompatible window size, or a truncated stream raises zlib.error. Catch that exception at the boundary where external data enters the application.

Select a compression level

The level argument ranges from 0 through 9, with -1 selecting the library default.

fast = zlib.compress(source, level=zlib.Z_BEST_SPEED)
small = zlib.compress(source, level=zlib.Z_BEST_COMPRESSION)
default = zlib.compress(source, level=zlib.Z_DEFAULT_COMPRESSION)

Level 9 spends more CPU trying to reduce output size. The best choice depends on data shape, storage costs, network bandwidth, and how often the content is read. Benchmark representative payloads.

Understand wbits

wbits controls both the history window and the stream wrapper.

• 9 through 15: zlib header and checksum.
• -9 through -15: raw DEFLATE without wrapper data.
• 25 through 31: basic gzip stream.

zlib_stream = zlib.compress(source, wbits=15)
raw_stream = zlib.compress(source, wbits=-15)
gzip_stream = zlib.compress(source, wbits=31)

The decompressor must use a compatible mode. Passing raw DEFLATE to the default zlib mode is a common integration failure.

Auto-detect zlib or gzip

During decompression, wbits values from 40 through 47 accept either zlib or gzip framing.

output = zlib.decompress(external_data, wbits=47)

Auto-detection is useful for tolerant integrations. If a protocol requires exactly one framing format, reject alternatives instead of accepting them silently.

Compress large streams incrementally

compressobj() retains state across chunks and avoids loading a complete file into memory.

compressor = zlib.compressobj(level=6, wbits=15)

with open("input.bin", "rb") as source_file, open("input.bin.z", "wb") as destination:
    while chunk := source_file.read(64 * 1024):
        destination.write(compressor.compress(chunk))
    destination.write(compressor.flush())

compress() may buffer part of the input internally. Write every returned block and always finish the stream with flush().

Choose the correct flush mode

The default flush() mode is Z_FINISH. After it, the compressor cannot accept more data.

final_bytes = compressor.flush(zlib.Z_FINISH)

Z_SYNC_FLUSH keeps the stream open for interactive protocols, but inserts synchronization data and reduces compression efficiency. Z_FULL_FLUSH resets more state and may help recovery-oriented formats at a greater size cost.

Decompress incrementally

decompressor = zlib.decompressobj(wbits=15)

with open("input.bin.z", "rb") as source_file, open("restored.bin", "wb") as destination:
    while chunk := source_file.read(64 * 1024):
        destination.write(decompressor.decompress(chunk))
    destination.write(decompressor.flush())

if not decompressor.eof:
    raise ValueError("Compressed stream is incomplete")

The eof attribute distinguishes a valid ending from a truncated file. The absence of an early exception is not enough.

Limit decompressed output

A small compressed payload may expand to an enormous output. Use max_length together with a total output limit to reduce decompression-bomb risk.

decompressor = zlib.decompressobj()
maximum = 100 * 1024 * 1024
produced = 0
pending = payload

while pending:
    part = decompressor.decompress(pending, max_length=1024 * 1024)
    produced += len(part)
    if produced > maximum:
        raise ValueError("Decompressed output exceeds the limit")
    save(part)
    pending = decompressor.unconsumed_tail
    if not pending:
        break

Also limit compressed input size, CPU time, stream count, and process memory. Consider an isolated worker for high-risk external content.

Handle unconsumed_tail

When max_length prevents complete consumption, remaining compressed bytes are stored in unconsumed_tail. Feed them back to the decompressor before reading more input.

Ignoring this property silently loses data and can create a partial output that appears successful.

Inspect unused_data

unused_data contains bytes located after the end of the compressed stream.

obj = zlib.decompressobj()
output = obj.decompress(stream_with_suffix)
output += obj.flush()
remaining = obj.unused_data

This supports protocols that append fields after a compressed payload. Validate unexpected trailing bytes because they may indicate corruption or parser-confusion attempts.

Process concatenated streams

A decompressobj stops at the first completed stream. If a format permits several members, create a new decompressor for unused_data and repeat with a strict member limit.

Never run an unbounded loop over concatenated members; an attacker may provide thousands of tiny or empty streams.

Use compression dictionaries

A zdict can improve compression for short messages that share a vocabulary.

dictionary = b'"type":"","id":,"timestamp":,"payload":'
compressor = zlib.compressobj(level=6, zdict=dictionary)
compressed = compressor.compress(message) + compressor.flush()

decompressor = zlib.decompressobj(zdict=dictionary)
original = decompressor.decompress(compressed) + decompressor.flush()

The compressor and decompressor must use identical bytes. Frequently used sequences should be placed near the end of the dictionary.

Version the dictionary

A protocol should transmit or negotiate a dictionary identifier. Do not attempt an unlimited list of candidate dictionaries because that increases CPU cost and creates ambiguous behavior.

If a mutable bytearray is used as a decompression dictionary, do not modify it between object creation and the first decompression call.

Tune memLevel and strategy

compressobj() accepts memory and strategy controls.

compressor = zlib.compressobj(
    level=6,
    method=zlib.DEFLATED,
    wbits=15,
    memLevel=8,
    strategy=zlib.Z_DEFAULT_STRATEGY,
)

Z_FILTERED may help filtered data, Z_HUFFMAN_ONLY disables match searching, Z_RLE favors runs, and Z_FIXED uses fixed Huffman tables. Benchmark before changing defaults.

Copy compressor state

Compress.copy() can branch outputs that share a common compressed prefix.

base = zlib.compressobj()
prefix = base.compress(common_header)

branch_a = base.copy()
output_a = prefix + branch_a.compress(payload_a) + branch_a.flush()

branch_b = base.copy()
output_b = prefix + branch_b.compress(payload_b) + branch_b.flush()

Decompressors can also be copied, which may support indexed seeking. Document the state boundary carefully.

Calculate CRC-32

checksum = zlib.crc32(data)
print(f"{checksum:08x}")

For incremental input:

crc = 0
for chunk in chunks:
    crc = zlib.crc32(chunk, crc)

CRC-32 detects accidental corruption but is not cryptographic authentication. An attacker can recompute it.

Calculate Adler-32

adler = zlib.adler32(data)

Adler-32 is fast and suitable for non-adversarial integrity checks. Use HMAC for authenticated integrity and hashlib for content hashes.

Choose between gzip, zipfile, and zlib

zlib processes DEFLATE streams and buffers. gzip is a single-stream file format. zipfile stores multiple files and metadata in one archive.

The Python zipfile guide covers safe extraction, Python zipapp packages applications, and Python zipimport imports modules from ZIP archives.

Compress only when it helps

Very small messages can grow because of headers. Images, video, PDFs, and ZIP files are commonly compressed already. Detect content type and compare sizes before storing a compressed version when the protocol permits.

Use bytes and explicit encoding

The module does not accept text strings.

data = text.encode("utf-8")
compressed = zlib.compress(data)
restored = zlib.decompress(compressed).decode("utf-8")

Encoding belongs to the application protocol. Specify UTF-8 explicitly and handle decoding errors only after validating the compressed stream.

Inspect library versions

print(zlib.ZLIB_VERSION)
print(zlib.ZLIB_RUNTIME_VERSION)
print(getattr(zlib, "ZLIBNG_VERSION", None))

The compile-time library may differ from the runtime library. Python 3.14 exposes ZLIBNG_VERSION when built with zlib-ng. Record these values during compatibility or performance investigations.

Handle zlib.error

try:
    output = zlib.decompress(payload, wbits=47)
except zlib.error as exc:
    log_failure_without_payload(exc)
    raise ValueError("Invalid compressed payload") from exc

Do not log complete payload bytes. They may contain secrets and can make logs enormous.

Security checklist

• Set a maximum decompressed size.
• Apply CPU and time limits.
• Verify eof for truncation.
• Validate trailing unused_data.
• Limit concatenated stream members.
• Do not use CRC or Adler for authentication.
• Do not trust a user-supplied framing claim.
• Isolate high-risk external payloads.

Recommended practices

• Use incremental objects for large data.
• Benchmark levels, strategies, and chunk sizes.
• Always finish compressors with flush().
• Reprocess unconsumed_tail.
• Check eof.
• Version compression dictionaries.
• Avoid recompressing existing compressed formats.
• Record zlib versions in diagnostics.

Conclusion

Python zlib provides flexible DEFLATE compression for buffers and streams, including zlib, raw, and gzip wrapping, dictionaries, strategies, checksums, and incremental state.

External compressed data requires strict limits to prevent excessive expansion and resource consumption. Consult the official zlib documentation and the official zlib library manual.