Python lzma: Compress XZ Files

The Python lzma module provides compression and decompression with LZMA and LZMA2. It supports modern .xz files, legacy .lzma containers, and raw streams configured with explicit filters. Its API includes one-shot functions, incremental objects, and a compressed file interface similar to bz2.

LZMA frequently produces smaller output than gzip or bzip2, but it uses more CPU and can require substantial memory. Preset 9 may make a compressor consume hundreds of megabytes. The default preset 6 is therefore a safer starting point, and production settings should be chosen from benchmarks with representative data.

When lzma fits

Use lzma when XZ compatibility is required, when final size matters more than maximum speed, or when creating long-term datasets that are not opened continuously. For another text-friendly compressor with lower complexity, see Python bz2.

Compress data in memory

import lzma

data = ("audit record\n" * 2000).encode("utf-8")
compressed = lzma.compress(data, preset=6)
restored = lzma.decompress(compressed)

assert restored == data
print(len(data), len(compressed))

lzma.compress() creates XZ by default. It is convenient for small payloads, but input and output coexist in memory. Use files or incremental processing for large workloads.

Write an XZ text file

import lzma

with lzma.open("events.log.xz", "wt", encoding="utf-8", preset=6) as file:
    file.write("process started\n")
    file.write("processing completed\n")

Modes follow the built-in file conventions: rt, wt, xt, and at for text, with corresponding binary modes. Use exclusive creation when accidental replacement must fail.

Read without loading everything

import lzma

with lzma.open("events.log.xz", "rt", encoding="utf-8") as file:
    for line in file:
        process(line.rstrip())

Iteration yields one line at a time and works well with the practices in the guide to reading giant files in Python. The decompressor still maintains an internal dictionary, so observe process memory in long-running services.

Use LZMAFile for binary access

LZMAFile accepts a path or an already-open object and implements most of io.BufferedIOBase. It supports context management, iteration, reading, writing, and seeking when the underlying resource permits.

from lzma import LZMAFile

with LZMAFile("payload.bin.xz", "wb", preset=5) as target:
    target.write(b"header\x00")
    target.write(b"data" * 10_000)

A single instance is not thread-safe for concurrent readers or writers. Use a lock or separate objects.

Select the container format

• FORMAT_XZ: modern container with integrity checks and filter support.
• FORMAT_ALONE: legacy .lzma format with fewer features.
• FORMAT_RAW: no container; both sides must know the exact filter chain.
• FORMAT_AUTO: detects XZ or legacy LZMA while decompressing.

Choose XZ for new files. RAW is appropriate only when a protocol already defines all parameters.

Integrity checks

XZ can use CRC32, CRC64, or SHA-256 checks. CRC64 is the default. Call lzma.is_check_supported() before requesting an optional check.

import lzma

check = lzma.CHECK_SHA256
if not lzma.is_check_supported(check):
    check = lzma.CHECK_CRC64

output = lzma.compress(b"content", check=check)

These checks detect accidental corruption. They do not authenticate a sender and do not replace signatures or authenticated hashes.

Incremental compression

import lzma

compressor = lzma.LZMACompressor(preset=6)
parts = []

for chunk in generate_chunks():
    part = compressor.compress(chunk)
    if part:
        parts.append(part)

parts.append(compressor.flush())
compressed = b"".join(parts)

The object may buffer input, so calls can return empty bytes. flush() finishes the stream and prevents further use.

Limit decompressor memory

LZMADecompressor accepts memlimit. Decompression raises LZMAError when the stream cannot be decoded within that budget. This is important for untrusted input.

import lzma

decoder = lzma.LZMADecompressor(memlimit=128 * 1024 * 1024)
output = bytearray()
maximum_output = 50 * 1024 * 1024

for chunk in receive_chunks():
    pending = chunk
    while pending or not decoder.needs_input:
        part = decoder.decompress(pending, max_length=64 * 1024)
        pending = b""
        output.extend(part)
        if len(output) > maximum_output:
            raise ValueError("expanded output exceeded the limit")
        if decoder.eof:
            break

memlimit controls decoder workspace, not total expanded size. Combine it with max_length and an application counter.

Handle concatenated streams

lzma.decompress() and LZMAFile transparently process concatenated streams. One LZMADecompressor does not. After eof, inspect unused_data and create another decoder when additional members are allowed.

Custom filter chains

A chain may contain up to four filters and must end with LZMA1 or LZMA2. Delta filters can improve regularly changing numeric data; BCJ filters target executable machine code.

import lzma

filters = [
    {"id": lzma.FILTER_DELTA, "dist": 4},
    {"id": lzma.FILTER_LZMA2, "preset": 6},
]
compressed = lzma.compress(data, format=lzma.FORMAT_RAW, filters=filters)

Custom filters reduce interoperability. Record the exact configuration and test with every implementation that must decode it.

Avoid extreme presets by default

PRESET_EXTREME can be combined with levels 0 through 9, but it often increases runtime significantly for modest size improvements. Do not use preset 9 extreme inside a request handler without memory controls, a benchmark, and a bounded job queue.

Operational security

• Limit compressed input size.
• Use memlimit and a total output limit.
• Check eof to identify truncation.
• Do not trust extensions or supplied names.
• Store files in an isolated directory.
• Bound CPU time.
• Do not share one decoder across threads.

If XZ is embedded in an archive with paths, apply the path validation covered in the ZIP guide.

Benchmark representative data

Measure time, peak memory, and size for text, binary data, already-compressed media, and empty files. Include corrupted and truncated inputs, multiple members, and different checks. Compare presets 3, 6, and 9 before making a policy.

Best practices

• Prefer FORMAT_XZ for new work.
• Keep preset 6 until measurements justify a change.
• Specify text encodings.
• Use context managers.
• Stream large data.
• Call compressor flush().
• Combine memory and output limits.
• Handle LZMAError without exposing sensitive input.

Conclusion

Python lzma provides efficient XZ compression, text and binary files, integrity checks, and advanced filters. Its strongest advantage is output size; its tradeoffs are CPU, memory, and operational complexity.

Read the official lzma documentation and the XZ Utils project. To configure presets and limits per environment, see Python configparser, and to inspect memory behavior use Python tracemalloc.