compression.zstd: Compress Data with Zstandard

The compression.zstd module brings the Zstandard format to Python’s standard library and provides a fast, modern way to reduce the size of files and byte sequences. It is a strong choice for backups, log pipelines, caches, data exports, network payloads, and systems that need better compression than gzip without accepting extremely slow processing.

This guide covers byte compression, streaming, compression levels, dictionaries, integrity checks, and defensive decompression. For related workflows, review the Academify guides to safe ZIP archives with zipfile, file handling with pathlib, reading huge files without exhausting memory, and copying and moving files with shutil.

What Zstandard is

Zstandard, commonly called Zstd, is a lossless compression algorithm designed to balance speed and compression ratio. On many text and structured-data workloads, it can create smaller output than gzip while compressing and decompressing quickly. It also supports adjustable levels, reusable dictionaries, independent frames, and streaming.

The official Python compression.zstd documentation is the primary source for version-specific signatures. The underlying frame format is standardized in RFC 8878.

Compressing bytes

For small payloads, work directly with bytes. Encode text, compress it, and decompress it when needed.

from compression import zstd

text = "Python and Zstandard compression" * 100
data = text.encode("utf-8")

compressed = zstd.compress(data)
restored = zstd.decompress(compressed)

print(len(data), len(compressed))
print(restored.decode("utf-8") == text)

This approach is convenient, but both the source and result may coexist in memory. For large inputs, use a stream so memory consumption stays predictable.

Choosing a compression level

Higher levels usually spend more CPU to produce smaller output. A low or medium level is often best for APIs, caches, and frequently generated files. Higher levels may be worthwhile for archival data written once and downloaded many times.

compressed = zstd.compress(data, level=6)

Benchmark your own workload. JSON lines, source code, database dumps, and plain text behave differently from images or data that is already compressed. Measure compressed size, compression time, decompression time, memory, and total infrastructure cost.

Streaming large files

A streaming workflow reads and writes bounded chunks instead of loading an entire file.

from pathlib import Path
from compression import zstd

source = Path("events.jsonl")
target = Path("events.jsonl.zst")

with source.open("rb") as input_file, target.open("wb") as output_file:
    with zstd.open(output_file, mode="wb", level=5) as compressor:
        while chunk := input_file.read(1024 * 1024):
            compressor.write(chunk)

A one-megabyte chunk is a reasonable starting point, not a universal rule. Smaller chunks reduce temporary memory but increase function calls. Larger chunks may improve throughput until disk or CPU becomes the bottleneck.

Safe decompression

Compressed data from an external source is untrusted input. A tiny compressed file may expand into gigabytes. Enforce an output limit, write only to controlled paths, check available disk space, and remove partial files after errors.

from pathlib import Path
from compression import zstd

source = Path("events.jsonl.zst")
target = Path("events-restored.jsonl")
max_output = 2 * 1024 * 1024 * 1024
written = 0

try:
    with zstd.open(source, mode="rb") as reader, target.open("wb") as writer:
        while chunk := reader.read(1024 * 1024):
            written += len(chunk)
            if written > max_output:
                raise ValueError("Decompressed output exceeds the limit")
            writer.write(chunk)
except Exception:
    target.unlink(missing_ok=True)
    raise

Services should also apply request-size limits, timeouts, quotas, and cancellation. Never let a user-provided archive choose an arbitrary output path.

Compression dictionaries

Dictionaries improve compression when you process many small, structurally similar values, such as JSON events, telemetry records, or messages with repeated field names. Training captures common byte patterns that would otherwise be too short for the compressor to learn within each item.

Collect representative samples, train a dictionary, assign it a version, and use the same dictionary for compression and decompression. The receiver must possess the matching version. Store a dictionary identifier next to each payload and retain old dictionaries for as long as dependent data exists.

Checksums and integrity

Compression is not a complete integrity strategy. Calculate a cryptographic hash such as SHA-256 and keep it in a manifest with the original size, compressed size, creation time, algorithm, and dictionary version.

import hashlib

def sha256_file(path):
    digest = hashlib.sha256()
    with open(path, "rb") as file:
        while chunk := file.read(1024 * 1024):
            digest.update(chunk)
    return digest.hexdigest()

After restoring a backup, calculate the hash again and compare it before replacing production data.

Zstd versus gzip and ZIP

Choose Zstd when throughput and compression ratio matter and you control the reader. Choose gzip for maximum compatibility with older tooling and web infrastructure. Choose ZIP when you need a container that preserves multiple filenames, directory entries, and per-file metadata. For directory backups, a common design is to create a tar stream and compress that stream with Zstandard.

Atomic output files

Write compressed output to a temporary file in the same filesystem, flush it, validate it, and then rename it over the final destination. Filesystem rename operations are typically atomic within one filesystem, so consumers never observe a half-written result.

from pathlib import Path
import os

final = Path("events.jsonl.zst")
temporary = final.with_suffix(final.suffix + ".tmp")

# Write and validate temporary here
os.replace(temporary, final)

This pattern is valuable for scheduled exports, backup jobs, and caches read concurrently by other processes.

Production practices

Record compression ratio, duration, failures, input bytes, and output bytes. Alert when ratios change sharply because that may reveal a schema change, accidental double compression, or unexpected binary content. Add retention rules and periodically perform restore tests instead of assuming backups are valid.

Avoid recompressing JPEG, PNG, MP4, ZIP, and other formats that already use compression. The output may become slightly larger while consuming CPU. Do not automatically choose the maximum level either. A middle level frequently gives a better cost-to-size balance for continuous pipelines.

Reusable helper

from pathlib import Path
from compression import zstd

def compress_file(source, target, level=5, chunk_size=1024 * 1024):
    source = Path(source)
    target = Path(target)
    temporary = target.with_suffix(target.suffix + ".tmp")

    try:
        with source.open("rb") as reader, temporary.open("wb") as raw_writer:
            with zstd.open(raw_writer, mode="wb", level=level) as writer:
                while chunk := reader.read(chunk_size):
                    writer.write(chunk)
        temporary.replace(target)
    except Exception:
        temporary.unlink(missing_ok=True)
        raise

The helper keeps memory bounded and avoids leaving a final file in a partially written state. Add logging, hashes, permissions, and free-space checks according to your environment.

Conclusion

compression.zstd gives Python applications an efficient lossless compression option with simple APIs and advanced features. Use direct byte functions for small values, streaming for large files, dictionaries for repeated small records, and strict output limits for untrusted data. Combined with hashes, metrics, restore tests, and atomic writes, Zstandard can reliably support backups, logs, data pipelines, and network storage.