Python gzip: Compress .gz Files

The Python gzip module reads and writes GZIP streams through an interface that resembles ordinary files. It uses DEFLATE through zlib and provides in-memory helpers, the GzipFile class, text wrappers, and a small command-line interface.

GZIP is widely supported by Unix tools, web servers, data pipelines, and distribution formats. It compresses one logical stream rather than acting as a multi-file container. To preserve a directory tree, applications commonly create a TAR archive and then compress that stream with GZIP.

When gzip is appropriate

Use GZIP for logs, JSON Lines, CSV exports, HTTP payloads, and backups containing one stream. For lower-level DEFLATE control, see Python zlib. When final size matters more than speed, compare it with Python lzma.

Compress bytes in one call

import gzip

data = ("access record\n" * 2000).encode("utf-8")
compressed = gzip.compress(data, compresslevel=6)
restored = gzip.decompress(compressed)

assert restored == data
print(len(data), len(compressed))

These helpers keep the complete input and output in memory. In Python 3.14, gzip.compress() defaults to mtime=0, making the result independent of the creation time. Pass mtime=None to store the current timestamp.

Create reproducible output

Reproducible builds require identical content to produce identical bytes. A varying header timestamp prevents that. The convenience function now favors deterministic output:

import gzip
import hashlib

artifact = gzip.compress(b"version=1\n", mtime=0)
print(hashlib.sha256(artifact).hexdigest())

When constructing GzipFile directly, set mtime=0 explicitly if this property matters.

Write compressed text

import gzip

with gzip.open("events.log.gz", "wt", encoding="utf-8", compresslevel=6) as file:
    file.write("service started\n")
    file.write("job completed\n")

Text modes create an io.TextIOWrapper. Specify the encoding instead of relying on the machine default. Use xt when an existing destination should make the operation fail.

Read line by line

import gzip

with gzip.open("events.log.gz", "rt", encoding="utf-8") as file:
    for number, line in enumerate(file, 1):
        process(number, line.rstrip())

Iteration avoids materializing the entire text. It complements the techniques in the giant-file guide. Applications should still limit line count and total expanded bytes.

Compress an existing file

import gzip
import shutil

with open("data.csv", "rb") as source:
    with gzip.open("data.csv.gz", "wb", compresslevel=6) as target:
        shutil.copyfileobj(source, target, length=1024 * 1024)

The transfer happens in chunks. For a production export, write to a temporary path, flush and close it, then rename it atomically so readers never see a partial file.

Use GzipFile with BytesIO

GzipFile accepts any binary file-like object, including an in-memory buffer:

import gzip
import io

buffer = io.BytesIO()
with gzip.GzipFile(fileobj=buffer, mode="wb", compresslevel=6, mtime=0) as gz:
    gz.write(b"content" * 1000)

payload = buffer.getvalue()

Closing GzipFile does not close the supplied fileobj. This lets code retrieve the buffer or append protocol data after the compressed member.

Header metadata

A GZIP header can include modification time and an original filename. When fileobj is supplied, the filename argument is used only for header metadata. Avoid leaking usernames, temporary paths, tenant names, or internal directory structures.

Concatenated members

gzip.decompress() supports multiple GZIP members concatenated into one byte string. Append mode also adds a new member to the end of a file. This is valid, but external tools may display or process members differently, so test every required consumer.

Handle invalid input

Malformed data can raise gzip.BadGzipFile, EOFError, or zlib.error.

import gzip
import zlib

try:
    with gzip.open("input.gz", "rb") as file:
        data = file.read()
except (gzip.BadGzipFile, EOFError, zlib.error) as exc:
    raise ValueError("invalid GZIP input") from exc

Log a safe identifier rather than raw content or sensitive paths.

Bound decompression

gzip.decompress() returns the full output and has no total-output parameter. Do not apply it blindly to large uploads. Read chunks through GzipFile and maintain a counter:

import gzip

maximum = 100 * 1024 * 1024
total = 0

with gzip.open("upload.gz", "rb") as file:
    while chunk := file.read(64 * 1024):
        total += len(chunk)
        if total > maximum:
            raise ValueError("expanded content exceeded the limit")
        consume(chunk)

Also limit compressed size, processing time, and member count. GZIP checks detect accidental corruption; they do not authenticate the source.

Compression levels

• 0: store without compression.
• 1: fastest, larger output.
• 6: common balance and CLI default.
• 9: slowest attempt at best compression.

Already-compressed images, video, and many PDF files typically gain little. Benchmark representative content instead of assuming level 9 is best.

GZIP is not a directory archive

GZIP represents a stream. To bundle filenames, permissions, and directories, create TAR first. ZIP combines container and compression; the practical differences are covered in the Python ZIP guide.

Command-line interface

Run python -m gzip to compress or decompress files. The CLI keeps input files and supports --fast, --best, and --decompress. The programmatic API is preferable when paths, errors, limits, and atomic replacement require strict control.

Testing strategy

Test empty input, Unicode encoded to bytes, truncated streams, concatenated members, several levels, deterministic headers, and output limits.

import gzip

def test_reproducible_gzip():
    first = gzip.compress(b"abc", mtime=0)
    second = gzip.compress(b"abc", mtime=0)
    assert first == second
    assert gzip.decompress(first) == b"abc"

Best practices

• Use context managers.
• Specify mode and encoding.
• Set mtime=0 for reproducible artifacts.
• Write output atomically.
• Limit input, output, members, and CPU.
• Keep sensitive names out of headers.
• Test interoperability.
• Do not confuse integrity checks with authentication.

Conclusion

Python gzip is a practical choice for widely compatible compressed streams. It supports memory operations, files, text, and arbitrary buffers, while Python 3.14 makes deterministic one-shot output the default.

Read the official gzip documentation and RFC 1952. To manage levels per environment, see Python configparser.