Python bz2: Compress with bzip2

The Python bz2 module compresses and decompresses data with the bzip2 algorithm without requiring a third-party package. It supports small in-memory operations, compressed files, text wrappers, and incremental streams that process data in chunks.

Bzip2 often produces smaller results than gzip for repetitive text, although it usually needs more CPU. The correct choice depends on file size, speed, compatibility, and how often the content will be opened. This guide explains the main APIs and the safeguards needed when compressed data comes from outside your application.

When bz2 is a good choice

Use bz2 when an existing workflow expects .bz2, when good compression matters more than maximum speed, or when exchanging logs and datasets with Unix tools. For broadly distributed archives, ZIP or gzip may be more convenient. For a lower-level DEFLATE interface, read the guide to Python zlib.

One-shot compression in memory

bz2.compress() accepts bytes-like data and returns compressed bytes. Compression levels range from 1 to 9. Higher values favor size but may take longer.

import bz2

data = ("report row\n" * 1000).encode("utf-8")
compressed = bz2.compress(data, compresslevel=9)
restored = bz2.decompress(compressed)

assert restored == data
print(len(data), len(compressed))

This is easy, but the original and compressed values coexist in memory. For very large inputs, use the file interface or incremental objects. The article about reading giant files in Python shows complementary techniques for controlling memory.

Write a .bz2 text file

bz2.open() resembles the built-in open(). Binary modes accept bytes. Text modes use an io.TextIOWrapper, so specify an encoding.

import bz2

with bz2.open("report.csv.bz2", "wt", encoding="utf-8") as file:
    file.write("product,quantity\n")
    file.write("keyboard,12\n")
    file.write("monitor,4\n")

Mode wt overwrites the destination. Use xt when accidental replacement must fail. Append mode creates another compressed stream at the end; BZ2File can read concatenated streams transparently.

Read line by line

import bz2

with bz2.open("report.csv.bz2", "rt", encoding="utf-8") as file:
    for number, line in enumerate(file, start=1):
        print(number, line.rstrip())

Iteration avoids loading the full text at once. Handle UnicodeDecodeError when the encoding is uncertain. Avoid using errors="ignore" by default because silent character loss can corrupt identifiers and values.

Use BZ2File directly

BZ2File works in binary mode and accepts a path or an existing file object. It implements most of io.BufferedIOBase.

from bz2 import BZ2File
from pathlib import Path

path = Path("payload.bin.bz2")
with BZ2File(path, "wb", compresslevel=7) as target:
    target.write(b"header\x00")
    target.write(b"content" * 500)

The class supports context management, iteration, seeking when available, read1(), and readinto(). A shared instance is not safe for simultaneous readers or writers. Give each worker its own object or protect access with synchronization.

Incremental compression

BZ2Compressor is useful when data arrives in chunks. A call to compress() may return bytes or an empty result because the compressor buffers information internally. Finish the stream with flush().

import bz2

compressor = bz2.BZ2Compressor(compresslevel=6)
parts = []

for chunk in generate_chunks():
    output = compressor.compress(chunk)
    if output:
        parts.append(output)

parts.append(compressor.flush())
result = b"".join(parts)

The compressor cannot be used after flush(). Create a new object for every independent stream.

Bound incremental decompression

BZ2Decompressor.decompress() accepts max_length. It limits how much output a single call can return, which helps protect services from unexpectedly large expansion.

import bz2

decoder = bz2.BZ2Decompressor()
output = bytearray()
maximum = 20_000_000

for chunk in receive_chunks():
    pending = chunk
    while pending or not decoder.needs_input:
        part = decoder.decompress(pending, max_length=64 * 1024)
        pending = b""
        output.extend(part)
        if len(output) > maximum:
            raise ValueError("decompressed content exceeded the limit")
        if decoder.eof:
            break

When needs_input is false, more output is available without new compressed input. unused_data contains bytes found after the stream end. Unlike bz2.decompress() and BZ2File, one incremental decompressor does not automatically continue through concatenated streams.

Verify stream completion

Do not treat partial output as proof that the file is valid. Check eof to distinguish a complete stream from truncated content. Handle OSError, EOFError, and application-level validation failures.

Protect against hostile compressed data

Compression is not authentication. Limit compressed bytes, expanded bytes, CPU time, and the number of processing stages. Store files in a controlled directory, normalize supplied names, and do not trust extensions. When an archive contains multiple paths, apply the path-safety practices from the guide to creating ZIP files with Python.

Choose a practical compression level

Level 9 is the default, not a universal optimum. Benchmark representative data. Levels 5 or 6 may be better for frequent pipelines. JPEG, MP4, and many PDF files are already compressed, so running bzip2 over them often adds CPU with little size reduction.

Compare common algorithms

• gzip: usually faster and widely supported.
• bz2: often effective for text and archival datasets.
• lzma: can produce smaller output with higher cost.
• zlib: useful for protocols and raw DEFLATE streams.

For directory backups, a container such as TAR groups paths before compression. Keep serialization, archiving, and compression as separate design decisions.

Test round trips and failures

A core test compresses, decompresses, and compares the exact original bytes. Include empty data, Unicode encoded as bytes, random input, repetitive input, truncated streams, several compression levels, and concatenated members.

import bz2
import pytest

@pytest.mark.parametrize("data", [b"", b"abc", b"x" * 100_000])
def test_round_trip(data):
    assert bz2.decompress(bz2.compress(data)) == data

Operational checklist

• Specify encodings in text mode.
• Use context managers for files.
• Prefer chunked processing for large data.
• Call compressor flush() exactly once.
• Check eof for partial input.
• Limit total expanded output.
• Do not share one file object across threads.
• Benchmark with real payloads.

Conclusion

Python bz2 provides one-shot functions, compressed files, text handling, and incremental classes for bzip2. The API is approachable, but reliable systems must account for memory, stream boundaries, concatenated members, and untrusted expansion.

Consult the official bz2 documentation and the bzip2 project. For application settings around compression jobs, see Python configparser, and for execution diagnostics use Python trace.