Python codecs: Master Encodings

The Python codecs module exposes the encoding registry, generic encode and decode functions, incremental processors, stream reader and writer base classes, byte-order-mark constants, and mechanisms for custom codecs and error handlers.

Most applications do not need to import codecs to open a UTF-8 file. The built-in open() function and the io module are the recommended interfaces. The module becomes valuable when code must inspect the registry, process partial byte sequences, transcode streams, use non-text transformations, or implement an encoding.

Text and bytes are different

A Python string contains Unicode code points. Files and network protocols carry bytes, so text must be encoded and bytes must be decoded:

text = "Hello, Python"
data = text.encode("utf-8")
restored = data.decode("utf-8")

assert restored == text

The encoding name is part of the protocol. Decoding UTF-8 bytes as Latin-1 may produce plausible but corrupted text without raising an exception. Do not silently guess when the producer should declare the encoding.

codecs.encode and codecs.decode

import codecs

utf8_bytes = codecs.encode("price €", "utf-8", errors="strict")
text = codecs.decode(utf8_bytes, "utf-8", errors="strict")

print(utf8_bytes)
print(text)

These functions use the registry and can also invoke transformations unavailable through str.encode() and bytes.decode(). For ordinary text encodings, the object methods remain simpler.

Inspect the codec registry

import codecs

info = codecs.lookup("utf-8")
print(info.name)
print(info.encode)
print(info.decode)
print(info.incrementalencoder)
print(info.incrementaldecoder)

lookup() resolves aliases and returns a CodecInfo. An unknown name raises LookupError. Validate a configurable encoding during application startup instead of waiting for the first file operation.

Aliases and normalized names

Names such as utf-8, UTF_8, and utf8 normally resolve to the same codec. Spaces and hyphens are normalized. Still, use consistent, documented names. CPython has optimized paths for a limited set of common aliases, while unusual spellings may miss those optimizations.

Use open for text files

from pathlib import Path

path = Path("report.txt")

with path.open("w", encoding="utf-8", newline="\n") as file:
    file.write("line 1\nline 2\n")

with path.open("r", encoding="utf-8") as file:
    content = file.read()

codecs.open() is deprecated since Python 3.14 and has been superseded by open(). The modern API integrates buffering, newline handling, and the io text stack.

Migrate codecs.open

# Old
import codecs
file = codecs.open("data.txt", "r", encoding="utf-8")

# Current
file = open("data.txt", "r", encoding="utf-8")

Review newline behavior during migration. When codecs.open() receives an encoding, the underlying file is opened in binary mode and does not perform automatic newline conversion. Test files produced on Windows and Unix.

Error handling

The default strict policy raises UnicodeEncodeError or UnicodeDecodeError. It is the safest choice when data loss is unacceptable.

data = b"name: Jos\xe9"

try:
    text = data.decode("utf-8", errors="strict")
except UnicodeDecodeError as exc:
    print(exc.start, exc.end, exc.reason)

Common handlers include:

• strict: raise an exception.
• replace: insert a replacement character or question mark.
• ignore: discard malformed data silently.
• backslashreplace: preserve information as escape sequences.
• surrogateescape: round-trip undecodable operating-system bytes.
• xmlcharrefreplace: emit numeric references while encoding.
• namereplace: emit Unicode names in escapes.

Avoid errors ignore

errors="ignore" can silently remove letters, separators, digits, or security-relevant characters. It turns corrupted input into apparently valid data. Use it only when loss is intentional, measurable, and cannot change meaning.

Preserve unknown filesystem bytes

data = b"file_\xff.txt"
text = data.decode("utf-8", errors="surrogateescape")
restored = text.encode("utf-8", errors="surrogateescape")
assert restored == data

The intermediate string contains surrogate code points. Its purpose is to return the original bytes to the same system interface; do not send it casually to JSON, databases, or user interfaces.

Register an error handler

import codecs
import logging

logger = logging.getLogger(__name__)

def log_and_replace(exc):
    logger.warning("encoding failure from %d to %d", exc.start, exc.end)
    return ("?", exc.end)

codecs.register_error("app_log_replace", log_and_replace)
result = "price: €".encode("ascii", errors="app_log_replace")

A handler must advance the position or it can create an infinite loop. Avoid logging sensitive source content, and prefix custom names to prevent collisions.

Incremental decoding

A UTF-8 character may be split across network or file chunks. Decoding each chunk independently can fail. An incremental decoder buffers incomplete sequences:

import codecs

decoder = codecs.getincrementaldecoder("utf-8")(errors="strict")
parts = []

for chunk in receive_chunks():
    parts.append(decoder.decode(chunk, final=False))

parts.append(decoder.decode(b"", final=True))
text = "".join(parts)

The final call flushes buffered state and raises an error if the stream ends with an incomplete sequence.

Incremental encoding

import codecs

encoder = codecs.getincrementalencoder("utf-16")()
output = bytearray()

for piece in generate_text():
    output.extend(encoder.encode(piece, final=False))

output.extend(encoder.encode("", final=True))

Stateful encodings may emit markers or preserve state between calls. Do not instantiate a new encoder for every fragment.

iterencode and iterdecode

import codecs

text_chunks = ["Hello ", "world", "!\n"]
for chunk in codecs.iterencode(text_chunks, "utf-8"):
    send(chunk)

byte_chunks = [b"Ol\xc3", b"\xa1 mundo"]
text = "".join(codecs.iterdecode(byte_chunks, "utf-8"))

iterencode() requires strings, while iterdecode() requires bytes. Some binary or text-to-text transformations are not compatible with these helpers.

Large text files

The text object returned by open() already uses an incremental decoder. Iterating over lines avoids loading everything. The techniques from the giant-file guide still apply: chunking, limits, streaming, and incremental output.

BOM in UTF-16 and UTF-32

UTF-16 and UTF-32 can use a byte order mark to identify endianness. The module exposes constants such as BOM_UTF16_LE, BOM_UTF16_BE, BOM_UTF32_LE, and BOM_UTF32_BE.

import codecs

data = codecs.BOM_UTF16_LE + "text".encode("utf-16-le")
print(data.startswith(codecs.BOM_UTF16_LE))

The utf-16 codec interprets or emits a BOM. The explicit utf-16-le and utf-16-be variants define byte order directly and should not rely on an implicit mark.

UTF-8 with a signature

UTF-8 has no byte-order problem, but some software writes the bytes EF BB BF as a signature. The utf-8-sig codec removes it while reading and writes it at the beginning:

from pathlib import Path

text = Path("spreadsheet.csv").read_text(encoding="utf-8-sig")

Use it when interoperating with software that expects a BOM. For new protocols, prefer plain UTF-8 and declare the encoding externally.

Encoding detection is uncertain

There is no perfect algorithm for identifying arbitrary byte encodings. Many byte sequences are valid in several code pages. Prefer metadata, headers, supplier contracts, or validated configuration. Heuristics should report confidence and allow review rather than silently changing critical data.

Transcode files

from pathlib import Path

source = Path("legacy.txt")
target = Path("new.txt")

with source.open("r", encoding="latin-1", errors="strict") as input_file:
    with target.open("w", encoding="utf-8", newline="") as output_file:
        for line in input_file:
            output_file.write(line)

Transcoding means decode and then encode. Do not replace bytes directly. Write to a temporary file and rename after success when readers must not see partial output.

Custom codecs

codecs.register() adds a search function that receives a normalized name and returns CodecInfo or None. unregister(), added in Python 3.10, removes it and clears the registry cache.

A complete codec may define stateless functions, incremental classes, and stream factories. Implement one only for a real shared format; an ordinary function is simpler for a private transformation.

Binary transformations in the registry

import codecs

encoded = codecs.encode(b"data", "base64_codec")
restored = codecs.decode(encoded, "base64_codec")

The registry includes bytes-to-bytes codecs such as Base64, hex, bz2, and zlib. For application Base64, Python base64 is clearer and supports strict validation. Use the dedicated compression modules for compression.

Encoding is not a binary layout

A text encoding converts characters into bytes. It does not define fields, numbers, or headers. Use Python struct for versioned binary records with explicit endianness and lengths.

Configurable encodings

When an encoding comes from configuration, validate it with codecs.lookup() at startup. Python configparser can load the name, but use an allowlist when only text encodings are supported because the registry also contains binary transformations.

Security and limits

• Limit source bytes and resulting text.
• Use strict for critical data.
• Avoid automatic detection without metadata.
• Keep surrogate values out of external formats.
• Finalize incremental decoders.
• Do not log sensitive malformed content.
• Allowlist encodings.
• Limit untrusted IDNA and Punycode inputs because processing can scale poorly.

Testing strategy

Test ASCII, accents, emoji, unsupported characters, UTF-8 sequences split across chunks, incomplete final chunks, correct and reversed BOMs, UTF-8 signatures, empty files, Windows and Unix newlines, error handlers, aliases, round trips, and exact expected bytes.

Best practices

• Use UTF-8 for new formats.
• Declare encoding in the protocol.
• Use open() instead of codecs.open().
• Prefer strict and handle failures.
• Use incremental decoders for manual chunks.
• Validate registry names.
• Keep application text as str and transport as bytes.
• Transcode in a streaming pipeline.

Conclusion

Python codecs is the infrastructure connecting encoding names, functions, streams, error policies, and incremental processing. It is especially useful in libraries, protocols, and interoperability work; ordinary text files should continue to use open().

Read the official codecs documentation and the Unicode Standard. The correct encoding should come from a contract, not trial and error.