The stringprep module exposes the tables defined by RFC 3454 for preparing Unicode strings used in Internet protocols. Preparation can map or remove selected characters, normalize text, reject prohibited categories, and apply bidirectional rules before identifiers are compared, stored, or transmitted.
The module does not provide one universal cleaning function. It exposes table lookup and mapping functions. A protocol profile must define which tables apply, whether case folding is required, which normalization form to use, and how to handle unassigned characters.
A historical standards module
RFC 3454 supported profiles such as Nameprep in early internationalized domain-name systems. Its tables are based on Unicode 3.2, so recently added characters may appear unassigned in this historical model.
Do not invent a profile for authentication, domain names, or security-sensitive identifiers. Use the current standard and a dedicated implementation when available. stringprep is most useful for legacy interoperability, audits, and understanding protocol preparation.
Tables exposed as functions
The RFC tables are too large to expose as ordinary dictionaries, so Python provides characteristic functions for sets and mapping functions for transformations.
import stringprep
character = "\u00ad" # soft hyphen
print(stringprep.in_table_b1(character))Table B.1 includes characters commonly mapped to nothing. The lookup does not modify text; the application performs the transformation.
Unassigned code points
in_table_a1() reports whether a code point was unassigned in Unicode 3.2.
def has_unassigned(text):
return any(stringprep.in_table_a1(c) for c in text)A profile may reject these points so future assignments cannot change identifier meaning. However, modern characters may also be marked unassigned by this legacy table.
Map characters to nothing
def apply_b1(text):
return "".join(
c for c in text
if not stringprep.in_table_b1(c)
)Removing characters can make two different inputs converge. Detect collisions after the complete preparation and preserve the original value for display and auditing.
Case folding with B.2 and B.3
map_table_b2() provides case folding intended for use with NFKC. map_table_b3() provides a mapping used without normalization.
def map_b2(text):
return "".join(stringprep.map_table_b2(c) for c in text)A mapping result may contain multiple characters. Do not assume one input character produces one output character or validate length before transformation.
NFKC normalization
Many historical profiles apply NFKC after mapping.
import unicodedata
def prepare_base(text):
text = apply_b1(text)
text = map_b2(text)
return unicodedata.normalize("NFKC", text)NFKC removes compatibility distinctions and may change width variants, stylistic characters, and symbols. Apply it only when the protocol requires it and retain original display text.
ASCII and non-ASCII spaces
Tables C.1.1 and C.1.2 identify ASCII and non-ASCII spaces. in_table_c11_c12() checks their union.
special_spaces = [
c for c in text
if stringprep.in_table_c11_c12(c)
]A profile may map, prohibit, or selectively allow these characters. Do not replace every space automatically without the specification.
Control characters
Tables C.2.1 and C.2.2 cover ASCII and non-ASCII control characters.
def has_control(text):
return any(
stringprep.in_table_c21_c22(c)
for c in text
)Controls can manipulate logs, terminals, and protocol framing. Escape invisible characters in diagnostics even if a protocol permits some of them.
Private use, noncharacters, and surrogates
in_table_c3(), in_table_c4(), and in_table_c5() identify private-use points, noncharacters, and surrogate codes.
Reject them according to the profile and before serialization into systems that require valid Unicode scalar values.
Other prohibited categories
C.6 and C.7 identify characters inappropriate for plain text or canonical representation. C.8 covers characters that change display properties or are deprecated. C.9 covers tagging characters.
These categories demonstrate why strip() plus lower() is not protocol preparation.
Bidirectional rules
D.1 identifies characters with R or AL bidi properties. D.2 identifies L characters. RFC 3454 profiles usually impose special rules when right-to-left characters occur.
def validate_bidi(text):
has_randal = any(stringprep.in_table_d1(c) for c in text)
if not has_randal:
return True
if any(stringprep.in_table_d2(c) for c in text):
return False
return (
stringprep.in_table_d1(text[0])
and stringprep.in_table_d1(text[-1])
)This demonstrates the historical basic rule, but production code must follow the exact profile. Logical and visual order can differ, so security tools should display escaped representations.
A profile-oriented example
import unicodedata
class StringPrepError(ValueError):
pass
def prepare_legacy(text):
mapped = "".join(
"" if stringprep.in_table_b1(c)
else stringprep.map_table_b2(c)
for c in text
)
normalized = unicodedata.normalize("NFKC", mapped)
prohibited = (
stringprep.in_table_c12,
stringprep.in_table_c21_c22,
stringprep.in_table_c3,
stringprep.in_table_c4,
stringprep.in_table_c5,
stringprep.in_table_c6,
stringprep.in_table_c7,
stringprep.in_table_c8,
stringprep.in_table_c9,
)
for c in normalized:
if any(table(c) for table in prohibited):
raise StringPrepError(
f"prohibited character U+{ord(c):04X}"
)
if not validate_bidi(normalized):
raise StringPrepError("invalid bidi rule")
return normalizedThis is educational, not a complete named profile. The table set, A.1 policy, and exact order must come from the relevant standard.
Validate after transformation
Check character and byte limits after mapping and normalization because output length can change. If the prepared string becomes a unique database key, enforce uniqueness transactionally on the final value.
IDNA and domain names
Do not manually rebuild Nameprep for modern domain processing. Use an IDNA implementation matching the required standard. Domain handling includes label processing, Punycode, validation, and version-specific rules.
Passwords and usernames
Do not apply generic stringprep rules to passwords without a protocol specification. Mapping or removing characters can change the secret and reduce entropy. Modern protocols may use PRECIS profiles or other explicit rules.
For usernames, keep the original form and a separate prepared key. Monitor collisions and visually confusable identifiers.
Homographs
Case folding and NFKC do not merge every visually similar character. Latin, Greek, and Cyrillic characters may remain distinct. Sensitive public identifiers need script policies and specialized confusable detection.
Testing
Test control characters, non-ASCII spaces, B.1 removals, case-fold expansion, RTL text, mixed L/R text, empty input, and modern characters. Compare behavior with official protocol test vectors.
Version the preparation algorithm. Changing it for existing identifiers requires a migration and collision analysis.
Common mistakes
- Assuming the module provides a complete sanitizer.
- Creating a profile without a protocol standard.
- Ignoring the Unicode 3.2 basis.
- Checking size before mapping.
- Failing to detect collisions.
- Applying NFKC or removal rules to passwords arbitrarily.
- Ignoring bidi rules and homographs.
Recommended practices
- Use a protocol-specific library.
- Preserve original input.
- Version the preparation function.
- Follow the specified operation order.
- Validate length and uniqueness at the end.
- Escape invisible characters in logs.
- Test official vectors and adversarial Unicode.
Related guides
Continue with Python unicodedata, Python locale, Python textwrap, Python fnmatch, and Python contextvars.
See the official stringprep documentation and RFC 3454.
Conclusion
stringprep is a low-level toolbox for historical Unicode preparation profiles. It must be used under a defined specification, with awareness of Unicode 3.2, collisions, prohibited categories, and bidi rules. Prefer maintained protocol-specific implementations for modern systems.
