The unicodedata module exposes the Unicode Character Database compiled into the running Python version. Applications can inspect character names, categories, numeric values, bidirectional classes, combining marks, decompositions, and normalization forms. These features are essential for search, imports, identifiers, slugs, comparison, validation, and internationalization.
This guide explains the main APIs and shows how to normalize text without destroying information. Python 3.14.6 documents Unicode Character Database version 16.0.0.
Check the Unicode database version
import unicodedata
print(unicodedata.unidata_version)The version can affect names, categories, and recognized characters. Systems that persist normalized keys should record their Python version and test upgrades.
Get a character name
name() returns the official Unicode name. Some characters have no assigned name, so provide a default for arbitrary input.
print(unicodedata.name("½"))
print(unicodedata.name("\uFFFF", "UNNAMED"))Names are useful for diagnostics and reports. They are standardized identifiers, not localized labels for end users.
Look up a character by name
lookup() performs the reverse operation and raises KeyError when the name is unknown.
character = unicodedata.lookup("LEFT CURLY BRACKET")
print(character)The function also supports aliases and named sequences present in the database. Validate user-provided names and handle failures.
General categories
category() returns a two-letter code. The first letter represents a broad group such as letters, marks, numbers, punctuation, symbols, separators, controls, or unassigned characters.
for character in ["A", "a", "9", "!", " "]:
print(character, unicodedata.category(character))Examples include Lu for uppercase letters, Ll for lowercase letters, Nd for decimal digits, and Zs for space separators. Categories alone are not a complete identifier-security policy.
decimal, digit, and numeric values
Unicode distinguishes several numeric concepts. decimal() handles decimal digits, digit() includes other digit forms, and numeric() covers values such as fractions and numerals.
print(unicodedata.decimal("٩"))
print(unicodedata.digit("⁹"))
print(unicodedata.numeric("½"))Define whether an input field accepts ASCII digits only, international decimal digits, or any Unicode numeric character.
Combining characters
combining() returns the canonical combining class. Zero usually means the character is not a combining mark with a defined class.
text = "a\u0301"
for char in text:
print(repr(char), unicodedata.name(char), unicodedata.combining(char))The sequence contains a letter and a separate acute accent. It can look identical to a precomposed á while having different length and bytes.
Why normalization matters
Unicode allows canonically equivalent representations. Without normalization, comparisons, database keys, caches, and search indexes may treat visually identical text as different.
a = "café"
b = "cafe\u0301"
print(a == b)
print(unicodedata.normalize("NFC", a) == unicodedata.normalize("NFC", b))Normalization does not solve case differences, punctuation, language rules, or visually confusable characters.
NFC and NFD
NFD applies canonical decomposition. NFC decomposes and then recomposes characters when a precomposed form exists.
nfd = unicodedata.normalize("NFD", "action")
nfc = unicodedata.normalize("NFC", nfd)NFC is a common storage and comparison choice for human text. NFD is useful when analyzing combining marks or building an accent-insensitive auxiliary key.
NFKC and NFKD
Compatibility forms may replace stylistic or historical variants with simpler equivalents.
print(unicodedata.normalize("NFKC", "Ⅳ"))
print(unicodedata.normalize("NFKC", "Ｆｕｌｌ"))Compatibility normalization can lose meaningful distinctions. Use it for search or identifiers only after defining a domain policy. Do not apply it blindly to passwords, signatures, legal text, or content that must preserve typography.
Check whether text is normalized
is_normalized() tests NFC, NFD, NFKC, or NFKD.
text = "café"
if not unicodedata.is_normalized("NFC", text):
text = unicodedata.normalize("NFC", text)Direct normalization is often simple enough, while the check is useful for audits and metrics.
Removing accents carefully
A common pattern decomposes text and removes nonspacing marks. This is not universal transliteration.
def remove_accents(text):
decomposed = unicodedata.normalize("NFD", text)
filtered = "".join(
char for char in decomposed
if unicodedata.category(char) != "Mn"
)
return unicodedata.normalize("NFC", filtered)Use the result as an auxiliary search key, not as a replacement for original content. Many letters do not reduce correctly with this method.
Character decomposition
decomposition() returns hexadecimal code points and sometimes a compatibility tag.
print(unicodedata.decomposition("Ã"))
print(unicodedata.decomposition("①"))This is useful for diagnostics. For normal transformation, normalize() is usually the right API.
Bidirectional text
bidirectional() returns a character’s bidirectional class. mirrored() identifies symbols that may be mirrored in right-to-left layout.
print(unicodedata.bidirectional("٧"))
print(unicodedata.mirrored(">"))These properties do not replace a full layout engine. Bidirectional controls can make code, logs, and file names visually misleading, so security tools should offer escaped views.
East Asian width
east_asian_width() categorizes characters as narrow, wide, fullwidth, halfwidth, ambiguous, or neutral.
for char in "A界Ｆ":
print(char, unicodedata.east_asian_width(char))The property helps estimate display width, but combining marks, emoji sequences, and terminal policy still matter.
Casefolding and normalization
For caseless matching, normalize and apply casefold().
def search_key(text):
return unicodedata.normalize("NFKC", text).casefold()
print(search_key("Straße") == search_key("STRASSE"))The order and normalization form depend on the domain. Preserve the original and do not treat a simplified key as proof of identity.
Confusable characters and security
Normalization does not merge every visually similar character. Latin, Greek, and Cyrillic letters can remain distinct while looking alike. Homograph attacks affect usernames, domains, package names, and identifiers.
Use script restrictions, specialized confusable detection, and human review for sensitive contexts.
Preparing a search key
def prepare_search(text):
text = unicodedata.normalize("NFKC", text)
text = text.casefold()
return " ".join(text.split())Version this transformation as part of the search-index schema because Unicode database upgrades may change results.
Common mistakes
- Comparing text without a chosen normalization form.
- Using NFKC where original distinctions matter.
- Replacing original content with an accent-stripped copy.
- Treating every numeric character as an ASCII digit.
- Expecting normalization to prevent homographs.
- Using East Asian width as a complete display-width calculation.
- Ignoring the Unicode database version.
Recommended practices
- Always preserve original input.
- Define normalization per field and purpose.
- Normalize both sides of a comparison.
- Version derived keys and indexes.
- Test relevant languages and scripts.
- Use
casefold()for caseless matching. - Apply additional controls to sensitive identifiers.
Related guides
Continue with Python textwrap, Python locale, Python fnmatch, Python pydoc, and Python linecache.
See the official unicodedata documentation and the Unicode HOWTO.
Conclusion
unicodedata enables explicit, reproducible handling of international text. Normalization improves search and comparison, but it does not replace language, identity, security, or presentation policies. Preserve the source and choose transformations according to the domain.
