Formatters, syntax highlighters, style analyzers, and refactoring tools need to divide Python source into units such as names, numbers, strings, operators, comments, and indentation. The Python tokenize module provides a standard-library lexical scanner that preserves comments and positions, allowing tools to inspect or rewrite source code without executing it.
This guide covers tokenize(), generate_tokens(), untokenize(), detect_encoding(), and tokenize.open(). It complements our articles about bytecode with dis, symbol tables, interactive input with codeop, inspect, and text comparison.
What tokenization means
Tokenization groups characters into lexical units. In total = price * 2, for example, there are names, an assignment operator, a multiplication operator, a number, and a logical line ending.
The tokenizer does not build a complete syntax tree or resolve scopes. It preserves details useful for presentation and transformation, including comments, indentation text, and line-column coordinates.
Your first tokenize() example
tokenize.tokenize() accepts a readline callable that returns bytes.
from io import BytesIO
from tokenize import tokenize
source = b"total = price * 2 # calculation\n"
for item in tokenize(BytesIO(source).readline):
print(item)The stream includes tokens such as ENCODING, NAME, OP, NUMBER, COMMENT, NEWLINE, and ENDMARKER.
The TokenInfo structure
Every item is a named tuple with these fields:
type: numeric token type;string: original token text;start: starting line and column;end: ending line and column;line: physical source line.
for item in tokenize(BytesIO(source).readline):
print(item.type, item.string, item.start, item.end)Lines start at 1 and columns at 0. Coordinates support editor highlighting, diagnostics, and source patches.
Readable token names
The token module maps numeric codes to names.
import token
for item in tokenize(BytesIO(source).readline):
print(token.tok_name[item.type], repr(item.string))tokenize reexports many constants, while the separate namespace makes the purpose clear.
OP and exact_type
Operators and delimiters are returned with the generic OP type. The exact_type property distinguishes parentheses, colons, addition, multiplication, and other symbols.
for item in tokenize(BytesIO(b"x += 1\n").readline):
print(
token.tok_name[item.type],
token.tok_name[item.exact_type],
item.string,
)The official tokenize documentation recommends exact_type when a tool needs the precise operator.
Comments are preserved
Unlike several parser interfaces, tokenize returns comments.
from tokenize import COMMENT
comments = [
item
for item in tokenize(BytesIO(source).readline)
if item.type == COMMENT
]
for comment in comments:
print(comment.start, comment.string)This enables formatters, directive extractors, special-comment checkers, and documentation tools.
INDENT and DEDENT
Block structure appears as INDENT and DEDENT tokens.
code = b"if active:\n run()\nfinish()\n"
for item in tokenize(BytesIO(code).readline):
print(token.tok_name[item.type], repr(item.string))INDENT carries the original whitespace. DEDENT normally has an empty string and signals a return to a previous level.
NEWLINE versus NL
NEWLINE completes a logical statement. NL is a physical line break that does not complete the statement, such as inside parentheses or on a comment-only line.
code = b"result = (\n 1 +\n 2\n)\n"
for item in tokenize(BytesIO(code).readline):
if item.type in {token.NEWLINE, token.NL}:
print(token.tok_name[item.type], item.start)The distinction is essential for formatters that rearrange physical lines without changing meaning.
Read Unicode strings with generate_tokens()
generate_tokens() accepts a callable returning str instead of bytes.
from io import StringIO
from tokenize import generate_tokens
text_source = "message = 'hello'\n"
for item in generate_tokens(StringIO(text_source).readline):
print(item)This API does not yield an ENCODING token. It is convenient when text has already been decoded correctly.
Detect source encoding
detect_encoding() reads at most two lines to find a UTF-8 BOM or a PEP 263 encoding cookie.
from tokenize import detect_encoding
with open("program.py", "rb") as file:
encoding, lines = detect_encoding(file.readline)
print(encoding)
print(lines)If a BOM and cookie disagree, the function raises SyntaxError. Without a declaration, UTF-8 is used.
Open Python source correctly
tokenize.open() applies the same encoding detection and returns a text-mode file.
import tokenize
with tokenize.open("program.py") as file:
content = file.read()Prefer it when inspecting Python files whose encoding is not already known.
Rebuild source with untokenize()
untokenize() accepts token-type and token-string pairs and reconstructs Python source.
from tokenize import untokenize
pairs = [
(item.type, item.string)
for item in tokenize(BytesIO(source).readline)
]
rebuilt = untokenize(pairs)
print(rebuilt.decode("utf-8"))The round-trip guarantee preserves token types and strings, but spacing and column positions may change.
Transform numeric literals lexically
A tool can replace numeric tokens while leaving strings and comments untouched.
from tokenize import NUMBER, NAME, OP, STRING
result = []
for item in tokenize(BytesIO(b"rate = 1.25\n").readline):
if item.type == NUMBER and "." in item.string:
result.extend([
(NAME, "Decimal"),
(OP, "("),
(STRING, repr(item.string)),
(OP, ")"),
])
else:
result.append((item.type, item.string))
print(untokenize(result).decode("utf-8"))A complete transformer must also add imports and consider exponent notation, complex numbers, and underscores.
Rename identifiers carefully
NAME tokens can be replaced, but a correct renamer must understand scope.
from tokenize import NAME
for item in tokenize(BytesIO(b"value = value + 1\n").readline):
if item.type == NAME and item.string == "value":
new_item = item._replace(string="counter")
else:
new_item = itemTokenize alone cannot determine whether a name is a variable, attribute, import, parameter, or pattern component. Combine it with AST and symtable for semantic refactoring.
Syntax highlighting
A highlighter can map token types to visual classes.
CLASSES = {
token.NAME: "name",
token.NUMBER: "number",
token.STRING: "string",
token.OP: "operator",
token.COMMENT: "comment",
}
for item in generate_tokens(StringIO(text_source).readline):
css_class = CLASSES.get(item.type, "other")
render(item.string, css_class)Keywords also arrive as NAME. Use the keyword module to distinguish them.
TokenError
TokenError is raised when a multiline string or delimited expression remains unfinished at end of file.
from tokenize import TokenError
try:
list(tokenize(BytesIO(b"items = [1, 2\n").readline))
except TokenError as error:
print("Incomplete source:", error)Other syntax mistakes may pass tokenization because lexical scanning does not replace parsing.
Syntactically invalid code
The documentation warns that the module is designed for source that would also be accepted by ast.parse(). Behavior for invalid code is undefined and may change.
Editors that process temporarily incomplete files need tolerant recovery, exception handling, and no reliance on an exact token stream after the invalid region.
Command-line usage
The module provides a simple command-line interface.
python -m tokenize program.pyThe -e option displays exact operator names.
python -m tokenize -e program.pyWithout a filename, source is read from standard input.
Positions and Unicode
Columns refer to positions in the decoded Python string, not necessarily byte offsets in the original file. Tools that patch bytes need an encoding-aware mapping.
Also test tabs, combining characters, non-ASCII identifiers, and different line endings across platforms.
Preserving original formatting
untokenize() does not promise identical spacing. That is acceptable for a formatter, but a minimal code fix may need to edit the original text ranges directly.
Use token coordinates to build patches, apply edits from the end toward the beginning, and validate the result with ast.parse().
Security and resource limits
Tokenizing does not execute source, which is safer than importing a module. An enormous or adversarial file can still consume substantial CPU and memory.
Limit source size, processing time, and token count. Never conclude that source is safe merely because it was tokenized or parsed.
Structured token report
def report(source: bytes):
for item in tokenize(BytesIO(source).readline):
yield {
"type": token.tok_name[item.type],
"exact_type": token.tok_name[item.exact_type],
"text": item.string,
"start": item.start,
"end": item.end,
}The report can feed an interactive table, source viewer, or regression test.
Common mistakes
- Passing bytes to
generate_tokens(). - Ignoring the encoding token.
- Treating all operators only as generic
OP. - Confusing
NLandNEWLINE. - Expecting
untokenize()to preserve exact spacing. - Renaming names without scope analysis.
- Depending on behavior for invalid source.
- Using tokenization as a security check.
Best practices
- Use bytes and
tokenize()for complete files. - Open source with
tokenize.open(). - Read
exact_typefor operators. - Combine tokens with AST and symtable for semantics.
- Validate reconstructed source.
- Test comments, Unicode, tabs, and multiline constructs.
- Apply resource limits to untrusted inputs.
- Document which formatting details are preserved.
Conclusion
The Python tokenize module turns source code into a rich stream containing comments, indentation, encoding information, operators, and positions. It provides a strong foundation for syntax highlighters, formatters, style analyzers, and lexical transformations.
Its limitation is equally important: tokens describe lexical form, not complete meaning. By combining tokenize with AST, symtable, and post-transformation validation, tools can modify source precisely without executing it or confusing strings and comments with active syntax.
