The tokenize module converts Python source code into a sequence of lexical tokens. Each token describes a fragment such as a name, number, string, operator, comment, indentation change, newline, or end marker. This layer is useful for formatters, linters, converters, documentation tools, comment analysis, and small transformations that must preserve more textual detail than an AST.
Tokens do not represent the complete meaning of a program. They do not resolve scopes, imports, types, or runtime behavior. Use ast for syntax structure, symtable for symbol scopes, and dis for bytecode. Choose tokenize when whitespace, comments, positions, and literal spelling matter.
Tokenize bytes
tokenize.tokenize() accepts a readline function that returns bytes.
from io import BytesIO
from tokenize import tokenize
source = b"x = 10 + 2\n"
for item in tokenize(BytesIO(source).readline):
print(item)
The sequence includes an ENCODING token, source tokens, and an ENDMARKER.
TokenInfo
Each result is a TokenInfo object with type, string, start, end, and line fields.
for item in tokenize(BytesIO(source).readline):
print(item.type, item.string, item.start, item.end)
start and end are line-column pairs. line contains the original physical source line.
Readable token names
Token types are integers defined by the token module. Use token.tok_name for readable output.
import token
name = token.tok_name[item.type]
Do not persist raw numeric token IDs as a long-term format. They are tied to Python’s implementation and version.
exact_type for operators
Operators and delimiters generally use the broad OP type. TokenInfo.exact_type identifies the exact symbol.
import token
if item.type == token.OP:
print(token.tok_name[item.exact_type])
This distinguishes +, +=, parentheses, colons, and other operators.
generate_tokens for text
generate_tokens() accepts a function that returns strings, which is convenient after source has already been decoded.
from io import StringIO
from tokenize import generate_tokens
for item in generate_tokens(StringIO("x = 1\n").readline):
print(item)
This API does not produce an ENCODING token. For real Python files, prefer bytes or tokenize.open().
Source encoding
Python files may declare an encoding in their first lines. tokenize() detects it according to language rules.
Decoding every file as UTF-8 before detection can fail on legacy projects.
detect_encoding
detect_encoding() receives a byte-reading function and returns the encoding plus lines already consumed.
from tokenize import detect_encoding
with open("module.py", "rb") as file:
encoding, lines = detect_encoding(file.readline)
print(encoding, lines)
It may read up to two lines to inspect a BOM and encoding cookie.
tokenize.open
tokenize.open(filename) opens a Python file as text using the detected encoding.
import tokenize
with tokenize.open("module.py") as file:
source = file.read()
This is a safe default for source-analysis tools that need decoded text.
Comments
Unlike the AST, tokenization preserves comments as COMMENT tokens.
import token
comments = [
item for item in tokens
if item.type == token.COMMENT
]
This enables TODO detection, pragmas, type comments, tool directives, and documentation rules.
Do not treat every comment as a directive
A directive needs a defined prefix, valid positions, and parsing rules. Searching for a substring creates false positives.
Define syntax such as # tool: and validate the remaining fields.
NL versus NEWLINE
NEWLINE ends a logical statement. NL represents a physical line break that does not end the statement, such as one inside parentheses or on a blank line.
result = (
1
+ 2
)
Formatters and line-based analyzers must distinguish them.
Indentation
INDENT and DEDENT describe block changes.
if enabled:
run()
The INDENT token contains the actual whitespace. Inconsistent tabs and spaces can produce errors.
Indentation errors
Tokenization or compilation may raise IndentationError or TabError. Report the filename, position, and source context.
Do not automatically rewrite ambiguous indentation without a documented formatting policy.
String tokens
A complete literal usually appears as one STRING token, including prefixes and quote style.
r"path\file"
f"value={x}"
Details around f-strings and newer syntax can vary by Python version. Understanding expressions inside them may require the parser.
Numbers
Integers, floats, complex values, and different bases appear as NUMBER with their original spelling.
0xff
1_000_000
3.14e-2
This preserves underscores and style that an AST normally reduces to a value.
Names and keywords
Identifiers and language keywords are both NAME tokens. Use keyword.iskeyword() to distinguish standard keywords.
import keyword
if item.type == token.NAME and keyword.iskeyword(item.string):
print("keyword", item.string)
Soft keywords depend on syntactic context and require parser knowledge.
untokenize
untokenize() reconstructs source from tokens.
from tokenize import untokenize
new_source = untokenize(tokens)
It preserves a round trip for token type and string under supported conditions, but exact spacing can change.
A simple transformation
A tool can replace selected name tokens.
import token
from tokenize import TokenInfo, untokenize
new_tokens = []
for item in tokens:
if item.type == token.NAME and item.string == "old_name":
item = TokenInfo(
item.type, "new_name", item.start, item.end, item.line
)
new_tokens.append(item)
result = untokenize(new_tokens)
A correct rename must account for scopes, attributes, imports, and shadowing. Tokens alone do not solve semantics.
Type-string pairs
untokenize() also accepts (type, string) pairs and determines necessary spacing.
Keeping full TokenInfo preserves more context, but original positions become stale after edits.
Positions after edits
Changing token length invalidates later offsets. Reconstructed source is valid, but diagnostics based on original columns need remapping.
Editorial tools should maintain a source map between old and new text.
Preserving comments
AST transformations and ast.unparse() generally lose comments. Tokens preserve them, but large structural refactoring becomes difficult.
For faithful round trips, consider a concrete syntax tree library.
TokenError
TokenError is raised for cases such as unfinished multiline strings or unclosed brackets.
from tokenize import TokenError
try:
tokens = list(tokenize(readline))
except TokenError as error:
print("incomplete source", error)
Temporarily incomplete code is normal in editors, so handle the condition without crashing the full analysis.
ERRORTOKEN
Invalid characters and selected special cases may appear as ERRORTOKEN. Inspect text and position before deciding the diagnostic.
Some whitespace can appear in special contexts, so do not label every error token as hostile input.
Partial editor buffers
An IDE may tokenize while a user types. Add debounce, cancellation, and partial results.
Source execution is never required to generate tokens.
Large files
Converting the generator into a list uses memory proportional to file size. Process tokens as a stream whenever possible.
Transformations that need lookahead can use a bounded window.
Abuse limits
Untrusted input can contain huge files, extremely long lines, and deep nesting. Limit bytes, lines, token count, and processing time.
Public services should isolate expensive analysis in a separate process.
Unicode identifiers
Python supports Unicode identifiers. Do not assume ASCII when validating names.
Security rules can normalize and detect confusing characters while still supporting legitimate languages.
Invisible characters
A scanner may report controls, unusual spaces, or bidirectional characters in comments and strings. Distinguish legitimate text from risk.
Display code points and source positions in diagnostics.
Combine with AST
Use tokens for comments and spelling and AST for structure. A common workflow associates nodes with line ranges and locates matching tokens.
See Python ast for visitors and structural transformations.
Combine with symtable
Tokens reveal spelling; symtable identifies scopes, locals, globals, and free variables.
This combination is safer for renaming than replacing every NAME token.
Formatters
A formatter must understand tokens, syntax, comments, and layout rules. Adding spaces around every OP is insufficient.
Use a complete grammar and a large regression suite.
Comment linters
Tokenization is enough for rules such as overly long comments, TODOs without owners, or invalid pragmas.
A # inside a string is not a COMMENT token.
Secret detection
Tokens can help find strings assigned to names such as password or token, but lexical analysis creates false positives.
Never log a detected value. Report only location and rule type.
Command line
The module provides a CLI that displays tokens from a file.
python -m tokenize module.py
It is useful for learning and debugging analysis rules.
Testing
Include alternate encodings, BOM, comments, multiline strings, f-strings, tabs, blank lines, open brackets, Unicode, incomplete files, and very large input.
After a transformation, tokenize and compile the generated source again.
Common mistakes
Common failures include decoding everything as UTF-8, confusing NL and NEWLINE, treating keywords as a separate token type, renaming NAME tokens without scope analysis, losing comments after switching to AST, trusting stale positions, materializing huge token lists, and omitting limits for public input.
Conclusion
tokenize exposes the lexical form of Python source while preserving comments, spelling, lines, and operators. Use tokenize.open() for correct encoding, exact_type for operators, and untokenize() for controlled reconstruction.
Combine tokens with AST and symbol tables when transformations need semantic context. Consult the official tokenize documentation and the token documentation.
