Python tokenize: Analyze Source Code

Linters, formatters, security scanners, editors, and educational tools often need to understand Python source code without executing it. The Python tokenize module converts raw source text into a stream of lexical units such as names, numbers, operators, strings, comments, logical newlines, and indentation markers. This layer sits between the original file and higher-level structures such as the abstract syntax tree.

In this guide, you will learn how to read tokens, preserve source encodings, inspect exact positions, find comments, reconstruct code, handle incomplete input, and combine tokenize with AST and symbol-table analysis. The examples complement our guides to inspect, symtable, dis, regular expressions, and Ruff.

What a token represents

A token is a lexical unit recognized by Python’s tokenizer. The keyword def, a function name, parentheses, a number, a string literal, and a comment are separate tokens. Unlike a plain text search, tokenization understands multiline strings, escaped characters, comments, indentation, and compound operators.

import tokenize
from io import BytesIO

source = b"count = 10  # initial value\n"
for item in tokenize.tokenize(BytesIO(source).readline):
    print(item)

The byte-based API is important because Python must determine the source encoding before decoding the file.

Understanding TokenInfo

Each generated item is a TokenInfo tuple. It contains the token type, the exact text, the starting position, the ending position, and the original physical line.

for item in tokenize.tokenize(BytesIO(source).readline):
    print(item.type, item.string, item.start, item.end)

Positions are represented as (line, column). Lines start at one, while columns start at zero. Editors and diagnostics can use these coordinates to underline the precise part of a file.

Readable token names

The numeric type can be mapped to a readable name through token.tok_name.

import token

for item in tokenize.tokenize(BytesIO(source).readline):
    print(token.tok_name[item.type], repr(item.string))

Common values include ENCODING, NAME, OP, NUMBER, STRING, COMMENT, NEWLINE, INDENT, DEDENT, and ENDMARKER.

Detecting source encoding

Python files may declare their encoding in one of the first two lines. detect_encoding() follows the same rules as the interpreter.

with open("application.py", "rb") as file:
    encoding, consumed_lines = tokenize.detect_encoding(file.readline)

print(encoding)

The official tokenize documentation explains how UTF-8 byte-order marks and encoding cookies are handled. A conflicting declaration raises an error instead of silently choosing a codec.

Opening Python files correctly

tokenize.open() detects the declared encoding and returns a text stream.

with tokenize.open("application.py") as file:
    source_text = file.read()

This is preferable to forcing UTF-8 when a tool must process older or mixed projects. For new projects, UTF-8 remains the best default, but analysis tools should still respect Python’s language rules.

Tokenizing an existing string

If the code is already decoded, generate_tokens() accepts a text-returning readline function.

from io import StringIO

source_text = "total = price * quantity\n"
for item in tokenize.generate_tokens(StringIO(source_text).readline):
    print(item)

This form does not emit the ENCODING token. Prefer the byte API for files and the string API when encoding has already been resolved.

Finding comments reliably

Comments are represented by COMMENT tokens, so you do not need to search for the hash character manually.

comments = []
for item in tokenize.generate_tokens(StringIO(source_text).readline):
    if item.type == token.COMMENT:
        comments.append((item.start, item.string))

A regular expression or basic string scan would mistake a hash inside a string for a comment. The tokenizer distinguishes those cases.

Collecting identifiers

NAME tokens can be collected for indexes, metrics, or teaching tools.

names = {
    item.string
    for item in tokenize.generate_tokens(StringIO(source_text).readline)
    if item.type == token.NAME
}

The set includes keywords such as def and return. Use the keyword module to exclude them.

import keyword

identifiers = {name for name in names if not keyword.iskeyword(name)}

This still does not tell you which scope owns a name. For semantic name resolution, combine the result with AST and symtable.

Indentation tokens

Python emits INDENT and DEDENT tokens when block levels change.

source_text = "def double(value):\n    return value * 2\n"
for item in tokenize.generate_tokens(StringIO(source_text).readline):
    if item.type in {token.INDENT, token.DEDENT}:
        print(token.tok_name[item.type], repr(item.string))

These tokens can support block visualizations, indentation statistics, and diagnostics. They are more reliable than counting leading spaces without understanding logical lines.

NEWLINE versus NL

NEWLINE ends a logical statement. NL is a physical line break that does not end the statement, usually because the expression continues inside parentheses, brackets, or braces.

values = [
    1,
    2,
]

The internal breaks produce NL tokens. Formatters and comment-preserving tools must distinguish these two cases.

Exact operator types

Most punctuation arrives with the general OP type. The exact_type property identifies the specific operator.

for item in tokenize.generate_tokens(StringIO("value += 1\n").readline):
    if item.type == token.OP:
        print(token.tok_name[item.exact_type])

This makes it possible to distinguish assignment, augmented assignment, arrows, delimiters, and arithmetic operators without maintaining your own string table.

Reconstructing source code

untokenize() rebuilds source from token information.

items = list(tokenize.tokenize(BytesIO(source).readline))
rebuilt = tokenize.untokenize(items)
print(rebuilt)

The documented guarantee concerns token equivalence: tokenizing the result should produce the same token types and strings. Exact spacing between tokens may change.

A simple token transformation

You can replace selected NAME tokens and untokenize the modified stream.

result = []
stream = StringIO("value = value + 1\n")
for item in tokenize.generate_tokens(stream.readline):
    if item.type == token.NAME and item.string == "value":
        item = item._replace(string="counter")
    result.append(item)

new_source = tokenize.untokenize(result)

This is a lexical replacement, not a complete refactoring. It can rename unrelated variables with the same spelling in different scopes. Production refactoring requires scope resolution, imports, attributes, and project-wide references.

Handling TokenError

Incomplete constructs can raise TokenError. Typical examples include an unclosed triple-quoted string or parentheses that remain open at the end of a file.

try:
    list(tokenize.generate_tokens(StringIO("text = '''open").readline))
except tokenize.TokenError as error:
    message, position = error.args
    print(message, position)

A good diagnostic should report the position, preserve the source, and avoid guessing an automatic fix.

Indentation failures

Some malformed indentation produces IndentationError. Tools should handle both error families.

try:
    items = list(tokenize.generate_tokens(StringIO(source_text).readline))
except (tokenize.TokenError, IndentationError) as error:
    print(f"Invalid source: {error}")

Tokenization is not full parsing

A source file may tokenize successfully and still contain invalid grammar. Use ast.parse() when you need syntax validation.

import ast

ast.parse(source_text)

Tokenization answers which lexical pieces exist. Parsing explains how those pieces form expressions, statements, functions, classes, and control flow.

Combining tokens with AST

AST nodes provide semantic structure but traditionally omit most comments and formatting details. Tokens retain comments and exact source spans. Formatters, codemods, documentation systems, and static analyzers frequently use both representations.

For name ownership and closure information, add symtable. For runtime objects, inspect is appropriate, but importing untrusted source introduces execution risk.

Security and resource limits

Tokenizing source does not execute it, which is safer than importing a module. It is not free from denial-of-service risks. Very large files or adversarial input can consume time and memory. Limit file size, line length, nesting, total project files, and processing time.

When changing files, write to a temporary location first. Tokenize and parse the result, run tests, and replace the original atomically. Never overwrite the only copy before validation.

Building a token report

from collections import Counter

def summarize(source_text):
    counts = Counter()
    reader = StringIO(source_text).readline
    for item in tokenize.generate_tokens(reader):
        counts[token.tok_name[item.type]] += 1
    return dict(counts)

The resulting dictionary can reveal the number of comments, strings, operators, names, logical lines, and indentation changes.

Command-line inspection

The module includes a command-line interface:

python -m tokenize application.py

Use -e to display exact operator token names. This is useful for learning the tokenizer and debugging transformations.

Common mistakes

• Finding comments with regular expressions and matching hashes inside strings.
• Ignoring the ENCODING token when working from bytes.
• Treating NL and NEWLINE as identical.
• Using lexical replacement as a scope-aware refactoring.
• Assuming untokenize preserves every space.
• Importing code when static token analysis is enough.
• Processing unlimited untrusted source.

Recommended practices

• Read source files as bytes with tokenize.tokenize.
• Use tokenize.open when you need decoded text.
• Map types through the token module.
• Use exact_type for punctuation and operators.
• Combine tokens, AST, and symtable according to the task.
• Validate transformed output before replacing files.
• Test Unicode, multiline strings, comments, continuations, and incomplete input.

Conclusion

The Python tokenize module provides a precise view of source code at the lexical level. It identifies names, numbers, strings, comments, operators, newlines, and indentation while preserving coordinates that are valuable for editors and diagnostics.

Use it for lightweight static analysis, comment extraction, metrics, teaching tools, and controlled source transformations. Combine it with AST and symtable for structure and scope. With encoding awareness, resource limits, careful error handling, and post-transformation validation, you can build dependable source-code tools without executing the files they inspect.