The ast module converts Python source code into an abstract syntax tree. Instead of treating a program as plain text, a tool works with nodes representing modules, functions, classes, calls, operators, names, literals, and control-flow structures. Linters, formatters, security scanners, automated migrations, documentation generators, and educational tools use this structural layer.
An AST describes syntax, not the complete runtime behavior of a program. Dynamic imports, reflection, monkey patching, descriptors, metaclasses, and data-dependent execution limit what static analysis can prove. Compiling or executing a modified tree also runs code with the same risks as any other Python program.
Parse source text
ast.parse() accepts source code and returns a Module node.
import ast
source = "result = add(2, 3)"
tree = ast.parse(source, filename="example.py", mode="exec")
print(type(tree).__name__)
The filename appears in errors and tracebacks. Supply the real path when known.
exec, eval, and single modes
mode="exec" parses a module containing statements. eval accepts one expression. single represents interactive input.
expression = ast.parse("1 + 2 * 3", mode="eval")
Using the wrong mode produces SyntaxError or a tree incompatible with the intended compilation mode.
Inspect with dump
ast.dump() creates a readable representation of a tree.
print(ast.dump(tree, indent=2, include_attributes=True))
include_attributes=True includes line and column positions, which are valuable for diagnostics and source editing.
Basic node structure
A module contains a body list. An assignment is an Assign node, a function call is Call, identifiers are Name, and ordinary literals are commonly represented by Constant.
assignment = tree.body[0]
print(type(assignment).__name__)
print(type(assignment.value).__name__)
Real tools should not depend on fixed list indexes. Traverse the tree and validate node types.
Load, Store, and Del context
Name, Attribute, and Subscript nodes carry a context showing whether the expression is being read, assigned, or deleted.
for node in ast.walk(ast.parse("x = y + 1")):
if isinstance(node, ast.Name):
print(node.id, type(node.ctx).__name__)
This distinction is essential for tools that track definitions and uses.
Walk a tree
ast.walk() yields a node and all descendants without offering contextual entry and exit hooks.
calls = [
node for node in ast.walk(tree)
if isinstance(node, ast.Call)
]
It is convenient for simple searches. Use visitors when traversal order and scope matter.
NodeVisitor
Subclass ast.NodeVisitor and define methods named visit_NodeType.
class FunctionCollector(ast.NodeVisitor):
def __init__(self):
self.names = []
def visit_FunctionDef(self, node):
self.names.append(node.name)
self.generic_visit(node)
collector = FunctionCollector()
collector.visit(ast.parse(source_code))
print(collector.names)
Call generic_visit() when child nodes should also be visited. Omitting it stops traversal below that node.
Async functions
AsyncFunctionDef differs from FunctionDef. A function collector should handle both.
def visit_AsyncFunctionDef(self, node):
self.names.append(node.name)
self.generic_visit(node)
The same caution applies to async comprehensions, AsyncFor, and AsyncWith.
Classes and scopes
ClassDef, functions, lambdas, and comprehensions create different scope rules. Merely collecting names does not resolve lexical binding.
For symbol analysis, combine the AST with symtable and maintain an explicit stack of scopes.
Source positions
Many nodes contain lineno, col_offset, end_lineno, and end_col_offset.
for node in ast.walk(tree):
if isinstance(node, ast.Call):
print(node.lineno, node.col_offset, node.end_lineno, node.end_col_offset)
Column offsets require care when mapping parser positions to user interfaces containing Unicode text.
Recover source segments
ast.get_source_segment(source, node) returns the corresponding text when position data exists.
segment = ast.get_source_segment(source, calls[0])
The segment preserves that text, but an AST does not retain every comment and formatting decision needed for perfect round trips.
Comments and tokens
Ordinary comments are not AST nodes. Tools that need to preserve comments, whitespace, quote style, and exact formatting should use tokens or a concrete syntax tree.
The AST is designed for structural meaning, not byte-for-byte source preservation.
Safe literal parsing with literal_eval
ast.literal_eval() accepts supported literal structures such as strings, bytes, numbers, tuples, lists, dictionaries, sets, booleans, and None.
settings = ast.literal_eval("{'retries': 3, 'enabled': True}")
It is much narrower than eval(), but should not receive enormous or deeply nested hostile input. Such values can consume memory, CPU, or stack depth.
Do not use eval for untrusted code
ast.parse() does not execute code, but compile(), exec(), and eval() will execute a valid tree. Rejecting a few node types does not automatically produce a secure sandbox.
Python offers many indirect ways to reach objects and perform operations. Use process isolation, resource limits, and a truly restricted language for hostile input.
Transform with NodeTransformer
NodeTransformer allows a visitor method to replace a node by returning another node.
class Rename(ast.NodeTransformer):
def visit_Name(self, node):
if node.id == "old_name":
return ast.copy_location(
ast.Name(id="new_name", ctx=node.ctx),
node,
)
return node
tree = Rename().visit(tree)
ast.fix_missing_locations(tree)
Preserve the original context and copy locations for useful errors and debugging.
Remove or expand statements
A transformer can return None to remove a node from a statement list, or return a list to replace one statement with several. Not every field accepts those forms.
Compile and test every transformed tree.
fix_missing_locations
Manually created nodes may not include line information. fix_missing_locations() fills missing values from parent nodes.
This makes compilation possible but does not automatically create editorially perfect positions.
copy_location and increment_lineno
copy_location(new, old) copies source positions. increment_lineno() shifts line numbers, which can help when adding a generated prefix.
Useful tracebacks depend on coherent filenames and locations.
Generate code with unparse
ast.unparse() produces Python source from a tree.
new_source = ast.unparse(tree)
The output may change quotes, parentheses, whitespace, and layout. It aims for syntactic equivalence, not textual preservation.
Compile an AST
compile() accepts a valid AST when required fields and locations are present.
code_object = compile(tree, "transformed.py", "exec")
namespace = {}
exec(code_object, namespace)
Execute only trusted code and control the namespace. A limited globals dictionary reduces accidental exposure but does not create a sandbox.
Grammar versions
The tree shape changes as Python syntax evolves. Tools supporting several interpreter versions must test each target and avoid assuming that every node type exists everywhere.
feature_version can request an approximation of an older grammar within the current parser’s capabilities, but it does not replace running tests on the target version.
Type comments and annotations
Parsing can retain selected type comments, while modern annotations appear in annotation nodes on arguments, assignments, and definitions.
For complete type semantics, use a type checker or its API rather than attempting to infer everything from the AST alone.
Decorators
Functions and classes have a decorator_list. Decorators are executed expressions and can replace or substantially modify the object being defined.
A static analyzer should report uncertainty instead of assuming that a decorated function preserves its original behavior.
Searching for dangerous calls
A linter may look for patterns involving eval, exec, shell-enabled subprocesses, or unsafe deserialization. Comparing only text names produces both false positives and false negatives.
Aliases, imports, shadowing, and reassignment require symbol resolution, and static analysis still cannot guarantee runtime safety.
Imports
Import and ImportFrom expose declared modules and aliases.
for node in ast.walk(tree):
if isinstance(node, ast.ImportFrom):
print(node.module, [alias.name for alias in node.names])
Dynamic and conditional imports require separate treatment.
Complexity metrics
Tools can count branches, loops, exception handlers, comprehensions, and boolean operations to estimate complexity. Metrics are review signals, not proof of quality.
Document the algorithm and keep results stable across supported versions.
Analyze many files
Discover files, read them with the correct encoding, parse each independently, and aggregate diagnostics. One syntax error should not erase results from the rest of the project.
Batch analysis can use bounded concurrency; see Python concurrent.futures. Avoid transferring huge trees between processes when compact diagnostics are sufficient.
Python source encoding
Use tokenize.open() when reading Python files so declared source encodings are respected. Assuming UTF-8 can fail on legacy projects.
Syntax errors
ast.parse() raises SyntaxError. Record filename, line, offset, and message without aborting the whole batch.
try:
tree = ast.parse(source, filename=path)
except SyntaxError as error:
report(path, error.lineno, error.offset, error.msg)
Recursion and huge input
Very deep trees can hit recursion limits in visitors and transformations. Huge files also consume substantial memory.
Set operational limits, handle failures, and do not raise the recursion limit without understanding native-stack risk.
Test transformations
Compare behavior before and after transformation, compile the tree, run tests, and inspect generated code. Include comprehensions, async syntax, pattern matching, decorators, f-strings, and annotations.
Use negative cases to verify that a rename does not affect the wrong scope.
Common mistakes
Common failures include forgetting generic_visit(), ignoring AsyncFunctionDef, losing Load/Store context, creating nodes without positions, expecting comments to survive, using literal_eval() without resource limits, executing untrusted trees, and assuming an AST resolves dynamic behavior.
Conclusion
ast turns Python code into a navigable and transformable structure. Use visitors for analysis, transformers for changes, locations for diagnostics, and unparse() when semantic equivalence is enough.
Apply resource limits, test every supported Python version, and treat results as static analysis rather than absolute truth. Consult the official ast documentation and the symtable documentation.
