Python token: Parser Constants

Tools that inspect source code must distinguish identifiers, numbers, strings, comments, operators, indentation, and end-of-input markers. The Python token module provides the numeric constants and lookup tables used to represent those lexical elements in tokenizers, parsers, debuggers, linters, and source transformers.

This guide covers tok_name, EXACT_TOKEN_TYPES, ISTERMINAL(), ISNONTERMINAL(), and ISEOF(), along with NAME, NUMBER, STRING, OP, indentation, f-strings, and template strings. It complements our articles about Python tokenize, keyword, symtable, bytecode with dis, and codeop.

What the token module provides

The token.py module gives symbolic names to token categories. Their numeric values are implementation details that may change between Python versions, so applications should compare with token.NAME rather than a copied integer.

import token

print(token.NAME)
print(token.NUMBER)
print(token.STRING)
print(token.tok_name[token.NAME])

The official token documentation explains that these values represent terminal nodes in the language grammar and mirror definitions used by Python’s parser.

token versus tokenize

The token module defines constants and mappings. The tokenize module reads bytes or text and emits records containing a type, original text, start position, end position, and source line.

import io
import token
import tokenize

source = b"total = price + tax\n"

for item in tokenize.tokenize(io.BytesIO(source).readline):
    name = token.tok_name.get(item.type, str(item.type))
    print(name, item.string)

Token is the vocabulary; tokenize is a scanner that produces values from that vocabulary.

Use tok_name for readable output

token.tok_name maps numeric codes back to symbolic names.

for code, name in sorted(token.tok_name.items()):
    print(code, name)

This mapping makes diagnostics and visualizers much easier to understand. A report containing INDENT or DOUBLESTAR is more useful than a bare integer.

The NAME token

NAME identifies identifiers and words that may need further interpretation by the parser.

source = b"for item in items:\n    print(item)\n"

Depending on the API and tokenizer options, text such as for, item, in, and print may be surfaced as names. Combine the token text with the keyword module when you need to classify reserved or contextual words.

import keyword

if item.type == token.NAME:
    if keyword.iskeyword(item.string):
        category = "keyword"
    elif keyword.issoftkeyword(item.string):
        category = "soft keyword"
    else:
        category = "identifier"

NUMBER preserves the lexeme

NUMBER represents integer, floating-point, complex, and alternate-base numeric literals.

values = b"10 3.14 0xff 1_000 2j\n"

The token retains the original source text. It does not convert hexadecimal notation or remove underscores. Let the parser interpret the value, or use a purpose-specific safe conversion.

STRING tokens

STRING represents ordinary string and byte literals. The text includes prefixes, quotes, and escape sequences without evaluating them.

source = b"path = r'C:\\temp'\n"

That preservation is valuable for formatters that need to keep quote style. When a trusted literal must be converted, ast.literal_eval() is safer than eval().

COMMENT tokens

COMMENT is produced by the public tokenize API for comments.

# configuration comment
limit = 10  # maximum value

The parser normally ignores comments, while formatters, documentation tools, and refactoring systems need to preserve them. This is a major difference between token streams and an abstract syntax tree.

NEWLINE and NL

NEWLINE terminates a logical statement. NL represents a physical line break that does not terminate the logical statement, such as one inside parentheses.

result = (
    first
    + second
)

Internal line breaks are NL; the completed expression ends with NEWLINE. Interactive compilers and formatters use this distinction to determine completeness without losing layout.

INDENT and DEDENT

Python represents block structure through indentation. INDENT starts a deeper block, while DEDENT returns to an outer level.

if active:
    execute()
    record()
finish()

The indent token contains the original whitespace prefix. A style checker may inspect it, but ambiguity and invalid indentation should be left to the tokenizer and specialized tools such as tabnanny.

ENCODING

tokenize.tokenize() reads bytes and always begins with an ENCODING token.

with open("module.py", "rb") as file:
    tokens = list(tokenize.tokenize(file.readline))

assert tokens[0].type == token.ENCODING

Python detects the encoding from a byte-order mark or a declaration in the first two lines. The older string-based generate_tokens() API does not produce this marker.

ENDMARKER

ENDMARKER indicates that the input stream is finished.

last = tokens[-1]
assert last.type == token.ENDMARKER

Consumers should process the marker so they can close pending state and verify that the full input was read.

OP and exact operator types

The tokenize module usually reports operators and delimiters using the generic OP category.

source = b"result += value ** 2\n"

Use TokenInfo.exact_type to distinguish +=, **, parentheses, commas, and other symbols.

for item in tokenize.tokenize(io.BytesIO(source).readline):
    if item.type == token.OP:
        print(item.string, token.tok_name[item.exact_type])

The generic token can therefore be refined to PLUSEQUAL, DOUBLESTAR, LPAR, or another exact constant.

EXACT_TOKEN_TYPES

token.EXACT_TOKEN_TYPES maps operator text to exact numeric codes.

assert token.EXACT_TOKEN_TYPES["+"] == token.PLUS
assert token.EXACT_TOKEN_TYPES[":="] == token.COLONEQUAL
assert token.EXACT_TOKEN_TYPES["->"] == token.RARROW

This table is useful when a tool already has a symbol string and needs the corresponding parser constant.

Operators and delimiters evolve

The module defines constants for parentheses, brackets, braces, commas, colons, dots, arithmetic operators, comparisons, compound assignments, annotation arrows, the assignment expression, and other symbols.

Do not maintain a handwritten copy. Python versions can add or remove values, as happened with COLONEQUAL, EXCLAMATION, and historical async tokens.

F-string tokens

Current Python versions expose FSTRING_START, FSTRING_MIDDLE, and FSTRING_END.

message = f"Hello, {user.name}!"

The start includes the prefix and opening quote. Literal content appears in middle segments, while replacement expressions use ordinary Python tokens between braces and formatting delimiters.

Source tools should test the exact Python versions they support because f-string grammar and tokenization have evolved.

Template strings in Python 3.14

Python 3.14 adds TSTRING_START, TSTRING_MIDDLE, and TSTRING_END for template string literals.

A package supporting older interpreters must not assume these names exist.

if hasattr(token, "TSTRING_START"):
    supports_template_strings = True

Feature detection or an explicit version matrix keeps a linter from failing during import.

SOFT_KEYWORD is not normally emitted

The module defines SOFT_KEYWORD for internal use, but the public tokenizer normally reports a contextual keyword as NAME.

if item.type == token.NAME and keyword.issoftkeyword(item.string):
    print("potential soft keyword")

Only the syntactic context determines whether the word acts as a keyword. Parse an AST when structural certainty is required.

ERRORTOKEN and tokenizer exceptions

ERRORTOKEN can represent invalid input in some situations. However, tokenize may instead raise TokenError, or emit tokens that the parser rejects later.

try:
    tokens = list(tokenize.generate_tokens(reader))
except tokenize.TokenError as error:
    print("incomplete or invalid input", error)

Do not use the absence of ERRORTOKEN as a complete syntax check. Use compile() or ast.parse() after token-level inspection.

TYPE_COMMENT and TYPE_IGNORE

TYPE_COMMENT and TYPE_IGNORE support compiler modes that recognize legacy type comments.

result = load()  # type: Result
ignore()         # type: ignore

The public tokenizer does not emit these types in every configuration. Type-analysis tools usually request the appropriate AST options.

ISTERMINAL()

token.ISTERMINAL(value) reports whether a numeric code represents a terminal token.

assert token.ISTERMINAL(token.NAME)

This helper is mainly useful in tools that interact with parser grammar tables or concrete parse trees.

ISNONTERMINAL()

ISNONTERMINAL() checks grammar symbols representing composite rules.

def classify(code):
    if token.ISTERMINAL(code):
        return "terminal"
    if token.ISNONTERMINAL(code):
        return "nonterminal"
    return "unknown"

Most application tools work with AST nodes, but parser infrastructure may still need this distinction.

ISEOF()

ISEOF() identifies the end-of-input marker without comparing a hard-coded number.

assert token.ISEOF(token.ENDMARKER)

The function keeps the intent explicit and avoids implementation coupling.

N_TOKENS

N_TOKENS reports the number of token types defined by the current interpreter.

print(token.N_TOKENS)

Do not use it as an unversioned persistence format. The count and individual codes can change.

Persist names, not only numbers

When token analysis results must be stored, record the symbolic name and interpreter version.

import platform

record = {
    "python": platform.python_version(),
    "type": token.tok_name[item.type],
    "text": item.string,
}

A numeric value written on one Python release may mean something different on another.

Formatters and refactoring tools

A transformer that must preserve comments and spacing can edit tokens and rebuild source with tokenize.untokenize().

updated = []
for item in tokens:
    if item.type == token.NAME and item.string == "old_name":
        item = item._replace(string="new_name")
    updated.append(item)

result = tokenize.untokenize(updated)

Correct renaming requires scope analysis. Combine tokens with symtable or AST information so unrelated attributes and local names are not changed.

Linting with exact types

A linter can detect prohibited operators or language features precisely.

if item.exact_type == token.COLONEQUAL:
    report_walrus_usage(item.start)

Always retain source positions and original lines for useful diagnostics.

Cross-version tests

Run a CI matrix covering every supported Python release. Verify added and removed token constants, f-string behavior, contextual keywords, and exact operator mappings.

def test_exact_power_operator():
    assert token.EXACT_TOKEN_TYPES["**"] == token.DOUBLESTAR

Feature detection is safer than importing a constant that does not exist on an older interpreter.

Security considerations

Tokenization does not execute source code, but it is not a security boundary. Extremely large or adversarial input may consume time and memory, and the resulting source remains dangerous if executed.

Apply size and time limits. Never send tokenized external input directly to eval() or exec().

Common mistakes

• Comparing token types with numeric literals.
• Confusing token with tokenize.
• Treating every OP as the same operator.
• Ignoring exact_type.
• Confusing NL and NEWLINE.
• Expecting soft keywords to arrive as SOFT_KEYWORD.
• Persisting codes without a Python version.
• Executing source merely because tokenization succeeded.

Best practices

• Compare against symbolic constants.
• Use tok_name in logs.
• Inspect exact_type for operators.
• Preserve positions and original lines.
• Detect new features safely.
• Test all supported Python versions.
• Combine tokens with AST and symbol tables.
• Limit untrusted input.

Conclusion

The Python token module defines the vocabulary used to classify lexical elements in source code. It provides readable constants for names, numbers, strings, operators, indentation, f-strings, template strings, and internal markers.

Combined with tokenize, keyword, AST, and symbol-table analysis, it supports accurate source tools without depending on unstable numeric values. Use symbolic names, account for the target Python version, and never confuse successful parsing with safe execution.