Python compileall: Compile Directories

Python projects and packages can contain hundreds of source files. Compiling them individually with py_compile works, but requires directory traversal, filters, error handling, and cache policy. The Python compileall module automates .pyc generation for directory trees with recursion, parallel workers, optimization levels, exclusions, and controlled traceback paths.

This guide covers the command-line interface and compile_dir(), compile_file(), and compile_path(). It complements our articles about py_compile, bytecode with dis, tokenize, tabnanny, and tracebacks.

When to use compileall

Compileall is useful during library installation, container-image creation, syntax validation, and deployments where users can read package files but cannot create __pycache__.

It does not turn a project into a standalone executable and does not replace tests. Compilation only verifies that selected source files have syntax accepted by the current interpreter.

Compile a directory from the command line

python -m compileall src

The command recursively scans src, compiles .py files, and writes PEP 3147 caches, normally under __pycache__.

Compile several paths

python -m compileall src tests tools/script.py

Positional arguments may be files or directories. Directory traversal is recursive by default.

Running without arguments

Without arguments, the interface behaves as if it received -l plus directories from sys.path.

python -m compileall

This can inspect much more code than expected. Automation should pass explicit project roots.

Non-recursive mode

The -l option compiles only source files directly inside each named directory.

python -m compileall -l src

Subdirectories are skipped. This is useful for flat layouts or a deliberately limited check.

Control recursion depth

-r sets the maximum number of recursive levels.

python -m compileall -r 2 src

-r 0 is equivalent to non-recursive compilation. When -r is provided, -l is ignored.

Force rebuilding

Up-to-date caches are normally skipped. -f forces compilation.

python -m compileall -f src

Use it for clean builds, policy changes, or diagnostics. Always forcing recompilation wastes time and writes when artifacts are already current.

Quiet output

-q suppresses the list of compiled files while keeping errors. -qq suppresses all output.

python -m compileall -q src

A pipeline must still check the exit status.

Parallel compilation with -j

-j N uses worker processes.

python -m compileall -j 4 src

-j 0 selects a count based on os.process_cpu_count(). Parallelism helps large trees but may saturate CPU or storage in small containers.

Multiple optimization levels

The -o option can be repeated.

python -m compileall -o 0 -o 1 -o 2 src

Each level creates a cache variant. Level 1 removes assertions; level 2 also removes many docstrings. Use these variants only with a clear runtime policy.

Hardlink duplicate caches

If optimization variants have identical content, --hardlink-dupes can consolidate them through hard links.

python -m compileall \
  -o 0 -o 1 -o 2 \
  --hardlink-dupes \
  src

The filesystem must support hard links and files must be on the same volume. Packaging tools may preserve or break the relationship.

Invalidation mode

--invalidation-mode accepts timestamp, checked-hash, or unchecked-hash.

python -m compileall \
  --invalidation-mode checked-hash \
  src

Timestamp mode compares metadata. Checked hashes verify content during import. Unchecked hashes trust an external build system to keep caches current.

SOURCE_DATE_EPOCH

Without the variable, timestamp is the default. With SOURCE_DATE_EPOCH, checked hashes become the default, supporting reproducible builds.

Explicitly specifying the policy makes build intent easier to audit.

Exclude paths with -x

-x accepts a regular expression searched against each full path.

python -m compileall \
  -x '[/\\](tests|migrations|vendor)[/\\]' \
  src

A match skips the file. Test patterns on Windows and Unix because separators differ.

Read path lists with -i

-i adds files and directories read from a list file.

python -m compileall -i paths.txt

Use -i - to read from standard input. This integrates with tools that already selected relevant source paths.

Traceback paths with -d

-d prepends a directory to each source name stored in bytecode.

python -m compileall \
  -d /app \
  src

The build path may differ from the deployed path. A coherent value makes later tracebacks easier to understand.

Strip and prepend prefixes

-s removes a prefix and -p prepends another.

python -m compileall \
  -s /workspace/project \
  -p /app \
  /workspace/project/src

-s and -p can be combined, but neither combination may use -d. The feature helps reproducible builds and containers.

Limit symbolic links

-e DIR ignores symbolic links that point outside the permitted directory.

python -m compileall -e src src

This reduces traversal outside the intended root. Still use a controlled root and minimal permissions.

Legacy output with -b

-b writes .pyc files beside source files using the legacy location.

python -m compileall -b src

This can overwrite caches created by another interpreter version and loses the coexistence provided by __pycache__. Use it only for a specific compatibility requirement.

compile_dir()

The primary programmatic function traverses a tree and returns true only when every selected file compiled successfully.

import compileall

ok = compileall.compile_dir(
    "src",
    quiet=1,
)

if not ok:
    raise SystemExit("Compilation failed")

The official compileall documentation describes parameters corresponding to command-line options.

Regular-expression filters

rx accepts a compiled regular expression whose search() method receives every complete path.

import re

ok = compileall.compile_dir(
    "src",
    rx=re.compile(r"[/\\](tests|vendor)[/\\]"),
    quiet=1,
)

A skipped file is not a failure. Log important exclusions to avoid silently omitting production code.

Programmatic workers

ok = compileall.compile_dir(
    "src",
    workers=0,
    quiet=1,
)

workers=0 chooses a CPU-based count. Negative values raise ValueError. Unsupported platforms may fall back to sequential compilation.

Several optimization levels in one call

ok = compileall.compile_dir(
    "src",
    optimize=[0, 1, 2],
    hardlink_dupes=True,
    quiet=1,
)

The sequence generates several variants for every source file. Hard links consolidate only identical contents.

compile_file()

compile_file() compiles one file with the same path, optimization, exclusion, and invalidation policy.

ok = compileall.compile_file(
    "src/app.py",
    force=True,
    quiet=1,
)

It returns true on successful compilation and when an rx filter intentionally skips the file.

compile_path()

compile_path() byte-compiles entries found on sys.path.

ok = compileall.compile_path(
    skip_curdir=True,
    quiet=1,
)

Unlike compile_dir(), its default maximum depth is zero. Applications rarely need to compile their entire import path.

sys.pycache_prefix

Compilation respects sys.pycache_prefix. Generated caches are useful only when runtime uses the same prefix.

Containers can place caches in a separate writable directory. Keep build and runtime configuration synchronized.

WASI availability

The documentation marks compileall as unavailable on WASI. WebAssembly-targeted tools must detect the platform rather than assuming the module works.

Container build example

RUN python -m compileall \
    -q \
    -j 0 \
    --invalidation-mode checked-hash \
    /app

Run it after copying dependencies and application source. Measure whether cache size is justified by startup improvements.

Compilation does not run tests

A file can compile while still containing missing imports, logic errors, platform incompatibilities, and type mistakes.

Use compileall as a quick build step followed by tests, linting, type checking, and real application startup.

Security and limits

Compilation does not execute module bodies, which is safer than importing them. Untrusted trees can still be huge, contain symlinks, and consume CPU or disk space.

Limit roots, recursion, file counts, worker counts, and storage. Do not import generated caches from untrusted source.

Common mistakes

• Running without arguments and compiling all of sys.path.
• Using -b without a compatibility need.
• Writing a regex that excludes required source.
• Using too many workers in a small container.
• Generating several optimization levels without considering space.
• Embedding incorrect deployment paths.
• Using unchecked hashes without a trusted build system.
• Confusing compilation with testing.

Best practices

• Pass explicit roots.
• Use PEP 3147 cache locations.
• Select invalidation according to the build model.
• Control workers and recursion depth.
• Test path filters and symbolic links.
• Embed traceback paths that match deployment.
• Check return values or process exit codes.
• Run tests after compilation.

Conclusion

The Python compileall module turns directory-wide byte compilation into a controlled installation and build step. It offers recursion, parallel workers, exclusions, several optimization levels, hash invalidation, and source-path rewriting.

An effective configuration depends on the environment: explicit roots, moderate parallelism, interpreter-tagged caches, and a coherent invalidation policy. Used this way, compileall prepares .pyc files without traversing unintended directories or confusing syntax validation with functional quality.