Python projects and packages can contain hundreds of source files. Compiling them individually with py_compile works, but requires directory traversal, filters, error handling, and cache policy. The Python compileall module automates .pyc generation for directory trees with recursion, parallel workers, optimization levels, exclusions, and controlled traceback paths.
This guide covers the command-line interface and compile_dir(), compile_file(), and compile_path(). It complements our articles about py_compile, bytecode with dis, tokenize, tabnanny, and tracebacks.
When to use compileall
Compileall is useful during library installation, container-image creation, syntax validation, and deployments where users can read package files but cannot create __pycache__.
It does not turn a project into a standalone executable and does not replace tests. Compilation only verifies that selected source files have syntax accepted by the current interpreter.
Compile a directory from the command line
python -m compileall srcThe command recursively scans src, compiles .py files, and writes PEP 3147 caches, normally under __pycache__.
Compile several paths
python -m compileall src tests tools/script.pyPositional arguments may be files or directories. Directory traversal is recursive by default.
Running without arguments
Without arguments, the interface behaves as if it received -l plus directories from sys.path.
python -m compileallThis can inspect much more code than expected. Automation should pass explicit project roots.
Non-recursive mode
The -l option compiles only source files directly inside each named directory.
python -m compileall -l srcSubdirectories are skipped. This is useful for flat layouts or a deliberately limited check.
Control recursion depth
-r sets the maximum number of recursive levels.
python -m compileall -r 2 src-r 0 is equivalent to non-recursive compilation. When -r is provided, -l is ignored.
Force rebuilding
Up-to-date caches are normally skipped. -f forces compilation.
python -m compileall -f srcUse it for clean builds, policy changes, or diagnostics. Always forcing recompilation wastes time and writes when artifacts are already current.
Quiet output
-q suppresses the list of compiled files while keeping errors. -qq suppresses all output.
python -m compileall -q srcA pipeline must still check the exit status.
Parallel compilation with -j
-j N uses worker processes.
python -m compileall -j 4 src-j 0 selects a count based on os.process_cpu_count(). Parallelism helps large trees but may saturate CPU or storage in small containers.
Multiple optimization levels
The -o option can be repeated.
python -m compileall -o 0 -o 1 -o 2 srcEach level creates a cache variant. Level 1 removes assertions; level 2 also removes many docstrings. Use these variants only with a clear runtime policy.
Hardlink duplicate caches
If optimization variants have identical content, --hardlink-dupes can consolidate them through hard links.
python -m compileall \
-o 0 -o 1 -o 2 \
--hardlink-dupes \
srcThe filesystem must support hard links and files must be on the same volume. Packaging tools may preserve or break the relationship.
Invalidation mode
--invalidation-mode accepts timestamp, checked-hash, or unchecked-hash.
python -m compileall \
--invalidation-mode checked-hash \
srcTimestamp mode compares metadata. Checked hashes verify content during import. Unchecked hashes trust an external build system to keep caches current.
SOURCE_DATE_EPOCH
Without the variable, timestamp is the default. With SOURCE_DATE_EPOCH, checked hashes become the default, supporting reproducible builds.
Explicitly specifying the policy makes build intent easier to audit.
Exclude paths with -x
-x accepts a regular expression searched against each full path.
python -m compileall \
-x '[/\\](tests|migrations|vendor)[/\\]' \
srcA match skips the file. Test patterns on Windows and Unix because separators differ.
Read path lists with -i
-i adds files and directories read from a list file.
python -m compileall -i paths.txtUse -i - to read from standard input. This integrates with tools that already selected relevant source paths.
Traceback paths with -d
-d prepends a directory to each source name stored in bytecode.
python -m compileall \
-d /app \
srcThe build path may differ from the deployed path. A coherent value makes later tracebacks easier to understand.
Strip and prepend prefixes
-s removes a prefix and -p prepends another.
python -m compileall \
-s /workspace/project \
-p /app \
/workspace/project/src-s and -p can be combined, but neither combination may use -d. The feature helps reproducible builds and containers.
Limit symbolic links
-e DIR ignores symbolic links that point outside the permitted directory.
python -m compileall -e src srcThis reduces traversal outside the intended root. Still use a controlled root and minimal permissions.
Legacy output with -b
-b writes .pyc files beside source files using the legacy location.
python -m compileall -b srcThis can overwrite caches created by another interpreter version and loses the coexistence provided by __pycache__. Use it only for a specific compatibility requirement.
compile_dir()
The primary programmatic function traverses a tree and returns true only when every selected file compiled successfully.
import compileall
ok = compileall.compile_dir(
"src",
quiet=1,
)
if not ok:
raise SystemExit("Compilation failed")The official compileall documentation describes parameters corresponding to command-line options.
Regular-expression filters
rx accepts a compiled regular expression whose search() method receives every complete path.
import re
ok = compileall.compile_dir(
"src",
rx=re.compile(r"[/\\](tests|vendor)[/\\]"),
quiet=1,
)A skipped file is not a failure. Log important exclusions to avoid silently omitting production code.
Programmatic workers
ok = compileall.compile_dir(
"src",
workers=0,
quiet=1,
)workers=0 chooses a CPU-based count. Negative values raise ValueError. Unsupported platforms may fall back to sequential compilation.
Several optimization levels in one call
ok = compileall.compile_dir(
"src",
optimize=[0, 1, 2],
hardlink_dupes=True,
quiet=1,
)The sequence generates several variants for every source file. Hard links consolidate only identical contents.
compile_file()
compile_file() compiles one file with the same path, optimization, exclusion, and invalidation policy.
ok = compileall.compile_file(
"src/app.py",
force=True,
quiet=1,
)It returns true on successful compilation and when an rx filter intentionally skips the file.
compile_path()
compile_path() byte-compiles entries found on sys.path.
ok = compileall.compile_path(
skip_curdir=True,
quiet=1,
)Unlike compile_dir(), its default maximum depth is zero. Applications rarely need to compile their entire import path.
sys.pycache_prefix
Compilation respects sys.pycache_prefix. Generated caches are useful only when runtime uses the same prefix.
Containers can place caches in a separate writable directory. Keep build and runtime configuration synchronized.
WASI availability
The documentation marks compileall as unavailable on WASI. WebAssembly-targeted tools must detect the platform rather than assuming the module works.
Container build example
RUN python -m compileall \
-q \
-j 0 \
--invalidation-mode checked-hash \
/appRun it after copying dependencies and application source. Measure whether cache size is justified by startup improvements.
Compilation does not run tests
A file can compile while still containing missing imports, logic errors, platform incompatibilities, and type mistakes.
Use compileall as a quick build step followed by tests, linting, type checking, and real application startup.
Security and limits
Compilation does not execute module bodies, which is safer than importing them. Untrusted trees can still be huge, contain symlinks, and consume CPU or disk space.
Limit roots, recursion, file counts, worker counts, and storage. Do not import generated caches from untrusted source.
Common mistakes
- Running without arguments and compiling all of sys.path.
- Using
-bwithout a compatibility need. - Writing a regex that excludes required source.
- Using too many workers in a small container.
- Generating several optimization levels without considering space.
- Embedding incorrect deployment paths.
- Using unchecked hashes without a trusted build system.
- Confusing compilation with testing.
Best practices
- Pass explicit roots.
- Use PEP 3147 cache locations.
- Select invalidation according to the build model.
- Control workers and recursion depth.
- Test path filters and symbolic links.
- Embed traceback paths that match deployment.
- Check return values or process exit codes.
- Run tests after compilation.
Conclusion
The Python compileall module turns directory-wide byte compilation into a controlled installation and build step. It offers recursion, parallel workers, exclusions, several optimization levels, hash invalidation, and source-path rewriting.
An effective configuration depends on the environment: explicit roots, moderate parallelism, interpreter-tagged caches, and a coherent invalidation policy. Used this way, compileall prepares .pyc files without traversing unintended directories or confusing syntax validation with functional quality.
