Command-line programs often need to process several files as one continuous stream. They may also need to read standard input when no paths are provided. The Python fileinput module handles both cases with one iterator: it reads filenames from sys.argv, falls back to sys.stdin, and exposes information such as the current filename, cumulative line number, and line number within each file.
This guide covers fileinput.input(), the FileInput class, compressed-file hooks, explicit encodings, standard input, in-place rewriting, backups, and safe error handling. It complements our guides to pathlib, tempfile, shlex, filecmp, and linecache.
When fileinput is useful
The module is designed for scripts that apply the same line-based operation to zero, one, or many input files.
import fileinput
for line in fileinput.input(encoding="utf-8"):
process(line)By default, filenames come from sys.argv[1:]. If the list is empty, input comes from sys.stdin. The same filter can therefore work with filenames, pipes, and redirected input.
Terminal examples
python analyze.py app.log api.log
cat app.log | python analyze.pyThe first command iterates over two files. The second reads the pipe. A filename equal to - is also interpreted as standard input.
Pass files explicitly
Library code should not rely on process arguments when it already has a list of paths.
files = ["january.csv", "february.csv"]
with fileinput.input(files=files, encoding="utf-8") as source:
for line in source:
process(line)The context manager closes the active file and the sequence even if processing raises an exception.
A single filename is accepted
with fileinput.input(files="data.txt", encoding="utf-8") as source:
for line in source:
print(line, end="")Use a tuple or list when the collection is built dynamically.
Newlines are preserved
Returned lines keep their newline characters when present. The final line of a file may have no newline.
for line in fileinput.input(files="data.txt", encoding="utf-8"):
print(repr(line))Avoid calling strip() blindly because it removes meaningful spaces and tabs. If only line terminators should be removed, use a deliberate policy such as rstrip("\r\n").
Current filename
fileinput.filename() returns the source that produced the most recently read line.
for line in fileinput.input(files=files, encoding="utf-8"):
print(fileinput.filename(), line, end="")Before the first line, the result is None. Module-level helper functions depend on an active global instance created by fileinput.input().
Cumulative line number
lineno() counts all lines read across the complete sequence.
for line in fileinput.input(files=files, encoding="utf-8"):
print(fileinput.lineno(), line, end="")If the first file contains 100 lines, the first line of the second file has cumulative number 101.
Per-file line number
filelineno() restarts at 1 for each file.
for line in fileinput.input(files=files, encoding="utf-8"):
print(
fileinput.filename(),
fileinput.filelineno(),
line,
end="",
)This is the number normally used in validation errors and diagnostics.
Detect the first line
isfirstline() identifies the first line of the current file.
for line in fileinput.input(files=files, encoding="utf-8"):
if fileinput.isfirstline():
print(f"--- {fileinput.filename()} ---")
print(line, end="")The helper is convenient for headers, separators, and per-file initialization.
Detect standard input
isstdin() returns whether the most recently read line came from stdin.
origin = "stdin" if fileinput.isstdin() else fileinput.filename()If stdin appears more than once, later occurrences normally produce no lines because the stream has already been consumed.
Skip the rest of a file
nextfile() closes the current file and advances to the next one. Skipped lines do not increase the cumulative count.
for line in fileinput.input(files=files, encoding="utf-8"):
if line.startswith("END"):
fileinput.nextfile()
continue
process(line)It cannot skip the first file before any line has been read, and the reported filename changes only after reading from the next source.
Use FileInput directly
The class exposes the same operations without relying on module-level global state.
from fileinput import FileInput
with FileInput(files=files, encoding="utf-8") as source:
for line in source:
print(
source.filename(),
source.filelineno(),
line,
end="",
)This style is preferable in reusable libraries, tests, servers, and programs that need more than one independent sequence.
Avoid overlapping global input
fileinput.input() installs one global active state. Starting another global sequence before closing the first can raise errors or create confusing behavior. Explicit FileInput objects make ownership visible and easier to test.
Encoding and error policy
Modern Python versions accept encoding and errors directly.
with fileinput.input(
files=files,
encoding="utf-8",
errors="strict",
) as source:
for line in source:
process(line)Use strict when invalid text must stop the job. Policies such as replace and surrogateescape should be documented because they can change or preserve problematic bytes in different ways.
Binary mode
FileInput accepts mode="r" or mode="rb".
with fileinput.FileInput(files=files, mode="rb") as source:
for line in source:
process_bytes(line)Binary lines are bytes. Do not pass text encoding parameters in this mode.
I/O errors
Fileinput raises OSError when opening or reading fails.
try:
with fileinput.input(files=files, encoding="utf-8") as source:
for line in source:
process(line)
except OSError as error:
print(f"read failed: {error}")A batch process must decide whether one bad file should abort everything or be recorded for later handling. Fileinput does not automatically skip failures.
Empty files
An empty file is opened and closed without yielding any lines. If empty files are invalid, validate file size or keep a separate record of paths encountered.
Read gzip and bzip2 files
hook_compressed() opens .gz and .bz2 files transparently.
with fileinput.FileInput(
files=["app.log", "app.log.gz", "old.log.bz2"],
openhook=fileinput.hook_compressed,
encoding="utf-8",
) as source:
for line in source:
process(line)Other extensions are opened normally. Detection is based on the filename suffix, not a verified content signature.
Custom open hooks
An openhook receives a filename and mode and returns an opened file-like object. When encoding and errors are supplied, they are passed as keyword arguments.
from pathlib import Path
def open_validated(filename, mode, *, encoding=None, errors=None):
path = Path(filename).resolve()
if path.suffix not in {".txt", ".log"}:
raise ValueError("unsupported extension")
return open(path, mode, encoding=encoding, errors=errors)
with fileinput.FileInput(
files=files,
openhook=open_validated,
encoding="utf-8",
) as source:
for line in source:
process(line)An open hook cannot be combined with in-place editing.
In-place filtering
With inplace=True, the original file is moved to a backup and sys.stdout is redirected to the original path. Everything printed becomes the new file contents.
for line in fileinput.input(
files=["config.txt"],
inplace=True,
backup=".bak",
encoding="utf-8",
):
print(line.replace("old", "new"), end="")After completion, config.txt contains transformed text and config.txt.bak contains the original.
Risks of in-place editing
The operation is destructive. A crash, wrong path, truncated output, or accidental print can corrupt data. Test the transformation without inplace, operate on copies, preserve backups, and verify available disk space.
The official fileinput documentation notes that an existing backup with the same name may be replaced silently. Critical workflows should create unique backup names themselves.
Default backup behavior
When no persistent backup suffix is supplied, fileinput uses a temporary backup and removes it after closing the output. Important data should use an explicit suffix or a separate archival policy.
Logging during rewriting
Normal stdout is redirected to the file during in-place mode. Progress messages must go to stderr.
import sys
print("processing", fileinput.filename(), file=sys.stderr)Keep transformed output and operational logs strictly separated.
In-place mode and stdin
In-place filtering is disabled for standard input because there is no original file to replace. A pipe-based filter should write its result to stdout and let the caller choose a destination.
A safer transactional rewrite
For critical files, write to a temporary file, flush and validate it, then replace the original with os.replace(). This provides clearer control over permissions, failures, backups, and recovery.
Sequential access only
FileInput is designed for strict sequential iteration. Do not expect indexing or random access, and avoid mixing iteration with manual readline() calls in confusing ways. Use linecache or a dedicated data structure when access by arbitrary line number is required.
Path security
Validate user-provided filenames against an allowed root and resolve symlinks.
BASE = Path("/srv/imports").resolve()
def allowed_path(name):
path = (BASE / name).resolve()
if path != BASE and BASE not in path.parents:
raise ValueError("path outside allowed directory")
return pathAlso limit the number, size, and type of files accepted by a service.
Hostile compressed inputs
A small compressed file may expand to gigabytes. Apply limits on bytes, time, and lines. hook_compressed() simplifies opening but does not protect against decompression bombs.
Testing multiple files
def test_multiple_files(tmp_path):
first = tmp_path / "a.txt"
second = tmp_path / "b.txt"
first.write_text("one\ntwo\n", encoding="utf-8")
second.write_text("three\n", encoding="utf-8")
with fileinput.FileInput(
files=[first, second],
encoding="utf-8",
) as source:
lines = list(source)
assert lines == ["one\n", "two\n", "three\n"]Also test empty files, a final line without newline, invalid encodings, stdin, compressed files, and failure during rewriting.
fileinput versus open()
Use open() for one file and explicit lifecycle control. Use fileinput when the core abstraction is one continuous sequence of lines from multiple sources, especially in command-line filters.
Performance
Fileinput processes one line at a time and does not load every file into memory. The dominant costs are decoding, decompression, storage, and the transformation itself. Avoid expensive regular-expression compilation inside the loop and measure real workloads before optimizing.
Common mistakes
- Depending on
sys.argvunintentionally. - Omitting an explicit encoding.
- Opening overlapping global sequences.
- Calling
strip()and removing significant whitespace. - Printing logs to stdout during in-place mode.
- Rewriting important files without persistent backups.
- Assuming stdin can be consumed repeatedly.
- Accepting compressed input without resource limits.
Best practices
- Use a context manager.
- Prefer explicit
FileInputobjects in libraries. - Declare encoding and error policy.
- Send diagnostics to stderr.
- Preserve backups for destructive changes.
- Validate external paths and sizes.
- Test empty files and missing final newlines.
- Use atomic replacement for critical data.
Conclusion
The Python fileinput module simplifies filters that read many files or standard input while preserving useful origin and line-number information. It also provides compression hooks and convenient in-place rewriting.
That convenience requires discipline. Avoid global state in complex applications, choose text-decoding rules explicitly, handle OSError, constrain unknown input, and treat every rewrite as destructive. With validation, backups, and tests, fileinput is a compact foundation for reliable command-line utilities.
