itertools.batched strict: Validate Complete Batches

Processing large streams often means dividing values into smaller groups. Python provides itertools.batched() for that job, and recent versions also support the strict parameter. This guide explains how batching works, when strict validation is useful, and how to apply it to files, APIs, databases, generators, and data pipelines.

What itertools.batched does

itertools.batched(iterable, n) consumes an iterable and yields tuples with up to n items. Results are produced lazily, which makes the function suitable for files, generators, streams, and other sources that should not be loaded into memory all at once.

from itertools import batched
for batch in batched(range(10), 3):
    print(batch)

The final tuple may contain fewer items. This is useful when every source item must be processed and the total count does not need to be divisible by the batch size.

What strict=True changes

With strict=True, every batch must contain exactly n items. If the last batch is incomplete, Python raises ValueError. The option converts an implicit assumption into an explicit rule.

for batch in batched(range(9), 3, strict=True):
    print(batch)

Nine items form three complete groups. Ten items leave one extra value and therefore trigger an error. Strict mode is useful when each group represents a fixed structure such as coordinates, RGB values, matrix rows, paired records, protocol frames, or data imported in blocks.

When the default mode is better

Use the default behavior when a shorter final batch is valid. An email sender may send up to fifty recipients per request. An image processor may handle eight files at a time and finish with three. A database importer may insert five hundred rows per transaction without requiring every transaction to contain exactly five hundred rows.

In these situations the size is a maximum, not a schema. Rejecting the remainder would add complexity without protecting a real business rule.

When strict mode is appropriate

Use strict mode when missing values indicate malformed data. Suppose a flat stream stores latitude, longitude, and altitude for each point. Every record needs three values. A shorter final tuple means that the source ended in the middle of a record.

values = [10.2, -48.1, 800, 11.0, -47.9, 820]
points = list(batched(values, 3, strict=True))

The same principle applies to key-value pairs, sensor packets, fixed-width exports, training samples with rigid shapes, and command arguments expected in groups.

Validation with generators

Collections with len() can sometimes be checked before processing. Generators usually have no known size. Strict batching validates the structure while the source is consumed.

def process_records(source):
    try:
        for batch in batched(source, 4, strict=True):
            save(batch)
    except ValueError as error:
        log_error(str(error))
        raise

Earlier groups may already have been saved when the incomplete final batch is discovered. If the operation must be atomic, use a transaction, temporary file, staging table, or compensating action.

Processing large files

A text file can be read line by line and grouped without loading the complete file into memory.

with open('data.txt', encoding='utf-8') as file:
    for lines in batched(file, 1000):
        normalized = [line.strip() for line in lines]
        send(normalized)

The last group may be smaller, which is normally correct. If one logical record always occupies four lines, use strict=True. An error then reveals a truncated or corrupted export.

Calling APIs in batches

Many APIs limit the number of identifiers accepted in one request.

for ids in batched(all_ids, 50):
    response = client.fetch(ids=list(ids))
    store(response)

The tuple is converted to a list only if the client requires that JSON shape. Avoid strict mode unless the remote contract truly demands exactly fifty IDs. Most APIs define a maximum and accept a smaller final request.

Database inserts

Batch inserts reduce network round trips and statement overhead.

for records in batched(generate_records(), 500):
    cursor.executemany(sql, records)
    connection.commit()

Measure the batch size in the real environment. Very large groups can increase locks, memory use, transaction time, and retry cost. Related reading includes Python and SQLite and Python with MySQL.

Why not just slice a list

List slicing is a common solution.

batches = [data[i:i + 100] for i in range(0, len(data), 100)]

It works for an indexable sequence already in memory. batched() accepts any iterable and preserves lazy evaluation. That makes it more suitable for streams and generated values. See the introduction to itertools for more iterator patterns.

Generator pitfalls

A generator is consumed once. Do not count it and then expect to iterate over the same values again. Strict validation occurs only when the last batch is requested. If consumer code stops early, an incomplete ending may never be observed.

For validation jobs, consume the entire iterator. For infinite iterators, strict mode never reaches a final partial batch, so it cannot validate total divisibility.

Handling ValueError correctly

Do not catch the exception merely to ignore it. Strict mode exists to reveal invalid structure. Add domain context and preserve the original exception.

try:
    groups = list(batched(values, 3, strict=True))
except ValueError as exc:
    raise ValueError('Input must contain complete groups of three') from exc

A domain-specific message helps operators identify the real problem. Review exception handling in Python for broader practices.

Version compatibility

Confirm the Python version in development, production, containers, and CI before relying on the parameter. Different environments may expose different signatures. Consult the official itertools documentation and the Python release notes for exact availability.

Recommended tests

Test an empty iterable, an exact multiple, fewer items than one batch, several complete batches, and an incomplete final batch. Test a generator rather than only lists.

def test_exact_batches():
    assert list(batched(range(6), 3, strict=True)) == [(0, 1, 2), (3, 4, 5)]

Use the guide to unit testing in Python to structure edge cases and expected exceptions.

Performance considerations

batched() reduces boilerplate, but the ideal group size depends on the operation. Network calls, database writes, CPU work, and disk access have different costs. Benchmark representative data. Track throughput, latency, memory, failures, and retry size. A larger batch is not automatically faster.

Best practices

Name variables according to their domain, document why the size was chosen, and use a constant when the size is part of a protocol. Apply strict=True only when a partial batch is invalid. Keep lazy processing instead of converting every batch to a large list. Add logging with item counts and batch numbers. Protect side effects with transactions when late validation could leave partial results.

Conclusion

itertools.batched() is a clear and memory-efficient way to group iterable values. Default mode treats the final shorter group as valid, while strict=True enforces exact group sizes and raises an error for incomplete data. The correct choice comes from the domain contract: is a smaller final batch acceptable, or does it indicate corruption? Once that rule is explicit, batching code becomes simpler, safer, and easier to test.