Processing data in chunks is common in APIs, databases, queues, files, and pipelines. Instead of loading an entire collection or writing index-based loops, itertools.batched() groups any iterable into tuples of a chosen size while consuming input lazily. The final batch may be shorter, or it can be rejected with strict=True on Python versions that support that option.
This guide explains how to batch sequences and generators, handle incomplete groups, send requests in blocks, read files, combine batches with executors, control memory and backpressure, design retries, and avoid common mistakes in incremental processing.
Your first batch
from itertools import batched
numbers = range(10)
for batch in batched(numbers, 3):
print(batch)
The result contains tuples with up to three values: (0, 1, 2), (3, 4, 5), (6, 7, 8), and (9,). The iterable is not converted into one large list.
Lazy consumption
def generate():
for number in range(1_000_000):
yield number
first = next(batched(generate(), 100))
Only the first one hundred values are consumed to build the first tuple. This makes the function suitable for large or potentially infinite streams, provided downstream consumers also advance in a controlled way.
The incomplete final batch
By default, the final group may contain fewer than n values. This behavior is appropriate when every input must be processed even when the total length is not divisible by the batch size.
list(batched([1, 2, 3, 4, 5], 2))
# [(1, 2), (3, 4), (5,)]
Strict mode
for batch in batched(data, 4, strict=True):
process(batch)
With strict=True, an incomplete final group raises ValueError. This is useful for coordinate pairs, binary frames, matrices, or protocols that require exact group sizes. Check your minimum Python version because strict support was added after the original function.
Invalid sizes
The size must be at least one. Zero and negative values are invalid. Validate configuration obtained from environment variables, command-line options, or remote settings before creating the iterator.
Why slicing is not equivalent
batches = [data[i:i + 100] for i in range(0, len(data), 100)]
Slicing works only for indexable sequences and creates all groups immediately. It cannot consume files, generators, database cursors, or arbitrary iterators. batched() accepts any iterable and yields one tuple at a time.
Conceptual implementation
from itertools import islice
def manual_batched(iterable, n):
iterator = iter(iterable)
while batch := tuple(islice(iterator, n)):
yield batch
The official function follows this idea: create an iterator, take up to n values with islice(), and stop when no values remain. Prefer the standard API because it communicates intent and handles edge cases consistently.
Batching API requests
for batch in batched(ids, 100):
response = client.fetch_many(list(batch))
save(response)
External APIs often limit IDs per request. The chosen size must also consider payload bytes, rate limits, timeouts, response size, and retry cost. A limit of one hundred IDs does not guarantee every request has the same network weight.
Database inserts
for batch in batched(records, 500):
cursor.executemany(sql, batch)
connection.commit()
Batching reduces per-call overhead, but very large transactions can hold locks longer and make rollbacks expensive. Benchmark against the real database and choose transaction boundaries that match required atomicity.
Reading files in groups of lines
with open("events.log", encoding="utf-8") as file:
for lines in batched(file, 1000):
process_lines(lines)
A text file is already an iterator over lines. At each step, only the current tuple needs to remain in memory. Line strings normally include their newline character.
Binary data
Iterating directly over a bytes object produces integers. When you need binary blocks, file.read(size) is often more efficient. batched is best when the source already yields logical units such as records, tokens, or decoded messages.
Transforming each batch
for batch in batched(data, 50):
normalized = [normalize(item) for item in batch]
write(normalized)
Each batch is a tuple. Convert it to a list only when an API requires mutability or a JSON array.
Combining with executors
from concurrent.futures import ThreadPoolExecutor
from itertools import batched
with ThreadPoolExecutor(max_workers=4) as executor:
for results in executor.map(process_batch, batched(data, 100)):
save(results)
This composition submits batches to workers, but you must still consider prefetching, exception propagation, ordering, and pressure on external services. The concurrent.futures guide covers executor behavior.
Asynchronous iterators
itertools.batched() works with synchronous iterables. It cannot consume an async iterator directly. For asynchronous streams, create a helper using async for:
async def async_batched(source, n):
batch = []
async for item in source:
batch.append(item)
if len(batch) == n:
yield tuple(batch)
batch.clear()
if batch:
yield tuple(batch)
Add size validation and strict behavior if your application needs them.
Backpressure
Lazy batching limits source consumption, but it does not automatically provide backpressure for an entire pipeline. If every tuple is immediately placed in an unbounded queue, memory can still grow. Use bounded queues, semaphores, or sequential processing.
Choosing a batch size
The optimal size depends on fixed overhead per call, memory per item, latency targets, external limits, and failure cost. Small batches increase overhead. Large batches increase memory, lock duration, latency, and retry impact. Measure representative workloads rather than copying a universal number.
Retries and idempotency
When retrying a failed batch, operations should be idempotent or use idempotency keys. Otherwise, values processed before a timeout may be duplicated. If a service reports individual failures, record results per item or split failing groups.
Batching is not grouping by key
batched cuts the stream by position. It does not collect records sharing a category. Use itertools.groupby() for adjacent values with the same key. Sort first if all equal keys must be adjacent.
Batching is not a sliding window
batched(values, 2) produces non-overlapping pairs such as (a,b), (c,d). pairwise(values) produces overlapping pairs such as (a,b), (b,c), (c,d). Use the operation that matches the algorithm.
Single-pass iterators
The original iterator is consumed. You cannot assume it can be traversed again. Recreate the source or materialize data only when reuse is necessary. Be cautious with tee(), because uneven consumers can create a large hidden buffer.
Errors inside a batch
Define whether one invalid item rejects the entire group or is routed to a dead-letter stream. Financial and inventory operations may require atomic rollback. Analytics imports may prefer processing valid records and recording failures separately.
Observability
Record batch number, item count, duration, attempts, and error class without logging sensitive content. Throughput and latency metrics make batch-size tuning evidence-based.
Compatibility fallback
For older Python versions, implement the documented recipe using islice or provide a compatibility package. Centralize the fallback so application code uses one interface and can be deleted cleanly after the minimum version increases.
Common mistakes
- Materializing all batches: this removes the lazy-memory benefit.
- Ignoring a short final group: use strict when exact size matters.
- Accepting zero from configuration: validate the size.
- Confusing batches with windows: batches do not overlap.
- Feeding an unbounded queue: memory can still grow.
- Retrying non-idempotent operations: effects may be duplicated.
Complete resilient import example
from itertools import batched
import time
def import_records(records, size=200):
for index, batch in enumerate(batched(records, size), 1):
for attempt in range(1, 4):
try:
insert_batch(batch)
record_metric("batch_ok", len(batch))
break
except TemporaryError:
if attempt == 3:
send_to_failures(index, batch)
raise
time.sleep(2 ** (attempt - 1))
The example limits memory, records progress, and uses exponential backoff. A production implementation should add idempotency, transactions, jitter, cancellation, and structured diagnostics.
Conclusion
itertools.batched() provides a clear, lazy way to consume iterables in chunks. It works with lists, generators, files, cursors, and other streams while making the final-batch policy explicit.
The official Python itertools.batched documentation defines the API. Choose sizes from measurements, control backpressure, and use strict mode whenever incomplete groups represent invalid input.
