badger-batcher 0.3.0

Badger Batcher contains useful utilities for batching a sequence on records

Badger Batcher

Badger Batcher contains useful utilities for batching a sequence on records

• Free software: MIT license

• Documentation: https://badger-batcher.readthedocs.io.

Installation

$ pip install badger_batcher

Features

Import Batcher:

>>> from badger_batcher import Batcher

Split records based max limit for batch size:

>>> records = [f"record: {rec}" for rec in range(5)]
>>> batcher = Batcher(records, max_batch_size=2)
>>> batcher.batches()
[['record: 0', 'record: 1'], ['record: 2', 'record: 3'], ['record: 4']]

Split records with max limit for batch size and max limit for record size:

>>> records = [b"aaaa", b"bb", b"ccccc", b"d"]
>>> batcher = Batcher(
... records,
... max_batch_size=2,
... max_record_size=4,
... size_calc_fn=len,
... when_record_size_exceeded="skip",
... )
>>> batcher.batches()
[[b'aaaa', b'bb'], [b'd']]

When processing big chunks of data, consider iterating instead:

>>> import sys

>>> records = (f"record: {rec}" for rec in range(sys.maxsize))
>>> batcher = Batcher(records, max_batch_size=2)
>>> for batch in batcher:
...       # do something for each batch
...       some_fancy_fn(batch)

Credits

This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.

History

0.1.0 (2021-04-09)

• First release on PyPI.