Skip to main content

Demo batchcreate library

Project description

Batchcreator takes an array of records as input and splits it into suitably sized batches of records which can be further processed or passed to any other system/s.

Quick start:


$ pip install batchcreate

2.Import BatchCreator iterator class and instantiate it. You can use below parameters to define output batch limits. These parameters are optional. If neither of these parameters is specified then the default values will be used. The default limits as :

  • max_record_size=1MB

    The maximum size limit for a record in the output batch. Any record with larger size than this will be skipped from batching.

  • max_batch_size=5MB

    The maximum size limit for a batch.

  • max_batch_num_records=500

    The maximum number of records limit for a batch. BatchCreator will put maximum these many records per batch provided batch size satisfies the limit.

from batchcreate import BatchCreator

batches = BatchCreator(records,

3.The iterable BatchCreator object can give suitable batches as needed on iteration. The BatchCreator object can be used in a regular 'for' loop.

for batch in batches:
    print(batch) #batch processing here


batchItr = iter(batches)
print(next(batchItr)) #batch processing here

4.BatchCreator can return the list of all the batches as well.

batches = BatchCreator(records).batches()


Library available at : batchcreate

Documentation at : Batchcreate Documentation

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

batchcreate-0.1.5.tar.gz (3.1 kB view hashes)

Uploaded Source

Built Distribution

batchcreate-0.1.5-py3-none-any.whl (3.7 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page