yabp 0.1.1

Yet another batch processor. Small python module to do batch processing. Use this when you want something lightweight and quick to setup. For usecases when writing batch processing boilerplate code is too repetitive and boring and Apache Beam/Ray is overkill

YABP - Yet another Batch Processor

I wrote this because I needed a generic way to process large data (often in the form of lists) in batches. Other libraries are generally more suited for complexer problem and often require more effort to setup and is often overkill for small projects. Also provides a decorator for even more simple usage.

Another library that is similar and also accomplish the same goal: https://github.com/gillespied/batch_please/tree/main

This module provides functionality for processing data in batches with options for progress tracking, retries, and saving results to a file. It includes a BatchProcessor class and a batch_processor_decorator for easy integration into your projects.

Installation

pip install yabp==0.1

Usage

BatchProcessor Class

1. Import the BatchProcessor:

   from batchprocessor import BatchProcessor
   

2. Define a function to process each batch:

   def process_batch(batch):
       # Your processing logic here
       return {"processed": batch, "status": "success"}
   

3. Initialize the BatchProcessor:

   processor = BatchProcessor(
       iterable=your_data,
       batch_size=5,
       progress=True,
       save_to_file="output.json",
       retries=2,
       retry_delay=1.0,
   )
   

4. Process the batches:

   results = processor.process(process_batch)
   print("Final Results:", results)
   

Decorator version

1. Import the decorator:

   from batchprocessor import batch_process
   

2. Decorate your batch processing function:

   @batch_process(
       batch_size=5,
       progress=True,
       save_to_file="output_decorator.json",
       retries=2,
       retry_delay=1.0,
   )
   def process_batch(batch):
       # Your processing logic here
       return {"processed": batch, "status": "success"}
   

3. Call the decorated function with your data:

   process_batch(your_data)
   

Features

• Batch Processing: Process data in specified batch sizes.
• Progress Tracking: Optionally display a progress bar.
• Retries: Automatically retry failed batch processing with a specified delay.
• Save Results: Save batch results to a JSON file.