Skip to main content

Wrapper for tqdm and joblib to have a progressbar while batch processing

Project description

tqdm_batch

Batch processing using joblib including tqdm progress bars

Add batch processing to joblib, including tqdm progress bars.

Install

pip install tqdm_batch

Usage

Process a list of items using a function.

from tqdm_batch import batch_process
import random
import time

def batch_process_function(row, some_var):
    time.sleep(0.01)
    return row + random.randint(0, some_var)

N = 1_000
items = range(N)

result = batch_process(
    items,
    batch_process_function,
    some_var=42,
    n_workers=6,
    sep_progress=True,
)

Multi batch processing with progress bars

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

tqdm_batch-0.1.0.tar.gz (4.2 kB view hashes)

Uploaded Source

Built Distribution

tqdm_batch-0.1.0-py3-none-any.whl (5.1 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page