quick_batch
quick_batch is an ultra-simple command-line tool for large batch python-driven processing and transformation. It was designed to be fast to deploy, transparent, and portable. This allows you to scale any processor function that needs to be run over a large set of input data, enabling batch/parallel processing of the input with minimal setup and teardown.
Getting started
All you need to scale batch transformations with quick_batch is a
- transformation function(s) in a
processor.pyfile Dockerfilecontaining a container build appropriate to y our processor- an optional
requirements.txtfile containing required python modules
Document paths to these objects as well as other parameters in a config.yaml config file of the form below.
Under processor you can either define a dockerfile_path to your Dockerfile or an image_name to a pre-built image to be pulled.
data:
input_path: /path/to/your/input/data
output_path: /path/to/your/output/data
log_path: /path/to/your/log/file
queue:
feed_rate: <int - number of examples processed per processor instance>
order_files: <boolean - whether or not to order input files by size>
processor:
dockerfile_path: /path/to/your/Dockerfile OR
image_name: <image_name_to_pull>
requirements_path: /path/to/your/requirements.txt
processor_path: /path/to/your/processor/processor.py
num_processors: <int - instances of processor to run in parallel>
quick_batch will point your processor.py at the input_path defined in this config.yaml and process the files listed in it in parallel at a scale given by your choice of num_processors.
Output will be written to the output_path specified in the configuration file.
You can see the examples directory for examples of valid configs, processors, requirements, and dockerfiles.
Usage
To start processing with your config.yaml use quick_batch's config command at the terminal by typing
quick_batch config /path/to/your/config.yaml
This will start the build and deploy process for processing your data as defined in your config.yaml.
Scaling
Use the scale commoand to manually scale the number of processors / containers running your process
quick_batch scale <num_processors>
Here <num_processors> is an integer >= 1. For example, to scale to 3 parallel processors / containers: quick_batch scale 3
Installation
To install quick_batch, simply use pip:
pip install quick-batch
The processor.py file
Create a processor.py file with the following basic pattern:
import ...
def processor(todos):
for file_name in todos.file_paths_to_process:
# processing code
The todos object will carry in feed_rate number of file names to process in .file_paths_to_process.
Note: the function name processor is mandatory.
Why use quick_batch
quick_batch aims to be
-
dead simple to use: versus standard cloud service batch transformation services that require significant configuration / service understanding
-
ultra fast setup: versus setup of heavier orchestration tools like
airflowormlflow, which may be a hinderance due to time / familiarity / organisational constraints -
100% portable: - use quick_batch on any machine, anywhere
-
processor-invariant: quick_batch works with arbitrary processes, not just machine learning or deep learning tasks.
-
transparent and open source: quick_batch uses Docker under the hood and only abstracts away the not-so-fun stuff - including instantiation, scaling, and teardown. you can still monitor your processing using familiar Docker command-line arguments (like
docker service ls,docker service logs, etc.).
Metadata
Release files for quick-batch 0.1.13
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| quick_batch-0.1.13.tar.gz | 17.2 kB | Without provenance | Details for quick_batch-0.1.13.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| quick_batch-0.1.13-py3-none-any.whl | Python 3 | none | any | Details for quick_batch-0.1.13-py3-none-any.whl |
Total release size: 43.0 kB
Release files / quick_batch-0.1.13.tar.gz
| Download URL | quick_batch-0.1.13.tar.gz |
|---|---|
| Size | 17.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
f513b252003ca3a91925fff4b646af0b55bab3b95abaa86aa57482c28b56e44e
|
|
BLAKE2b-256 checksum How to use checksums |
c15007b96a46278dea5d7c014e44325091525e90ba6916a76b62d3fbf61e2ba9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.9.17
|
Release files / quick_batch-0.1.13-py3-none-any.whl
| Download URL | quick_batch-0.1.13-py3-none-any.whl |
|---|---|
| Size | 25.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
b8b700ca2e51b69ea46706fe04340718b2698f1f7e66592453ca194d41b25e13
|
|
BLAKE2b-256 checksum How to use checksums |
8517f3218faa94e8fadb32debf8234ed0a56157ee4801528df39cfe2982d9e57
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.9.17
|
Security for quick-batch 0.1.13
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.