paraffin
Paraffin, derived from the Latin phrase parum affinis meaning
little related, is a Python package designed to run DVC
stages in parallel. While DVC does not currently support this directly, Paraffin
provides an effective workaround. For more details, refer to the DVC
documentation on
parallel stage execution.
[!WARNING]
paraffinis still very experimental. Do not use it for production workflows.
Installation
Install Paraffin via pip:
pip install paraffin
Usage
https://github.com/user-attachments/assets/c248e669-7737-450b-9fd7-5b9b8e82605a
paraffin submit
You can submit your current DVC workflow to a database file paraffin.db for later execution.
[!TIP] The paraffin submit command supports globing patterns.
paraffin submit # submit all stages
paraffin submit C_AddNodeNumbers "A*" # select which stages to submit
paraffin submit --help # more information
paraffin worker
A submitted job will be executed by paraffin workers.
To start a worker you can run paraffin worker.
The worker will pick up all the jobs in the workeres queue and close once finished.
You can specify the number of stages a worker should process in parallel by using paraffin worker --jobs <n>.
Alternatively, you can start more workers by running the command multiple times.
paraffin worker
paraffin worker --help # more information
paraffin ui
Paraffin ships with a web application for visualizing the progress. You can start it using
paraffin ui
paraffin ui --help # more information
The UI allows you to visualize the progress in real-time, restart jobs and manage workers.
https://github.com/user-attachments/assets/034325fd-7035-434f-9eb8-b47ae4ecbb86
Queue Labels
To fine-tune execution, you can assign stages to specific Celery queues, allowing you to manage execution across different environments or hardware setups.
Define queues in a paraffin.yaml file:
queue:
"B_X*": BQueue
"A_X_AddNodeNumbers": AQueue
Then, start a worker with specified queues, such as celery (default) and AQueue:
paraffin worker -q AQueue,default
All stages not assigned to a queue in paraffin.yaml will default to the default queue.
[!TIP] If you are building Python-based workflows with DVC, consider trying our other project ZnTrack for a more Pythonic way to define workflows.
Metadata
Release files for paraffin 0.3.7
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| paraffin-0.3.7.tar.gz | 793.3 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| paraffin-0.3.7-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 1.5 MB
Release files / paraffin-0.3.7.tar.gz
| Download URL | paraffin-0.3.7.tar.gz |
|---|---|
| Size | 793.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
362e0cf56cff91b15e9afa1f6af523ba66cc8c2d60cbcd431874cc07f0b71b1c
|
|
BLAKE2b-256 checksum How to use checksums |
d013235dc9c6c38c22a6c5c7e94fb5cacadffd549d7c3eba7b591e4f123a9251
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.6.2
|
Release files / paraffin-0.3.7-py3-none-any.whl
| Download URL | paraffin-0.3.7-py3-none-any.whl |
|---|---|
| Size | 658.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
e5bc1c97f21c73128e34435b014528176d989f754d291b93a1a03928b80eb5d5
|
|
BLAKE2b-256 checksum How to use checksums |
f13c7227ccfd1d76b05595c5b45d23f26a6deaa9d89f4509569f768e36b40b96
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.6.2
|
Security for paraffin 0.3.7
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
