paraffin 0.1.3

Parallel execution of DVC stages

paraffin

Paraffin, derived from the Latin phrase parum affinis meaning little related, is a Python package designed to run DVC stages in parallel. While DVC does not currently support this directly, Paraffin provides an effective workaround. For more details, refer to the DVC documentation on parallel stage execution.

[!WARNING] Although DVC supports running multiple dvc repro commands simultaneously, spawning many workers that finish at the same time may lead to DVC lock issues and unexpected failures.

Installation

Install Paraffin via pip:

pip install paraffin

Usage

To use Paraffin, you can run the following to run up to 4 DVC stages in parallel:

paraffin -n 4 <stage names>

If you have pip install dash you can also access the dashboard by running

paraffin --dashboard <stage names>

For more information, run:

paraffin --help

[!TIP] If you are building Python-based workflows with DVC, consider trying our other project ZnTrack for a more Pythonic way to define workflows.