paraffin 0.2.0a3

Parallel execution of DVC stages

paraffin

Paraffin, derived from the Latin phrase parum affinis meaning little related, is a Python package designed to run DVC stages in parallel. While DVC does not currently support this directly, Paraffin provides an effective workaround. For more details, refer to the DVC documentation on parallel stage execution.

[!WARNING] paraffin is still very experimental. Do not use it for production workflows.

Installation

Install Paraffin via pip:

pip install paraffin

Usage

To use Paraffin, you can run the following to queue up the execution of these DVC stages.

paraffin <stage name> <stage name> ... <stage name>
# run max 4 jobs in parallel
celery -A paraffin.worker worker --loglevel=WARNING --concurrency=4

If you have pip install dash you can also access the dashboard by running

paraffin --dashboard <stage names>

For more information, run:

paraffin --help

Labels

You can run paraffin in multiple processes (e.g. on different hardware with a shared file system). To specify where a stage should run, you can assign labels to each worker.

paraffin --labels GPU # on a GPU node
paraffin --label CPU intel # on a CPU node

To configure the stages you need to create a paraffin.yaml file as follows:

labels:
    GPU_TASK:
        - GPU
    CPU_TASK:
        - CPU
    SPECIAL_CPU_TASK:
        - CPU
        - intel

All stages that are not part of the paraffin.yaml will choose any of the available workers.

[!TIP] If you are building Python-based workflows with DVC, consider trying our other project ZnTrack for a more Pythonic way to define workflows.