Skip to main content

Hyper-parallel multi-node task execution engine

Project description

Parallelic

Parallelic is a hyperparallel multi-node task execution engine with shared data and wokspace capabilities.

Note of warning

Parallelic is not a containerization/sandboxing engine. It does not constitute a full task isolation, and provides no guarantee of such. That may change in the future, and feel free to contribute your code towards that goal, but in the mean time, keep this in consideration when giving access to a Parallelic system to third parties.

Installation

From git

  1. Clone the git repo locally.
  2. Download python3(.7) and corresponding pip
  3. Install Poetry
  4. Run poetry install to create a virtualenv and install dependencies
    At this point, you can use parallelic through
    poetry run parallelic
  5. Run poetry build to build a wheel
  6. Run pip install dist/parallelic-[version]-py3-none-any.whl
    Now you can use parallelic without poetry:
    python -m parallelic

From pip

  1. Run pip install parallelic

Usage

Running a task

To run an already defined task, you upload the task package (a zipped up task root directory) via the Parallelic WebUI, or Parallelic CLI client, to the Parallelic manager instance.
You may need to provide access credentials before being allowed to upload the task package, as per your Parallelic system configuration.
From there, the Parallelic manager instance will take care of everything else.

Defining a task

The task root contains a task.toml file, that contains metadata required for the manager to set up and prepare resources for the compute nodes in order to run the particullar task.
If the task requires no additional files, the task definition can be only the task.toml file.

The directory tree doesn't follow a particullar convention, and can be different from task to task. The task definition file should contain a section where the entrypoint and working directory are defined. Both the entrypoint and the working directory have to be relative to the task root.

Credits

Package maintained by Trickster Animations

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for parallelic, version 0.1.3
Filename, size File type Python version Upload date Hashes
Filename, size parallelic-0.1.3-py3-none-any.whl (22.7 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size parallelic-0.1.3.tar.gz (8.7 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page