Skip to main content

A library on top of either pex or conda-packto make your Python code easily available on a cluster

Project description

cluster-pack

cluster-pack is a library on top of either pex or conda-pack to make your Python code easily available on a cluster.

Its goal is to make your prod/dev Python code & libraries easiliy available on any cluster. cluster-pack supports HDFS/S3 as a distributed storage.

The first examples use Skein. We will add more examples for other applications (like Pyspark, Dask) with other compute clusters (like mesos, kubernetes) soon.

Installation

Install with Pip

$ pip install cluster-pack

Install from source

$ git clone https://github.com/criteo/cluster-pack
$ cd cluster-pack
$ pip install .

Prerequisites

cluster-pack supports Python ≥3.6.

Features

  • ships a package with all the dependencies from your current virtual environment or your conda environment
  • provides config helpers to inject those dependencies to your application
  • when using pip with pex cluster-pack takes advantage of pip's editable installs mode, all editable requirements will be uploaded all the time separatly, making local changes direclty visible on the cluster, and not requiring to regenerate the packacke with all the dependencies again

Basic examples with skein

  1. Interactive mode

  2. Self shipping project

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for cluster-pack, version 0.0.6
Filename, size File type Python version Upload date Hashes
Filename, size cluster_pack-0.0.6-py3-none-any.whl (21.6 kB) File type Wheel Python version py3 Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page