Skip to main content

A library on top of either pex or conda-packto make your Python code easily available on a cluster

Project description

cluster-pack

cluster-pack is a library on top of either pex or conda-pack to make your Python code easily available on a cluster.

Its goal is to make your prod/dev Python code & libraries easiliy available on any cluster. cluster-pack supports HDFS/S3 as a distributed storage.

The first examples use Skein (a simple library for deploying applications on Apache YARN) and PySpark with HDFS storage. We intend to add more examples for other applications (like Dask, Ray) and S3 storage.

An introducing blog post can be found here.

cluster-pack

Installation

Install with Pip

$ pip install cluster-pack

Install from source

$ git clone https://github.com/criteo/cluster-pack
$ cd cluster-pack
$ pip install .

Prerequisites

cluster-pack supports Python ≥3.7.

Features

  • Ships a package with all the dependencies from your current virtual environment or your conda environment

  • Stores metadata for an environment

  • Supports "under development" mode by taking advantage of pip's editable installs mode, all editable requirements will be uploaded all the time, making local changes directly visible on the cluster

  • Interactive (Jupyter notebook) mode

  • Provides config helpers to directly use the uploaded zip file inside your application

  • Launching jobs from jobs by propagating all artifacts

Basic examples with skein

  1. Interactive mode

  2. Self shipping project

Basic examples with PySpark

  1. PySpark with HDFS on Yarn

  2. Docker with PySpark on S3

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distribution

cluster_pack-0.3.11-py3-none-any.whl (32.0 kB view details)

Uploaded Python 3

File details

Details for the file cluster_pack-0.3.11-py3-none-any.whl.

File metadata

  • Download URL: cluster_pack-0.3.11-py3-none-any.whl
  • Upload date:
  • Size: 32.0 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.1.0 CPython/3.9.13

File hashes

Hashes for cluster_pack-0.3.11-py3-none-any.whl
Algorithm Hash digest
SHA256 3e75d6e5143319d8c42d5f62cd4b504af0e840fc53f1d97e5c8df4beb8d801b3
MD5 031be91cc221da62939d405a58b3e077
BLAKE2b-256 f0765f29deed66e70878ed0bb475128775349cc9ef9c597ea0318e0ce5e7a261

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page