cluster-pack
cluster-pack is a library on top of either pex to make your Python code easily available on a cluster.
Its goal is to make your prod/dev Python code & libraries easiliy available on any cluster. cluster-pack supports HDFS/S3 as a distributed storage.
The first examples use Skein (a simple library for deploying applications on Apache YARN) and PySpark with HDFS storage. We intend to add more examples for other applications (like Dask, Ray) and S3 storage.
An introducing blog post can be found here.
Installation
Install with Pip
$ pip install cluster-pack
Install from source
$ git clone https://github.com/criteo/cluster-pack
$ cd cluster-pack
$ pip install .
Prerequisites
cluster-pack supports Python ≥3.9.
Feature flags
- C_PACK_USER: override the current user for HDFS path generation and Skein impersonation
- When set, this value is used instead of the system user (from
getpass.getuser()) - Useful for running jobs as a different user or in environments where the system user doesn't match the HDFS user
- If not set or empty, falls back to the current system user
- When set, this value is used instead of the system user (from
Features
-
Ships a package with all the dependencies from your current virtual environment
-
Stores metadata for an environment
-
Supports "under development" mode by taking advantage of pip's editable installs mode, all editable requirements will be uploaded all the time, making local changes directly visible on the cluster
-
Interactive (Jupyter notebook) mode
-
Provides config helpers to directly use the uploaded zip file inside your application
-
Launching jobs from jobs by propagating all artifacts
Basic examples with skein
Basic examples with PySpark
Info
Conda is no longer supported
4 maintainers
Release files for cluster-pack 0.3.19
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| cluster_pack-0.3.19-py3-none-any.whl | Python 3 | none | any | Details |
Release files / cluster_pack-0.3.19-py3-none-any.whl
| Download URL | cluster_pack-0.3.19-py3-none-any.whl |
|---|---|
| Size | 30.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
bc30d1cee080ef4e1c4bed021e4c286ade3745a2e686ef7c0792f5dbb805cb11
|
|
BLAKE2b-256 checksum How to use checksums |
46d5d7769cd94535699b921639d998b6020ef3030b6ba27c951e1702c6cc616b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.11.10
|
