sneks
Get your snakes in a row.
sneks lets you launch a Dask cluster in the cloud, matched to your local software environment*, in a single line of code. No more dependency mismatches or Docker image building.
from sneks import get_client
client = get_client()
*your local Poetry or PDM environment. You must use poetry or PDM. Locking package managers are what sensible people use, and you are sensible.
Neat! Sneks also supports ARM clusters! Just pass ARM instances in scheduler_instace_types=, worker_instace_types= and cross your fingers that all your dependencies have cross-arch wheels!
Installation
poetry add -G dev sneks-sync
A full example:
mkdir example && cd example
poetry init -n
poetry add -G dev sneks-sync
poetry add distributed==2022.5.2 dask==2022.5.2 bokeh pandas pyarrow # and whatever else you want
from sneks import get_client
import dask.dataframe as dd
client = get_client(name="on-a-plane")
ddf = dd.read_parquet(
"s3://nyc-tlc/trip data/yellow_tripdata_2012-*.parquet",
)
print(ddf.groupby('passenger_count').trip_distance.mean().compute())
Oh wait, we forgot to install a dependency!
poetry add foobar
When we reconnect to the cluster (using the same name), the dependencies on the cluster update automatically.
from sneks import get_client
import dask.dataframe as dd
import foobar # ah, how could we forget this critical tool
client = get_client(name="on-a-plane")
ddf = dd.read_csv(
"s3://nyc-tlc/csv_backup/yellow_tripdata_2012-*.csv",
)
means = ddf.groupby('passenger_count').trip_distance.mean()
means.apply(foobar.optimize).compute()
Caveats
This is still a proof-of-concept-level package. It's been used personally quite a bit, and proven reliable, but use at your own risk.
Metadata
Release files for sneks-sync 0.6.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| sneks_sync-0.6.2.tar.gz | 11.2 kB | Without provenance | Details for sneks_sync-0.6.2.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| sneks_sync-0.6.2-py3-none-any.whl | Python 3 | none | any | Details for sneks_sync-0.6.2-py3-none-any.whl |
Total release size: 24.2 kB
Release files / sneks_sync-0.6.2.tar.gz
| Download URL | sneks_sync-0.6.2.tar.gz |
|---|---|
| Size | 11.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
8ed946ded1f8d5f3ad54fef52d8c2e10256154b95e6f4de6a1ce12cca7dbc3e6
|
|
BLAKE2b-256 checksum How to use checksums |
20dfd44b1344d94176ec193bf0c758ad621e0f6521070b402a6d0e236dd85c37
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.4.0 CPython/3.9.4 Darwin/21.6.0
|
Release files / sneks_sync-0.6.2-py3-none-any.whl
| Download URL | sneks_sync-0.6.2-py3-none-any.whl |
|---|---|
| Size | 13.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
449b1fabae27d51a21b227fc6ca006d9e8e3bc3397bdf596360be3fc817b03e0
|
|
BLAKE2b-256 checksum How to use checksums |
41904dbf5861002cc149512573f9286a2739a6cb060ef097cfaf35d1b34efa55
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.4.0 CPython/3.9.4 Darwin/21.6.0
|
Security for sneks-sync 0.6.2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.