Launch a Dask cluster from a virtual environment
Project description
sneks
Get your snakes in a row.
sneks
lets you launch a Dask cluster in the cloud, matched to your local software environment*, in a single line of code. No more dependency mismatches or Docker image building.
from sneks import get_client
client = get_client()
*your local Poetry or PDM environment. You must use poetry or PDM. Locking package managers are what sensible people use, and you are sensible.
Installation
poetry add -G dev sneks-sync
A full example:
mkdir example && cd example
poetry init -n
poetry add -G dev sneks-sync
poetry add distributed==2022.5.2 dask==2022.5.2 bokeh pandas pyarrow # and whatever else you want
from sneks import get_client
import dask.dataframe as dd
client = get_client(name="on-a-plane")
ddf = dd.read_parquet(
"s3://nyc-tlc/trip data/yellow_tripdata_2012-*.parquet",
)
print(ddf.groupby('passenger_count').trip_distance.mean().compute())
Oh wait, we forgot to install a dependency!
poetry add foobar
When we reconnect to the cluster (using the same name), the dependencies on the cluster update automatically.
from sneks import get_client
import dask.dataframe as dd
import foobar # ah, how could we forget this critical tool
client = get_client(name="on-a-plane")
ddf = dd.read_csv(
"s3://nyc-tlc/csv_backup/yellow_tripdata_2012-*.csv",
)
means = ddf.groupby('passenger_count').trip_distance.mean()
means.apply(foobar.optimize).compute()
Caveats
This is still a proof-of-concept-level package. It's been used personally quite a bit, and proven reliable, but use at your own risk.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Hashes for sneks_sync-0.2.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5020fc64205890468fa49d80abe4f7c2a8a3700bc85efbb0b5a011d9caaef8a7 |
|
MD5 | a8502d0f7aeb59e7a5cddec0689a5fc2 |
|
BLAKE2b-256 | 7cdcda67a98ca53ee23bcd9c27b8db4652ac3a9f3fdd307d7d11f2af7838cf7c |