Skip to main content

HTCdaskGateway

  • A Dask Gateway client extension for heterogeneous cluster mode combining the Kubernetes backend for pain-free scheduler networking, with COFFEA-powered HTCondor workers and/or OKD [coming soon].
  • Latest PyPI version is installed by default and deployed to the COFFEA-DASK notebook on EAF (https://analytics-hub.fnal.gov). A few lines will get you going!
  • The current image for workers/schedulers is: coffeateam/coffea-dask-cc7-gateway:0.7.12-fastjet-3.3.4.0rc9-g8a990fa

Basic usage @ Fermilab EAF

  • Make sure the notebook launched supports this functionality (COFFEA-DASK notebook)
from htcdaskgateway import HTCGateway

gateway = HTCGateway()
cluster = gateway.new_cluster()
cluster

# Scale my cluster to 5 HTCondor workers
cluster.scale(5)

# Obtain a client for connecting to your cluster scheduler
# Your cluster should be ready to take requests
client = cluster.get_client()
client

# When computations are finished, shutdown the cluster
cluster.shutdown()

Other functions worth checking out

  • This is a multi-tenant environment, and you are authenticated via JupyterHub Oauth which means that you can create as many* clusters as you wish
  • To list your clusters:
# Verify that the gateway is responding to requests by asking to list all its clusters
clusters = gateway.list_clusters()
clusters
  • To connect to a specific cluster from the list:
cluster = gateway.connect(cluster_name)
cluster
cluster.shutdown()
  • To gracefully close the cluster and remove HTCondor worker jobs associated to it:
cluster.shutdown()
  • There are widgets implemented by Dask Gateway. Make sure to give them a try from your EAF COFFEA notebook, just execute the the client and cluster commands (after properly initializing them) in a cell like:
-------------
cluster = gateway.new_cluster()
cluster
< Widget will appear after this step>
-------------
client = cluster.get_client()
client
< Widget will apear after this step >
-------------
cluster
< Widget will appear after this step >

Metadata

Release files for htcdaskgateway 0.1.25

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for htcdaskgateway 0.1.25
File Size Uploaded Provenance Actions
htcdaskgateway-0.1.25.tar.gz 9.1 kB Without provenance Details for htcdaskgateway-0.1.25.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for htcdaskgateway 0.1.25
File Interpreter ABI Platform Actions
htcdaskgateway-0.1.25-py3-none-any.whl Python 3 none any Details for htcdaskgateway-0.1.25-py3-none-any.whl

Total release size: 21.9 kB

Release files / htcdaskgateway-0.1.25.tar.gz

Download URL htcdaskgateway-0.1.25.tar.gz
Size 9.1 kB
Tags Source
SHA-256 checksum
How to use checksums
e00da02e02f0d46fc1e09af2fdb4ad07877c432dbfb15e302e6131158e122f24
BLAKE2b-256 checksum
How to use checksums
0350d55c08dfd3620682a5628d4f53c38ffe6bf0f09c0f3852050d6d136300e5
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/5.0.0 CPython/3.10.19

Release files / htcdaskgateway-0.1.25-py3-none-any.whl

Download URL htcdaskgateway-0.1.25-py3-none-any.whl
Size 12.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
a8fba7def160f00a6f4e9367d388c05c58529b168f5c31291ea441dc86a4fd11
BLAKE2b-256 checksum
How to use checksums
391f52307262f4d70feb7e9a3dc1f85a081ac55d6507e049591b23cafca56c81
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/5.0.0 CPython/3.10.19

Security for htcdaskgateway 0.1.25

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page