Skip to main content
Pre-release

This release is a pre-release and may not be stable for production use.

License Github Release Python Versions PyPI Monthly Downloads Code of Conduct Documentation Status Tests

HyperShell is an elegant, cross-platform, high-throughput computing utility for processing shell commands over a distributed, asynchronous queue. It is a highly scalable workflow automation tool for many-task scenarios.

Built on Python and tested on Linux, macOS, and Windows.

Other tools may offer similar functionality in some places but not within a single tool and not with the flexibility, ergonomics, and scalability provided by HyperShell.

Design elements include but are not limited to:

  • Client-server: Run the server in stand-alone mode with SQLite or PostgreSQL.
    Scale clients elastically as needed (even down to zero).
  • Cross-platform: trivial to install, run on any platform where Python runs.
    Mix platforms within a running cluster (Server on Linux, Clients on Windows).
  • Staggered launch: Come up gradually to balance the workload.
    Scale to 1000+ nodes, 250k+ workers without crashing the server.
  • Database in-the-loop: persist task metadata across runs.
    Fault-tolerant by default. Automated retries. Task history.
  • User-defined tags: annotate tasks with key:value tags.
    Manage catalogs of large collections of tasks with ease.

Usage

HyperShell is primarily a command-line program. Most users will operate the hs cluster command (hsx for short) in a start-to-finish workflow scenario much like people tend to do with alternatives like xargs, GNU Parallel, or HPC-specific tools like ParaFly or TaskFarmer (NERSC-only) or Launcher (TACC).

seq 1000000 | hsx -t 'echo {}' -N64 --ssh 'a[00-32].cluster' > task.out

Documentation

Documentation is available at hypershell.readthedocs.io. For basic usage information on the command line use: hs --help. For a more comprehensive usage guide on the command line you can view the manual page with man hs.

Support and Contributions

Join the Discord server to post questions, discuss your project, share with the community, keep in touch with announcements and upcoming events!

HyperShell is an open-source project developed on GitHub. If you find bugs or issues with the software please create an Issue. Contributions are welcome in the form of Pull requests for bug fixes, documentation, and minor feature improvements.

We’ve added a Code of Conduct recently, adapted from the Contributor Covenant, version 2.0.

License

HyperShell is released under the Apache Software License (v2).

Citation

If HyperShell has helped in your research please consider citing us.

@inproceedings{lentner_2022,
    author = {Lentner, Geoffrey and Gorenstein, Lev},
    title = {HyperShell v2: Distributed Task Execution for HPC},
    year = {2022},
    isbn = {9781450391610},
    publisher = {Association for Computing Machinery},
    url = {https://doi.org/10.1145/3491418.3535138},
    doi = {10.1145/3491418.3535138},
    booktitle = {Practice and Experience in Advanced Research Computing},
    articleno = {80},
    numpages = {3},
    series = {PEARC '22}
}

Metadata

Release files for hypershell 2.9.0a1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for hypershell 2.9.0a1
File Size Uploaded
hypershell-2.9.0a1.tar.gz 3.8 MB Details

Built distribution (wheel)

Table of built distributions (wheels) for hypershell 2.9.0a1
File Interpreter ABI Platform
hypershell-2.9.0a1-py3-none-any.whl Python 3 none any Details

Total release size: 4.0 MB

Release files / hypershell-2.9.0a1.tar.gz

Download URL hypershell-2.9.0a1.tar.gz
Size 3.8 MB
Tags Source
SHA-256 checksum
How to use checksums
26748792bda191338dcab47c8c6a6f2cd04359984bcdd5c7e878e2fd22411c04
BLAKE2b-256 checksum
How to use checksums
d0c6461e01aff3ab0ed6cdd7ecf8f173d9e9c39e62e3513b8543fcdf336ce12a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.11.31 {"installer":{"name":"uv","version":"0.11.31","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / hypershell-2.9.0a1-py3-none-any.whl

Download URL hypershell-2.9.0a1-py3-none-any.whl
Size 266.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
cbfe9de180cf8753dfb05d563e2947cf3ed002e8d68ac5c6dfbd269decf3d3f6
BLAKE2b-256 checksum
How to use checksums
d8ccf48a501affb54137964b0fdad4f5340bab2eee3a8755779bb709590f0753
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.11.31 {"installer":{"name":"uv","version":"0.11.31","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release history Release notifications | RSS feed

This release

2.9.0a1 This release

2 release files

2.8.2

2 release files

2.8.1

2 release files

2.8.0

2 release files

2.7.2

2 release files

2.7.1

2 release files

2.7.0

2 release files

2.6.6

2 release files

2.6.5

2 release files

2.6.4

2 release files

2.6.2

2 release files

2.6.1

2 release files

2.6.0

2 release files

2.5.1

2 release files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page