hypershell

HyperShell is an elegant, cross-platform, high-throughput computing utility for processing shell commands over a distributed, asynchronous queue. It is a highly scalable workflow automation tool for many-task scenarios.

Built on Python and tested on Linux, macOS, and Windows.

Several tools offer similar functionality but not all together in a single tool with the user ergonomics we provide. Novel design elements include but are not limited to

Cross-platform: run on any platform where Python runs. In fact, the server and client can run on different platforms in the same cluster.
Client-server: workloads do not need to be monolithic. Run the server as a stand-alone service with SQLite or Postgres as a persistent database and dynamically scale clients as needed.
Staggered launch: At the largest scales (1000s of nodes, 100k+ of workers), the launch process can be challenging. Come up gradually to balance the workload.
Database in-the-loop: run in-memory for quick, ad-hoc workloads. Otherwise, include a database for persistence, recovery when restarting, and search.

Documentation

Documentation is available at hyper-shell.readthedocs.io. For basic usage information on the command line use: hs --help. For a more comprehensive usage guide on the command line you can view the manual page with man hs.

Contributions

Contributions are welcome. If you find bugs or have questions, open an Issue here. We’ve added a Code of Conduct recently, adapted from the Contributor Covenant, version 2.0.

Citation

If HyperShell has helped in your research please consider citing us.

@inproceedings{lentner_2022,
    author = {Lentner, Geoffrey and Gorenstein, Lev},
    title = {HyperShell v2: Distributed Task Execution for HPC},
    year = {2022},
    isbn = {9781450391610},
    publisher = {Association for Computing Machinery},
    url = {https://doi.org/10.1145/3491418.3535138},
    doi = {10.1145/3491418.3535138},
    booktitle = {Practice and Experience in Advanced Research Computing},
    articleno = {80},
    numpages = {3},
    series = {PEARC '22}
}

Metadata

Release files for hypershell 2.5.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for hypershell 2.5.1
File	Size	Uploaded
hypershell-2.5.1.tar.gz	78.6 kB	Apr 5, 2025	Details

Built distribution (wheel)

Table of built distributions (wheels) for hypershell 2.5.1
File	Interpreter	ABI	Platform	Reset
hypershell-2.5.1-py3-none-any.whl 96.0 kB Apr 5, 2025	Python 3	none	any	Details

Total release size: 174.7 kB

Release files / hypershell-2.5.1.tar.gz

Download URL	hypershell-2.5.1.tar.gz
Size	78.6 kB
Tags	Source
SHA-256 checksum How to use checksums	`0763c2b994a9942676b59c39c21036fb121403f7c9ac26826917564cd0a40da7`
BLAKE2b-256 checksum How to use checksums	`d0c91da7d8087ac373b4c8d50e045ae349ad93a5455d88a7a1de14791cf55c1b`
Upload date	Apr 5, 2025
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`uv/0.6.12`

Release files / hypershell-2.5.1-py3-none-any.whl

Download URL	hypershell-2.5.1-py3-none-any.whl
Size	96.0 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`600f6cdca77d91982121efd49130848f9a83333a36bc4f9f60eb799de2ec5587`
BLAKE2b-256 checksum How to use checksums	`4c328b3b862f0cd67dac02d3ad62fb6262196d0091ca5db2c34ca484405e4cc9`
Upload date	Apr 5, 2025
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`uv/0.6.12`