HyperShell is an elegant, cross-platform, high-throughput computing utility for processing shell commands over a distributed, asynchronous queue. It is a highly scalable workflow automation tool for many-task scenarios.
Built on Python and tested on Linux, macOS, and Windows.
Several tools offer similar functionality but not all together in a single tool with the user ergonomics we provide. Novel design elements include but are not limited to
Cross-platform: run on any platform where Python runs. In fact, the server and client can run on different platforms in the same cluster.
Client-server: workloads do not need to be monolithic. Run the server as a stand-alone service with SQLite or Postgres as a persistent database and dynamically scale clients as needed.
Staggered launch: At the largest scales (1000s of nodes, 100k+ of workers), the launch process can be challenging. Come up gradually to balance the workload.
Database in-the-loop: run in-memory for quick, ad-hoc workloads. Otherwise, include a database for persistence, recovery when restarting, and search.
Documentation
Documentation is available at hypershell.readthedocs.io. For basic usage information on the command line use: hs --help. For a more comprehensive usage guide on the command line you can view the manual page with man hs.
Contributions
Contributions are welcome. If you find bugs or have questions, open an Issue here. We’ve added a Code of Conduct recently, adapted from the Contributor Covenant, version 2.0.
Citation
If HyperShell has helped in your research please consider citing us.
@inproceedings{lentner_2022,
author = {Lentner, Geoffrey and Gorenstein, Lev},
title = {HyperShell v2: Distributed Task Execution for HPC},
year = {2022},
isbn = {9781450391610},
publisher = {Association for Computing Machinery},
url = {https://doi.org/10.1145/3491418.3535138},
doi = {10.1145/3491418.3535138},
booktitle = {Practice and Experience in Advanced Research Computing},
articleno = {80},
numpages = {3},
series = {PEARC '22}
}1 maintainer
Metadata
1 maintainer
Release files for hypershell 2.7.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| hypershell-2.7.0.tar.gz | 3.6 MB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| hypershell-2.7.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 3.7 MB
Release files / hypershell-2.7.0.tar.gz
| Download URL | hypershell-2.7.0.tar.gz |
|---|---|
| Size | 3.6 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
65737426ceccf0a9518f09c76af1b49a9f05406af740888b3e7d9501d48f00c3
|
|
BLAKE2b-256 checksum How to use checksums |
df5503bc8064a9de6d3a2003b427f2eb0b59ac4881b35c8734c91841eacb1fb0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.7.8
|
Release files / hypershell-2.7.0-py3-none-any.whl
| Download URL | hypershell-2.7.0-py3-none-any.whl |
|---|---|
| Size | 103.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
382f9a51a76a0284b9b03b450706828cec0b1a95f747ee385bb39f5be4e6b795
|
|
BLAKE2b-256 checksum How to use checksums |
fce830a061b8c9c6f22f0650b1b01ce39710d314ed8b7297779e79f48f28efcb
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.7.8
|
