SteamRoller 
Getting started with an example
First and foremost, steamroller builds upon the [SCons](https://scons.org/), which has a [very in-depth MAN page](https://scons.org/doc/production/HTML/scons-man.html) and [API](https://scons.org/doc/latest/HTML/scons-api/index.html), but a probably-more-useful [user guide](https://scons.org/doc/production/HTML/scons-user/index.html).
Assuming you’re starting in an empty directory named after the experiment (e.g. ~/my_experiment) and have a recent version of Python 3 on your path:
$ python -m venv local $ source local/bin/activate (my_experiment) $ pip install pip -U (my_experiment) $ pip install steamroller
Steamroller can create a simple dummy project to help get started:
(my_experiment) $ steamroller --new_project
As it runs, this will print out information about what it’s creating, and why. The only difference from a basic SCons project, as can be seen in the example SConstruct file, is that Environment is imported from steamroller rather than scons. Other than the “–new project” option, the steamroller command behaves like the normal scons command, and this will run the experiment one step at a time:
(my_experiment) $ steamroller -Q
However, steamroller makes it easy to instead run on a grid with maximal parallelism, simply by setting a few variables (these will depend on the details of your particular grid, and typically you would want to put these variables in “custom.py” so that you can use the simple steamroller -Q command):
(my_experiment) $ steamroller -Q STEAMROLLER_ENGINE=slurm CPU_QUEUE=parallel CPU_ACCOUNT=tlippin1 GPU_QUEUE=parallel GPU_ACCOUNT=tlippin1 GPU_COUNT=0
This will submit the experiment to the grid and immediately return: you can check the status of the experiment by running sacct -s R,PD,F (work is underway to make monitoring a first-order aspect of steamroller, but for the moment you want to be careful not to resubmit while jobs from a previous invocation are still pending). See below for the broader set of steamroller variables that can be overridden.
Scaling up
As described above, steamroller is indistinguishable from the [SCons](https://scons.org/) system itself, which is very well-documented: so, one could simply use it in this fashion, running experiments in serial, locally (e.g. on a laptop). Steamroller’s power comes from the ability to easily flip a switch to take advantage of the massively parallel architecture of a high-performance compute cluster, without changing the underlying code.
When following the pattern described above, this is accomplished by creating a “custom.py” file in the experiment directory and setting a few variables in it. The most important is STEAMROLLER_ENGINE, which defaults to “local”, but can alternatively be set to “slurm”, “univa”, or “sge” (depending on what grid system is used). There are a few other special variables, all starting with “STEAMROLLER”, though bear in mind these can also be set when a particular build rule is called (see, above, how the “PARAM” and “SPLIT” variables are being set):
STEAMROLLER_ACCOUNT = "my_account" STEAMROLLER_QUEUE = "some_queue" STEAMROLLER_TIME = "06:00:00" STEAMROLLER_MEMORY = "64G" STEAMROLLER_GPU_COUNT = 1
There are a few variables that steamroller uses internally and that you should only set if you really know what you’re doing:
STEAMROLLER_SUBMIT_STRING STEAMROLLER_NAME_PREFIX STEAMROLLER_NAME STEAMROLLER_LOG STEAMROLLER_WORKING_DIRECTORY STEAMROLLER_DEPENDENCIES STEAMROLLER_SHELL
Basically, if there’s no grid (e.g. you’re running on a laptop), steamroller should just behave like SCons and run each needed task as a simple command-line invocation, in linear order that respects the dependencies. If a grid is specified (e.g. STEAMROLLER_ENGINE = “slurm”), and is in fact available, steamroller will instead submit each task in the appropriate order, propagating the task IDs as appropriate so that the grid jobs respect the dependency structure.
FAQ
1 maintainer
Metadata
1 maintainer
Release files for steamroller 3.0.5
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| steamroller-3.0.5.tar.gz | 31.6 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| steamroller-3.0.5-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 68.4 kB
Release files / steamroller-3.0.5.tar.gz
| Download URL | steamroller-3.0.5.tar.gz |
|---|---|
| Size | 31.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
ea792eae2ab44464e7eb3dc5d73a4aff525147e54d8bb79c32ecba877171ff53
|
|
BLAKE2b-256 checksum How to use checksums |
d2cc93fc754e4d55cad44fe3f7e4f4771b23911631e0d81e6ea0ad786d9e0e06
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.5
|
Release files / steamroller-3.0.5-py3-none-any.whl
| Download URL | steamroller-3.0.5-py3-none-any.whl |
|---|---|
| Size | 36.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
667f71e87f2bfef16c4f8132c31d7126618af771b1a995bf74a393c9c920309a
|
|
BLAKE2b-256 checksum How to use checksums |
bca11a9f6d4657ea5b906e05c68eb7003701842f443f1415a3ea0a6516b6f8ec
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.5
|
Security for steamroller 3.0.5
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
