nestly
nestly is a collection of functions designed to ease running software with combinatorial choices of parameters. It can easily do so for “cartesian products” of parameter choices, but can do much more– arbitrary “backwards-looking” dependencies can be used.
The vision here is that we take a fixed set of parameters and generate a single type of output for each defined combination, which can then be combined in some way for comparison and retrieval. We would like to set things up tidily with nested directories for output reflecting nested parameter choices.
The full documentation is available on ReadTheDocs.
Installing
The easiest way is with pip:
$ pip install nestly
Or, for the latest commit from master:
$ pip install git+git://github.com/fhcrc/nestly.git@master
Python 2.7 is required.
Introductory example
Imagine you’d like to try all possible combinations of the following:
Option |
Choices |
strategy |
approximate, exhaustive |
run_count |
10, 100, 1000 |
input file |
any file matching inputs/file* |
For this we can write a little make_nest.py. The guts are:
nest = Nest()
nest.add('strategy', ('exhaustive', 'approximate'))
nest.add('run_count', [10**i for i in xrange(3)])
nest.add('input_file', glob.glob(os.path.join(input_dir, 'file*')),
label_func=os.path.basename)
nest.build('runs')
Running make_nest.py, you get a directory tree like:
runs
├── approximate
│ ├── 10
│ │ ├── file1
│ │ │ └── control.json
│ │ ├── file2
│ │ └── control.json
│ ├── 100
│ │ ├── file1
│ │ │ └── control.json
│ │ ├── file2
│ │ └── control.json
│ └── 1000
│ ├── file1
│ │ └── control.json
│ ├── file2
│ └── control.json
└── exhaustive
├── 10
│ ├── file1
│ │ └── control.json
│ ├── file2
│ └── control.json
├── 100
│ ├── file1
│ │ └── control.json
│ ├── file2
│ └── control.json
└── 1000
├── file1
│ └── control.json
├── file2
└── control.json
With the final control.json reading, for example:
{
"input_file": "/Users/cmccoy/Development/nestly/examples/basic_nest/inputs/file3",
"run_count": "1000",
"strategy": "exhaustive"
}
The control files created then serve as inputs to nestrun for template substition, for example:
nestrun --save-cmd-file command.sh \
--template='my_command -s {strategy} --count={run_count} {input_file}' \
$(find runs -name "control.json")
This command runs my_command in all of the tip directories with the appropriate values for the parameters.
This was a “cartesian product” example. The “meal” example in the repository exhibits a setup with more complex dependencies between the nests.
Templates
nestrun takes a template and a list of control.json files with variables to substitute. By default, substitution is performed using the Python built-in str.format method. See the Python Formatter documentation for details on syntax, and examples/jsonrun/do_nestrun.sh for an example.
SCons integration
There is also a nestly.scons module to integrate nestly with the make replacement SCons.
Running the tests
Run:
python setup.py test
The mock library is required, but will be downloaded if missing.
Paper
McCoy CO, Gallagher A, Hoffman NG, Matsen FA (2013) nestly–a framework for running software with nested parameter choices and aggregating results. Bioinformatics 29: 387-388. pubmed
License
nestly source code is freely available under the MIT License.
Metadata
Release files for nestly 0.6.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| nestly-0.6.1.tar.gz | 15.8 kB | Without provenance | Details for nestly-0.6.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| nestly-0.6.1-py2.py3-none-any.whl | Python 3, Python 2 | none | any | Details for nestly-0.6.1-py2.py3-none-any.whl |
Total release size: 34.3 kB
Release files / nestly-0.6.1.tar.gz
| Download URL | nestly-0.6.1.tar.gz |
|---|---|
| Size | 15.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
7f74a33a6ba4064ee6a7c0115eb39e517b07d64bdbc37330f365573ed0a0c43f
|
|
BLAKE2b-256 checksum How to use checksums |
7480834ae6d3a7aa75438fec5347c37c3009fef905659868c2d20881685d7d61
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
Release files / nestly-0.6.1-py2.py3-none-any.whl
| Download URL | nestly-0.6.1-py2.py3-none-any.whl |
|---|---|
| Size | 18.5 kB |
| Tags | Python 2 Python 3 |
|
SHA-256 checksum How to use checksums |
655094e0f4c29cadbb61adccd22c8339de4e3f890a6eada67741fa375d08442e
|
|
BLAKE2b-256 checksum How to use checksums |
9030979cf1e13afa5a6c351180f347d1bfb097909ab64e37a0897153b0662bd1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
Security for nestly 0.6.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.