Skip to main content

nestly

nestly is a collection of functions designed to ease running software with combinatorial choices of parameters. It can easily do so for “cartesian products” of parameter choices, but can do much more– arbitrary “backwards-looking” dependencies can be used.

The vision here is that we take a fixed set of parameters and generate a single type of output for each defined combination, which can then be combined in some way for comparison and retrieval. We would like to set things up tidily with nested directories for output reflecting nested parameter choices.

The full documentation is available on ReadTheDocs.

Installing

https://travis-ci.org/fhcrc/nestly.svg?branch=master

The easiest way is with pip:

$ pip install nestly

Or, for the latest commit from master:

$ pip install git+git://github.com/fhcrc/nestly.git@master

Python 2.7 is required.

Introductory example

Imagine you’d like to try all possible combinations of the following:

Option

Choices

strategy

approximate, exhaustive

run_count

10, 100, 1000

input file

any file matching inputs/file*

For this we can write a little make_nest.py. The guts are:

nest = Nest()

nest.add('strategy', ('exhaustive', 'approximate'))
nest.add('run_count', [10**i for i in xrange(3)])
nest.add('input_file', glob.glob(os.path.join(input_dir, 'file*')),
        label_func=os.path.basename)

nest.build('runs')

Running make_nest.py, you get a directory tree like:

runs
├── approximate
│   ├── 10
│   │   ├── file1
│   │   │   └── control.json
│   │   ├── file2
│   │       └── control.json
│   ├── 100
│   │   ├── file1
│   │   │   └── control.json
│   │   ├── file2
│   │       └── control.json
│   └── 1000
│       ├── file1
│       │   └── control.json
│       ├── file2
│           └── control.json
└── exhaustive
    ├── 10
    │   ├── file1
    │   │   └── control.json
    │   ├── file2
    │       └── control.json
    ├── 100
    │   ├── file1
    │   │   └── control.json
    │   ├── file2
    │       └── control.json
    └── 1000
        ├── file1
        │   └── control.json
        ├── file2
            └── control.json

With the final control.json reading, for example:

{
    "input_file": "/Users/cmccoy/Development/nestly/examples/basic_nest/inputs/file3",
    "run_count": "1000",
    "strategy": "exhaustive"
}

The control files created then serve as inputs to nestrun for template substition, for example:

nestrun --save-cmd-file command.sh \
        --template='my_command -s {strategy} --count={run_count} {input_file}' \
        $(find runs -name "control.json")

This command runs my_command in all of the tip directories with the appropriate values for the parameters.

This was a “cartesian product” example. The “meal” example in the repository exhibits a setup with more complex dependencies between the nests.

Templates

nestrun takes a template and a list of control.json files with variables to substitute. By default, substitution is performed using the Python built-in str.format method. See the Python Formatter documentation for details on syntax, and examples/jsonrun/do_nestrun.sh for an example.

SCons integration

There is also a nestly.scons module to integrate nestly with the make replacement SCons.

Running the tests

Run:

python setup.py test

The mock library is required, but will be downloaded if missing.

Paper

McCoy CO, Gallagher A, Hoffman NG, Matsen FA (2013) nestly–a framework for running software with nested parameter choices and aggregating results. Bioinformatics 29: 387-388. pubmed

License

nestly source code is freely available under the MIT License.

Metadata

Release files for nestly 0.6.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for nestly 0.6.1
File Size Uploaded Provenance Actions
nestly-0.6.1.tar.gz 15.8 kB Without provenance Details for nestly-0.6.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for nestly 0.6.1
File Interpreter ABI Platform Actions
nestly-0.6.1-py2.py3-none-any.whl Python 3, Python 2 none any Details for nestly-0.6.1-py2.py3-none-any.whl

Total release size: 34.3 kB

Release files / nestly-0.6.1.tar.gz

Download URL nestly-0.6.1.tar.gz
Size 15.8 kB
Tags Source
SHA-256 checksum
How to use checksums
7f74a33a6ba4064ee6a7c0115eb39e517b07d64bdbc37330f365573ed0a0c43f
BLAKE2b-256 checksum
How to use checksums
7480834ae6d3a7aa75438fec5347c37c3009fef905659868c2d20881685d7d61
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No

Release files / nestly-0.6.1-py2.py3-none-any.whl

Download URL nestly-0.6.1-py2.py3-none-any.whl
Size 18.5 kB
Tags Python 2 Python 3
SHA-256 checksum
How to use checksums
655094e0f4c29cadbb61adccd22c8339de4e3f890a6eada67741fa375d08442e
BLAKE2b-256 checksum
How to use checksums
9030979cf1e13afa5a6c351180f347d1bfb097909ab64e37a0897153b0662bd1
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No

Security for nestly 0.6.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page