This is a pre-production deployment of Warehouse, however changes made here WILL affect the production instance of PyPI.
Latest Version Dependencies status unknown Test status unknown Test coverage unknown
Project Description

nestly is a collection of functions designed to ease running software with combinatorial choices of parameters. It can easily do so for “cartesian products” of parameter choices, but can do much more– arbitrary “backwards-looking” dependencies can be used.

The vision here is that we take a fixed set of parameters and generate a single type of output for each defined combination, which can then be combined in some way for comparison and retrieval. We would like to set things up tidily with nested directories for output reflecting nested parameter choices.

The full documentation is available on ReadTheDocs.

Installing

The easiest way is with pip:

$ pip install nestly

Or, for the latest commit from master:

$ pip install git+git://github.com/fhcrc/nestly.git@master

Python 2.7 is required.

Introductory example

Imagine you’d like to try all possible combinations of the following:

Option Choices
strategy approximate, exhaustive
run_count 10, 100, 1000
input file any file matching inputs/file*

For this we can write a little make_nest.py. The guts are:

nest = Nest()

nest.add('strategy', ('exhaustive', 'approximate'))
nest.add('run_count', [10**i for i in xrange(3)])
nest.add('input_file', glob.glob(os.path.join(input_dir, 'file*')),
        label_func=os.path.basename)

nest.build('runs')

Running make_nest.py, you get a directory tree like:

runs
├── approximate
│   ├── 10
│   │   ├── file1
│   │   │   └── control.json
│   │   ├── file2
│   │       └── control.json
│   ├── 100
│   │   ├── file1
│   │   │   └── control.json
│   │   ├── file2
│   │       └── control.json
│   └── 1000
│       ├── file1
│       │   └── control.json
│       ├── file2
│           └── control.json
└── exhaustive
    ├── 10
    │   ├── file1
    │   │   └── control.json
    │   ├── file2
    │       └── control.json
    ├── 100
    │   ├── file1
    │   │   └── control.json
    │   ├── file2
    │       └── control.json
    └── 1000
        ├── file1
        │   └── control.json
        ├── file2
            └── control.json

With the final control.json reading, for example:

{
    "input_file": "/Users/cmccoy/Development/nestly/examples/basic_nest/inputs/file3",
    "run_count": "1000",
    "strategy": "exhaustive"
}

The control files created then serve as inputs to nestrun for template substition, for example:

nestrun --save-cmd-file command.sh \
        --template='my_command -s {strategy} --count={run_count} {input_file}' \
        $(find runs -name "control.json")

This command runs my_command in all of the tip directories with the appropriate values for the parameters.

This was a “cartesian product” example. The “meal” example in the repository exhibits a setup with more complex dependencies between the nests.

Templates

nestrun takes a template and a list of control.json files with variables to substitute. By default, substitution is performed using the Python built-in str.format method. See the Python Formatter documentation for details on syntax, and examples/jsonrun/do_nestrun.sh for an example.

SCons integration

There is also a nestly.scons module to integrate nestly with the make replacement SCons.

Running the tests

Run:

python setup.py test

The mock library is required, but will be downloaded if missing.

Paper

McCoy CO, Gallagher A, Hoffman NG, Matsen FA (2013) nestly–a framework for running software with nested parameter choices and aggregating results. Bioinformatics 29: 387-388. pubmed

License

nestly source code is freely available under the MIT License.

Release History

Release History

0.6

This version

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.5

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.4

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.3

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

Download Files

Download Files

TODO: Brief introduction on what you do with files - including link to relevant help section.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
nestly-0.6-py2.py3-none-any.whl (18.5 kB) Copy SHA256 Checksum SHA256 2.7 Wheel Nov 26, 2014
nestly-0.6.tar.gz (15.9 kB) Copy SHA256 Checksum SHA256 Source Nov 26, 2014

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS HPE HPE Development Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting