Skip to main content
This is a pre-production deployment of Warehouse. Changes made here affect the production instance of PyPI (pypi.python.org).
Help us improve Python packaging - Donate today!

Numerical regression testing

Project Description

nrtest

nrtest is an end-to-end regression testing framework, designed for scientific software that perform numerical calculations.

Features

nrtest aims to simplify your workflow:

  • JSON files describe the software under test and the tests themselves
  • result files are stored in a portable benchmark directory
  • benchmarks are compared by iterating through tests and their results
  • custom comparisons can easily be added through extensions

Basic Usage

As an example usage, we consider testing TOPAS. This is a Monte Carlo tool for particle simulation, designed for medical physics research. Of course, such a tool must be rigorously validated against experimental data. But it is also useful to frequently run shorter tests, checking for regressions by comparing results to a previous version.

First, we describe the software under test in a configuration file called apps/topas-2.0.3.json. Note that setup_script defines the environment needed to run the software.

{
    "name": "topas",
    "version" : "2.0.3",
    "setup_script" : "/path/to/topas-2.0.3/setup.sh",
    "exe" : "topas"
}

We then describe the test in second configuration file called tests/Scoring_01.json. In doing so, we define the command-line arguments presented to the executable and the input files needed for the test to run. Finally, we also specify the expected output files, and declare how they should be compared to a benchmark. Here we use topas binned, which is a custom comparison routine, though some comparison routines are bundled with nrtest. It is also easy to add your own.

{
    "name": "Scoring_01",
    "version": "1.0",
    "description": "Basic test shooting a 6cm diameter proton beam into a water phantom.",
    "args": [
        "Scoring_01.txt"
    ],
    "input_files": [
        "Scoring_01.txt",
        "GlobalParameters.txt"
    ],
    "output_files": {
        "Dose.csv": "topas binned"
    }
}

To execute the test, we tell nrtest where to find the configuration files and where to output the benchmark. Note that nrtest will search tests/ for tests, though we could have specified tests/Scoring_01.json.

$ nrtest execute apps/topas-2.0.3.json tests/ -o benchmarks/2.0.3
INFO: Found 1 tests
Scoring_01: pass
INFO: Finished

To compare to a previous benchmark:

$ nrtest compare benchmarks/2.0.3 benchmarks/2.0.2
Scoring_01: pass
INFO: Finished

More advanced usage is detailed in the documentation.

History

0.2.0 (2016-02-21)

  • Add extensions for compare functions

0.1.0 (2016-02-21)

  • First release on PyPI.
Release History

Release History

This version
History Node

0.2.2

History Node

0.2.1

History Node

0.2.0

History Node

0.1.2

History Node

0.1.1

History Node

0.1.0

Download Files

Download Files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
nrtest-0.2.2-py2.py3-none-any.whl (15.9 kB) Copy SHA256 Checksum SHA256 py2.py3 Wheel Jul 15, 2016
nrtest-0.2.2.tar.gz (21.2 kB) Copy SHA256 Checksum SHA256 Source Jul 15, 2016

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting