This is a pre-production deployment of Warehouse, however changes made here WILL affect the production instance of PyPI.
Latest Version Dependencies status unknown Test status unknown Test coverage unknown
Project Description

[![Build Status](https://travis-ci.org/posterior/goftests.svg?branch=master)](https://travis-ci.org/posterior/goftests) [![Code Quality](http://img.shields.io/scrutinizer/g/posterior/goftests.svg)](https://scrutinizer-ci.com/g/posterior/goftests/code-structure/master/hot-spots) [![Latest Version](https://badge.fury.io/py/goftests.svg)](https://pypi.python.org/pypi/goftests)

# Goftests

Generic goodness of fit tests for random plain old data.

Goftests is intended for unit testing random samplers that generate arbitrary plain-old-data, and focuses on robustness rather than statistical efficiency. In contrast to [scipy.stats][] and [statsmodels][], goftests does not make assumptions on the distribution being tested, and requires only a simple (sample, prob) interface provided by MCMC samplers.

## Installing

pip install goftests

## Using goodness of fit tests

Goftests implements generic statistical tests for Monte Carlo samplers that generate (sample, probability) pairs. The four basic generic tests are

  • discrete_goodness_of_fit(samples, probs_dict, …) - for discrete-valued data with most of the mass on a few values.
  • density_goodness_of_fit(samples, probs, …) - for continuously distributed univariate data with no discrete component.
  • vector_density_goodness_of_fit(samples, probs, …) - for continuously distributed multivariate data with no discrete component.
  • mixed_density_goodness_of_fit(samples, probs, …) - for discretely-indexed continuously distributed data, subsuming the all the other tests.

Each test outputs a goodness of fit number, say gof, which should be uniformly distributed in the interval [0,1], and which tends to fail with numbers close to zero. Thus to test a sampler we usually write

TEST_FAILURE_RATE = 1e-3

def test_my_sampler(count=100):
seed_all(0) samples = [my_sampler.rvs() for _ in xrange(count)] probs = [my_sampler.pdf(x) for x in samples] gof = mixed_density_goodness_of_fit(samples, probs) assert gof > TEST_FAILURE_RATE

By specifying a global TEST_FAILURE_RATE for a suite of tests, we can tune the number of tests expected to fail. In a large suite of tests, some may fail at the specified seeds despite the distribution being correct, typically with gof just below the threshold. In this case we usually increment the seed on those tests. In a suite of 1000 tests with TEST_FAILURE_RATE = 1e-3, we should expect to have to increment the seed on 1 test, on average.

### Diagnosing common errors

#### Too few samples

The discrete tests only look at the most-likely few values, and requires enough samples in the top few values to be able to use Pearson’s &chi;<sup>2</sup> test. This is typically at least 100 samples.

The univariate tests require around 100 samples, and the multivariate require at least 1000 per dimension.

#### Too many samples

When testing for millions of samples or more, tests can fail due to numeric imprecision rather than poor distribution. We have had success testing with 10<sup>2</sup>-10<sup>5</sup> samples.

#### Poorly mixing Markov chains

When testing samples generated by MCMC, the samples are often correlated so that testing the entire chain fails. In this case, try running the chain for longer and subsampling only every k<sup>th</sup> step in the chain.

#### Sticky Markov chains

The density tests are sensitive to Sticky Markov chains. As with poorly mixing chains, try running for longer.

#### Spuriously failing tests

Be sure to set a deterministic seed via seed_all(…) before generating each dataset. In particular, some test runners run tests in a nondeterministic order, so setting the seed before generating each sample ensures deterministic behavior.

## Implementing new tests

The goodness of fit tests are mostly implemented by reduction to other tests, eventually reducing to the multinomial goodness of fit test which uses Pearson’s &chi;<sup>2</sup> test on each of the multinomial’s bins.

![Reductions](/doc/reductions.png)

To implement a new test, you can implement from scratch, reduce to another test in goftests, or reduce to standard tests in another package like [scipy.stats][] or [statsmodels][].

[scipy.stats]: http://docs.scipy.org/doc/scipy/reference/stats.html#statistical-functions [statsmodels]: http://statsmodels.sourceforge.net/stable/stats.html#goodness-of-fit-tests-and-measures

## References

## License

Copyright (c) 2014 Salesforce.com, Inc. All rights reserved. <br/> Copyright (c) 2015 Gamelan Labs, Inc. <br/> Copyright (c) 2016 Google, Inc. <br/> Licensed under the Revised BSD License. See [LICENSE.txt](LICENSE.txt) for details.

Release History

Release History

0.2.3

This version

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.2.2

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.2.1

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.2.0

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.1.8

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.1.7

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.1.6

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.1.5

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.1.3

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.1.0

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

Download Files

Download Files

TODO: Brief introduction on what you do with files - including link to relevant help section.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
goftests-0.2.3.tar.gz (8.9 kB) Copy SHA256 Checksum SHA256 Source Jun 26, 2016

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS HPE HPE Development Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting