GOFevaluation

Evaluate the Goodness-of-Fit (GOF) for binned or unbinned data.

This GOF suite comprises the possibility to calculate different 1D / nD, binned / two-sample (unbinned) GOF measures and the corresponding approximate p-value. A list of implemented measures is given below.

Implemented GOF measures

Installation and Set-Up

Regular installation:

pip install GOFevaluation

Developer setup:

Clone the repository:

git clone https://github.com/XENONnT/GOFevaluation
cd GOFevaluation

Install the requirements in your environment:

pip install -r requirements.txt

Then install the package:

python setup.py install --user

You are now good to go!

Usage

The best way to start with the GOFevaluation package is to have a look at the tutorial notebook. If you click on the mybinder badge, you can execute the interactive notebook and give it a try yourself without the need of a local installation.

Individual GOF Measures

Depending on your data and reference input you can initialise a gof_object in one of the following ways:

import GOFevaluation as ge

# Data Sample + Binned PDF
gof_object = ge.BinnedPoissonChi2GOF(data_sample, pdf, bin_edges, nevents_expected)

# Binned Data + Binned PDF
gof_object = ge.BinnedPoissonChi2GOF.from_binned(binned_data, binned_reference)

# Data Sample + Reference Sample
gof_object = ge.PointToPointGOF(data_sample, reference_sample)

With any gof_object you can calculate the GOF and the corresponding p-value as follows:

gof = gof_object.get_gof()
p_value = gof_object.get_pvalue()

Multiple GOF Measures at once

You can compute GOF and p-values for multiple measures at once with the GOFTest class.

Example:

import GOFevaluation as ge
import scipy.stats as sps

# random_state makes sure the gof values are reproducible.
# For the p-values, a slight variation is expected due to
# the random re-sampling method that is used.
data_sample = sps.uniform.rvs(size=100, random_state=200)
reference_sample = sps.uniform.rvs(size=300, random_state=201)

# Initialise all two-sample GOF measures:
gof_object = ge.GOFTest(data_sample=data_sample,
                        reference_sample=reference_sample,
                        gof_list=['ADTestTwoSampleGOF',
                                  'KSTestTwoSampleGOF',
                                  'PointToPointGOF'])
# Calculate GOFs and p-values:
d_min = 0.01
gof_object.get_gofs(d_min=d_min)
# OUTPUT:
# OrderedDict([('ADTestTwoSampleGOF', 1.6301454042304904),
#              ('KSTestTwoSampleGOF', 0.14),
#              ('PointToPointGOF', -0.7324060759792504)])

gof_object.get_pvalues(d_min=d_min)
# OUTPUT:
# OrderedDict([('ADTestTwoSampleGOF', 0.08699999999999997),
#              ('KSTestTwoSampleGOF', 0.10699999999999998),
#              ('PointToPointGOF', 0.31200000000000006)])

# Re-calculate p-value only for one measure:
gof_object.get_pvalues(d_min=.001, gof_list=['PointToPointGOF'])
# OUTPUT:
# OrderedDict([('ADTestTwoSampleGOF', 0.08699999999999997),
#              ('KSTestTwoSampleGOF', 0.10699999999999998),
#              ('PointToPointGOF', 0.128)])

print(gof_object)
# OUTPUT:
# GOFevaluation.gof_test
# GOF measures: ADTestTwoSampleGOF, KSTestTwoSampleGOF, PointToPointGOF


# ADTestTwoSampleGOF
# gof = 1.6301454042304904
# p-value = 0.08499999999999996

# KSTestTwoSampleGOF
# gof = 0.13999999999999996
# p-value = 0.09799999999999998

# PointToPointGOF
# gof = -0.7324060759792504
# p-value = 0.128

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

Please make sure to update tests as appropriate.

v0.1.6

• Imitate equal probability binning for discretized values by @dachengx in https://github.com/XENONnT/GOFevaluation/pull/58
• Equal probability binning on integer sample by @dachengx in https://github.com/XENONnT/GOFevaluation/pull/59

v0.1.5

• Allow all kind of floats for binned inputs by @jjakob03 in https://github.com/XENONnT/GOFevaluation/pull/55
• Open right side of the bin to follow numpy convention by @dachengx in https://github.com/XENONnT/GOFevaluation/pull/56

v0.1.4

• Fix plotting of approximately equiprobable bins by @hammannr in https://github.com/XENONnT/GOFevaluation/pull/51
• Use pre-commit CI by @hammannr in https://github.com/XENONnT/GOFevaluation/pull/52

v0.1.3

• Should throw error when count_density mode with x or y limit specified by @dachengx in #46
• Fix location of sklearn's DistanceMetric by @dachengx in #48

v0.1.2

• Add colorbar switch, set 2D histogram x&y limit by @dachengx in #39
• Some plotting bug fixes by @hoetzsch in #41
• Homemade equiprobable_binning, still based on ECDF by @dachengx in #43
• a few patches by @hammannr in #38
• Exercise notebook by @hammannr in #44

v0.1.1

• Add an example notebook that can be used as a guide when using the package for the first time (#29)
• Improve and extend plotting of equiprobable binnings. This adds the option of plotting the binwise count density (#35)

v0.1.0

• Multiple GOF tests (binned and unbinned) can be performed (#1, #5, #10, #12, #13)
• The p-value is calculated based on toy sampling from the reference or a permutation test (#2, #14)
• A wrapper class makes it convenient to perform multiple GOF tests in parallel (#19)
• An equiprobable binning algorithm is implemented. The binning can be applied upon initialisation of the GOF object and a few visualization tools are provided. (#25, #26)
• CI workflow implemented (#7)