met 1.0.2

Multinomial Exact Tests

met.py Multinomial Exact Tests

met.py is a Python module that allows you to define a pair of multinomial distributions (conceptually ‘control’ and ‘test’ distributions) and then compute one- and two-sided p values to test whether the ‘test’ distribution is equivalent to the ‘control’ distribution. The likelihood of all possible ‘control’ distributions can be evaluated and the distribution of p values can be expressed in terms of the likelihood of the observed ‘control’ distribution.

Installation

You can install the module from [PyPi](https://pypi.org/) with the command

pip install met

or you can install in manually by downloading met.py and copying the module to wherever you want to use it. If you place it in a location other than the standard library for your Python distribution, you must add that location to your Python path prior to importing the module.

Purpose

Perform exact tests of a (site or test) distribution of multinomial count data against a distribution of equivalent ordered multinomial count data from another (reference or control) data set. Both two-sided and one-sided tests can be performed. One-sided tests require that categories be ordered.

A practical example of relevant data (and the motivation for writing this module) arises from the use of Sediment Profile Imaging to evaluate the benthic macroinvertebrate community at site and reference locations, and characterization of each sample in terms of the benthic successional stage presen. These categorical data are ordered, and so a one-sided exact multinomial test can be applied.

Notes

Exact Tests

Whereas most statistical tests compute p values based on the parameters of a continuous distribution, and those p value are therefore estimates, calculation of exact p values is possible when every possible rearrangement of the data can be enumerated. Each rearrangement of the data must have a specific probability of occurrence, which is computed based on a theoretical or reference distribution of probabilities. Summing of the probabilities of occurrence for the observed data and for all more extreme arrangments of the data produces an exact p value–hence, the result of an exact test.

For two-sided tests, “more extreme” means that the probability of an alternative arrangement of site data has a lower probability than the observed arrangement.

For one-sided tests, “more extreme” means a data arrangement in which one or more observations is shifted from a ‘better’ category to a ‘worse’ one. Therefore, to carry out one-sided exact tests, the categories must be ordered from ‘better’ to ‘worse’.

Issues

When reference area sample sizes are small relative to site sample sizes, the accuracy of the probabilities calculated from the reference area samples may be questionable. In particular, some categories that are represented in the site data may not be represented in the reference data, and so have zero reference aera probabilities. Whenever a reference area probability for any category is zero, the probability of any arrangement of site data with a non-zero count in that category is also zero. Small reference area sample sizes may therefore lead to underestimation of true reference probabilities for some categories, and consequently to an underestimation of p values in the exact tests.

The met module contains two features to help evaluate and account for uncertainties related to relatively small reference area sample sizes. The first of these allows assignment of small non-zero probabilities to categories with no reference area observations, and the second allows evaluation of the likelihood of different p values based on possible reference area probabilities that could have led to the observed reference data.

Example

This example illustrates the computation of one-sided and two-sided p values, with and without zeroes in the reference distribution. import met

# Counts observed in the reference area, ordered from ‘best’ to ‘worst’ # category. For SPI data, the categories are Stage 3, Stage 2 on 3, # Stage 1 on 3, Stage 2, and Stage 1.

ref = [9, 6, 6, 0, 5]

# Counts observed in the site. # These are skewed toward an impacted distribution. # Counts are low for speed of illustration. site = [10, 12, 14, 5, 6]

# MET with zeroes in the reference distribution.

t1 = met.Multinom(ref, site)

t1_p1 = t1.onesided_exact_test(save_cases=True) print(“Example 1 with no zero-fill; one-sided p = %s” % t1_p1) print(” Number of extreme cases = %s” % t1.n_extreme_cases)

The results are shown as:

Example 1 with no zero-fill; one-sided p = 0.00634738889819

Number of extreme cases = 63348

Although there are zero probabilities in the reference area distribution, a non-zero one-sided probability is calculated because some of the more extreme distributions (i.e., skewed further from the reference distribution than the actual size) have zeroes in the same category.

Calculation of a two-sided p value:

t1_p2 = t1.twosided_exact_test(save_cases=True) print(“Example 1 with no zero-fill; two-sided p = %s” % t1_p2) print(” Number of cases = %s” % len(t1.cases)) print(” Number of extreme_cases = %s” % t1.n_extreme_cases)

The result is:

Example 1 with no zero-fill; two-sided p = 0.0

Number of cases = 249900 Number of extreme_cases = 0

The zero probabilities in the reference distribution prevent the calculation of a two-sided p value. The multinomial probability for the site itself is zero, and the two-sided test sums that with the multinomial probabilities of all of the (249,900) other possible results for the site that are less probable than the observed site data.

Specifying a ‘zero-fill’ value results in a different p value for both the one-sided and two-sided tests.

# MET with zeros replaced with 1 and non-zeroes inflated by 10.

t2 = met.Multinom(met.fillzeroes(ref, 10), site)

t2_p1 = t2.onesided_exact_test() t2_p2 = t2.twosided_exact_test()

print(“Example 2 with a zero-fill factor of 10; one-sided p = %s, two-sided p = %s” % (t2_p1, t2_p2))

The result is:

Example 2 with a zero-fill factor of 10; one-sided p = 0.0070346, two-sided p = 1.95908e-06

A larger fill factor results in different calculated probabilities. # MET with zeros replaced with 1 and non-zeroes inflated by 100.

t3 = met.Multinom(met.fillzeroes(ref, 100), site)

t3_p1 = t3.onesided_exact_test() t3_p2 = t3.twosided_exact_test()

print(“Example 3 with a zero-fill factor of 100; one-sided p = %s, two-sided p = %s” % (t3_p1, t3_p2))

The result is:

Example 3 with a zero-fill factor of 100; one-sided p = 0.0064139, two-sided p = 1.73424e-11

Copyright and License

Copyright (c) 2009, 2019 R.Dreas Nielsen

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. The GNU General Public License is available at http://www.gnu.org/licenses/.