ConsistencyTEST 0.0.2

Consistency test

Description

ConTEST is a statistical test for assessing the consistency between observations and astrophysical models. It uses a combination of non-parametric methods and distance measures to obtain a test statistic that evaluates the closeness of the astrophysical model to the observations; hypothesis testing is then performed using bootstrap.

Table of Contents

• Step-by-step setup
• Tutorial

Step-by-step setup

Follow the instructions below to install and start using ConTEST in Python.

1. Install ConTEST:

   pip install ConsistencyTEST
   

   or git clone the repository:

   git clone https://github.com/FiorenSt/ConTEST.git
   

2. (Optional) Install the statistical software R. R is required only if you plan to use the smoothed_contest_reg() function. This function employs local linear regression using the np package in R.

3. (Optional) To ensure that Python can access R's libraries, run the three lines below in Python (of course, modify to match your folders). This step is only necessary if you intend to use the smoothed_contest_reg() function:

    import os
    os.environ['R_HOME'] = '~/Program Files/R/R-4.0.2'  #-> Your installed R folder
    os.environ['R_USER'] = '~/Miniconda3/envs/ConsistencyTest/lib/site-packages/'  #-> Your python environment
    os.environ['R_LIBS_USER'] = "~/Program Files/R/R-4.0.2/library/"  #-> Your R packages library
   

4. Install Python dependencies. Note that the rpy2 package, which facilitates interaction between R and Python, is required only if you plan to use the smoothed_contest_reg() function:

   pip intall matplotlib
   pip intall numpy
   pip intall pandas
   pip intall scipy
   pip intall seaborn
   pip intall rpy2
   

5. (Optional) If this is the first time you use ConTEST and you plan to use the smoothed_contest_reg() function, you need to install the R package used in Smoothed ConTEST. In Python, simply run:

    def install_R_functions(packnames=('np')):
        # import R's utility package
        utils = rpackages.importr('utils')
        # select a mirror for R packages
        utils.chooseCRANmirror(ind=1)  # select the first mirror in the list
        # R package install
        utils.install_packages(packnames)
    
    install_R_functions()
   

6. Use ConTEST in Python! Follow the tutorial below for more information about the individual functions.

Dependencies:

The following combination of package versions works on most Linux and Windows computers, however other package versions may also work. If a problem with the combination of packages occurs, raise an issue, and we will help you solve it.

Python 3 (or superior)

• Numpy 1.21.6
• Pandas 1.4.2
• Scipy 1.7.1
• Matplotlib 3.3.4
• Seaborn 0.11.2
• Rpy2 3.5.2 (For R and Python interaction)

R 3.6.0 (or superior)

• Np 0.60

Tutorial

ConTEST can be applied in different case scenarios depending on the nature of the model being tested.
For more details check out the paper: Stoppa et al., in preparation

There are 4 fundamental functions in ConTEST:

• ConTEST for regression: Test the consistency of a model with respect to an observed dataset and their uncertainties
• Smoothed ConTEST for regression (requires R and the rpy2 Python package): Test the consistency of a model with respect to an observed dataset and their uncertainties
• ConTEST for outliers: Test if an observed sample is likely to come from a density model (or a simulated dataset)
• ConTEST for densities: Test the consistency of a density model (or a simulated dataset) with respect to an observed dataset

Intro script

 # ensure that Python can access R
 import os
 os.environ['R_HOME'] = '~/Program Files/R/R-4.0.2'  #-> Your installed R folder
 os.environ['R_USER'] = '~/Miniconda3/envs/ConsistencyTest/lib/site-packages/'  #-> Your python environment
 os.environ['R_LIBS_USER'] = "~/Program Files/R/R-4.0.2/library/"  #-> Your R packages library
 
 # load contest functions
 from ConTEST.CONTEST import contest_reg, smoothed_contest_reg, contest_outliers, contest_dens

Regression models

Create synthetic model, observations, and uncertainties to test the functions:

# random sample 
n=100
x = np.random.rand(n)

# synthetic model 
beta1 = -0.3
beta2 = 8
m = 2
model = np.exp(beta1*x)*np.sin(beta2*x) + m

# error function (Not known in real scenarios)
err_model = model * .05

# sample observations from the model with the correct uncertainties
obs = np.zeros(N)
   for i in range(N):
  obs[i] = model[i] + stats.multivariate_normal.rvs(mean=0, cov=(err_model[i])**2,size=1)

# assign correct uncertainties to the observations
err_obs = err_model

ConTEST for regression

Test1 = contest_reg(y_obs = obs, x_obs = x, y_mod = model, y_obs_err = err_obs, K=1000,plot=True)

Smoothed ConTEST for regression (requires R and the rpy2 Python package)

Test2 = smoothed_contest_reg(y_obs = obs, x_obs = x, y_mod = model, y_obs_err = err_obs, K=1000,plot=True)   

Density models

Create synthetic model and observations:

n=100

#1D example
obs = stats.multivariate_normal.rvs(mean=5, cov= [1.5],size=n)
model = stats.multivariate_normal.rvs(mean=5, cov= [1.5],size=1000)
 
#2D example
obs_2d =  stats.multivariate_normal.rvs(mean=[5,5], cov= [[1.5,.8],[.8,2.5]],size=n)
model_2d = stats.multivariate_normal.rvs(mean=[5,5], cov= [[1.5,.8],[.8,2.5]],size=1000)

### ConTEST for outliers 

```sh
Test3 = contest_outliers(mod=model, obs=obs, K=10000, plot=True)
Test3 = contest_outliers(mod=model_2d, obs=obs_2d, K=10000, plot=True)

ConTEST for densities

Test4 = contest_dens(mod=model, obs=obs, K=10000, plot=True)   
Test4 = contest_dens(mod=model_2d, obs=obs_2d, K=10000, plot=True)