Skip to main content

A SciUnit library for validation testing of neural network models.

Project description

A SciUnit library for validation testing of spiking networks.

Binder Link

Installation

pip install networkunit

Concept

The NetworkUnit module builds upon the formalized validation scheme of the SciUnit package, which enables the validation of models against experimental data (or other models) via tests. A test is matched to the model by capabilities and quantitatively evaluated by a score. The following figure illustrates a typical test design within NetworkUnit. The blue boxes indicate the components of the implementation of the validation test, i.e., classes, class instances, data sets, and parameters. The relation between the boxes are indicated by annotated arrows.The basic functionality is shown by green arrows. The difference in the test design for comparing against experimental data (validation) and another simulation (substantiation) is indicated by yellow and red arrows, respectively. The relevant functionality of some components for the computation of test score is indicated by pseudo-code. The capability class ProducesProperty contains the function calc_property(). The test XYTest has a function generate_prediction() which makes use of this capability, inherited by the model class, to generate a model prediction. The initialized test instance XYTest_paramZ makes use of its judge() function to evaluate this model prediction and compute the score TestScore. The XYTest can inherit from multiple abstract test classes (BaseTest), which is for example used with the M2MTest to add the functionality of evaluating multiple model classes. To make the test executable it has to be linked to a ScoreType and all free parameters need to be set (by a Params dict) to ensure a reproducible result.

NetworkUnit Flowchart

Showcase examples on how to use NetworkUnit can be found in this repository and interactive reveal.js slides are accessible via the launch-binder button at the top.

Overview of tests

Class name

Parent class

Prediction measure

two_sample_test

-

-

correlation_test

two_sample_test

-

correlation_dist_test

correlation_test

correlation coefficients

correlation_matrix_test

correlation_test

correlation coefficient matrix

generalized_correlation_matrix_test

correlation_matrix_test

matrix of derived cross-correlation measures

eigenvalue_test

correlation_test

eigenvalues of the correlation coefficient matrix

covariance_test

two_sample_test

covariances

firing_rate_test

two_sample_test

firing rates

isi_variation_test

two_sample_test

inter-spike-intervals, their CV, or LV

graph_centrality_helperclass

sciunit.Test

graph centrality measures of given adjacency matrix

Inheritance order in case of multiple inheritance for derived test classes:

class new_test(sciunit.TestM2M, graph_centrality_helperclass, <base_test_class>)

Overview of scores

Class name

Test name

Comparison measure

students_t

Student’t test

sample mean

ks_distance

Kolmogorov-Smirnov test

sample distribution

kl_divergence

Kullback-Leibler divergence

sample entropy

mwu_statistic

Mann-Whitney U test

rank sum

levene_score

Levene’s test

sample variance

effect_size

Effect size

standardized mean

best_effect_size

Bayesian estimation effect size

standardized mean

Overview of model classes

Model name

Capability

Parent class

Purpose

loaded_data

-

sciunit.Model

loading simulated data

spiketrain_data

ProducesSpikeTrains

simulation_data

loading simulated spiking data

stochastic_activity

ProducesSpikeTrains

sciunit.Model

generating stochastic spiking data

Other validation test repositories

Acknowledgments

This open source software code was developed in part or in whole in the Human Brain Project, funded from the European Union’s Horizon 2020 Framework Programme for Research and Innovation under Specific Grant Agreements No. 720270 and No. 785907 (Human Brain Project SGA1 and SGA2).

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

networkunit-0.1.1.tar.gz (2.5 MB view hashes)

Uploaded Source

Built Distribution

networkunit-0.1.1-py3-none-any.whl (2.6 MB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page