NeEDS4BigDataPy

The python library “NeEDS4BigDataPy” provides approaches to implement subsampling methods to analyse big data and is the python version of NeEDS4BigData.

What is “NeEDS4BigData” an abbreviation for?

New Experimental Design based Subsampling methods for Big Data.

How to engage with NeEDS4BigDataPy for the first time?

# Installing from PyPI
pip install NeEDS4BigDataPy

# Importing the package
import NeEDS4BigDataPy

Subsampling Methods

1. A- and L-optimality based subsampling for GLMs.
2. A-optimality based subsampling for Gaussian Linear Models.
3. Leverage sampling for GLMs.
4. Local case control sampling for logistic regression.
5. A-optimality based subsampling under measurement constraints for GLMs.
6. Model robust subsampling method for GLMs.
7. Subsampling method for GLMs when the model is potentially misspecified.

These seven methods are described in the following articles under the topics

1. Introduction - explains the need for subsampling methods.
2. Model based subsampling
3. Model robust and misspecification
4. Benchmarking Functions

For $2)$ we assume the main effects model can describe the data. While for $3)$ first we consider there are several models that can describe the big data, then later we assume the given main effects model is misspecified. Under these conditions from $2)$ and $3)$ we explore subsampling for four given big data sets. Further, to explore the computation time we ran simulations for the scenarios $2)$ and $3)$ where we compare our subsampling functions against full data modelling in $4)$.

Thank You