Learned sample-based estimator for number of distinct values.
Project description
Learned NDV estimator
Learned model to estimate number of distinct values (NDV) of a population using a small sample. The model approximates the maximum likelihood estimation of NDV, which is difficult to obtain analytically. See our VLDB 2022 paper Learning to be a Statistician: Learned Estimator for Number of Distinct Values for more details.
How to use
-
Install the package
pip install estndv
-
Import and create an instance
from estndv import ndvEstimator
estimator = ndvEstimator()
-
Assume your sample is S=[1,1,1,3,5,5,12] and the population size is N=100000. You can estimate population ndv by:
ndv = estimator.sample_predict(S=[1,1,1,3,5,5,12], N=100000)
-
If you have the sample profile e.g. f=[2,1,1], you can estimate population NDV by:
ndv = estimator.profile_predict(f=[2,1,1], N=100000)
-
If you have multiple samples/profiles from multiple populations, you can estimate population NDV for all of them in a batch by method
estimator.sample_predict_batch()
orestimator.profile_predict_batch()
.
How to train the ndv estimator
You can directly use our packaged library for your datasets, as the pre-trained model in the packed library is agnostic to any workloads. However, if you want to train the model from scratch anyway, do the following:
-
Go to the model_training folder
cd model_training
-
Install requirements
pip install requirements.txt
-
Generate training data. (This uses a lot of memory.)
python training_data_generation.py
-
Train model
python model_training.py
-
Save trained pytorch model parameters to numpy, this generates a file model_paras.npy
python torch2npy.py
-
Test with your model parameters by specifying a path to your model_paras.npy
estimator = ndvEstimator(para_path=your path to model_paras.npy)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.