Learned sample-based estimator for number of distinct values.
Project description
Learned NDV estimator
Learned model to estimate number of distinct values (NDV) of a population using a small sample. The model approximates the maximum likelihood estimation of NDV, which is difficult to obtain analytically. See our VLDB 2022 paper Learning to be a Statistician: Learned Estimator for Number of Distinct Values for more details.
How to use
-
Create an instance
estimator = Estimator()
-
Assume your sample is S=[1,1,1,3,5,5,12] and the population size is N=100000. You can estimate population ndv by:
ndv = estimator.sample_predict(S=[1,1,1,3,5,5,12], N=100000)
-
If you have the sample profile e.g. f=[2,1,1], you can estimate population NDV by:
ndv = estimator.profile_predict(f=[2,1,1], N=100000)
-
If you have multiple samples/profiles from multiple populations, you can estimate population NDV for all of them in a batch by method
estimator.sample_predict_batch()
orestimator.profile_predict_batch()
.
How to train the ndv estimator
You can directly use our trained model for your datasets, as the trained model is agnostic to workloads. If you want to train the model from scratch anyway, do the following:
-
Go to the model_training folder
cd model_training
-
Install requirements
pip install requirements.txt
-
Generate training data. (This uses a lot of memory.)
python training_data_generation.py
-
Train model
python model_training.py
-
Save trained pytorch model parameters to numpy
python torch2npy.py
-
Use the estimate.py script to test your model. Make sure to change
para_path
.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.