Skip to main content

Matrix completion and feature imputation algorithms

Project description

Build Status Coverage Status DOI

plot

A variety of matrix completion and imputation algorithms implemented in Python 3.6.

To install:

pip install fancyimpute

If you run into tensorflow problems and use anaconda, you can try to fix them with conda install cudatoolkit.

Important Caveats

(1) This project is in "bare maintenance" mode. That means we are not planning on adding more imputation algorithms or features (but might if we get inspired). Please do report bugs, and we'll try to fix them. Also, we are happy to take pull requests for more algorithms and/or features.

(2) IterativeImputer started its life as a fancyimpute original, but was then merged into scikit-learn and we deleted it from fancyimpute in favor of the better-tested sklearn version. As a convenience, you can still from fancyimpute import IterativeImputer, but under the hood it's just doing from sklearn.impute import IterativeImputer. That means if you update scikit-learn in the future, you may also change the behavior of IterativeImputer.

Usage

from fancyimpute import KNN, NuclearNormMinimization, SoftImpute, BiScaler

# X is the complete data matrix
# X_incomplete has the same values as X except a subset have been replace with NaN

# Use 3 nearest rows which have a feature to fill in each row's missing features
X_filled_knn = KNN(k=3).fit_transform(X_incomplete)

# matrix completion using convex optimization to find low-rank solution
# that still matches observed values. Slow!
X_filled_nnm = NuclearNormMinimization().fit_transform(X_incomplete)

# Instead of solving the nuclear norm objective directly, instead
# induce sparsity using singular value thresholding
X_incomplete_normalized = BiScaler().fit_transform(X_incomplete)
X_filled_softimpute = SoftImpute().fit_transform(X_incomplete_normalized)

# print mean squared error for the  imputation methods above
nnm_mse = ((X_filled_nnm[missing_mask] - X[missing_mask]) ** 2).mean()
print("Nuclear norm minimization MSE: %f" % nnm_mse)

softImpute_mse = ((X_filled_softimpute[missing_mask] - X[missing_mask]) ** 2).mean()
print("SoftImpute MSE: %f" % softImpute_mse)

knn_mse = ((X_filled_knn[missing_mask] - X[missing_mask]) ** 2).mean()
print("knnImpute MSE: %f" % knn_mse)

Algorithms

  • SimpleFill: Replaces missing entries with the mean or median of each column.

  • KNN: Nearest neighbor imputations which weights samples using the mean squared difference on features for which two rows both have observed data.

  • SoftImpute: Matrix completion by iterative soft thresholding of SVD decompositions. Inspired by the softImpute package for R, which is based on Spectral Regularization Algorithms for Learning Large Incomplete Matrices by Mazumder et. al.

  • IterativeImputer: A strategy for imputing missing values by modeling each feature with missing values as a function of other features in a round-robin fashion. A stub that links to scikit-learn's IterativeImputer.

  • IterativeSVD: Matrix completion by iterative low-rank SVD decomposition. Should be similar to SVDimpute from Missing value estimation methods for DNA microarrays by Troyanskaya et. al.

  • MatrixFactorization: Direct factorization of the incomplete matrix into low-rank U and V, with per-row and per-column biases, as well as a global bias. Solved by SGD in pure numpy.

  • NuclearNormMinimization: Simple implementation of Exact Matrix Completion via Convex Optimization by Emmanuel Candes and Benjamin Recht using cvxpy. Too slow for large matrices.

  • BiScaler: Iterative estimation of row/column means and standard deviations to get doubly normalized matrix. Not guaranteed to converge but works well in practice. Taken from Matrix Completion and Low-Rank SVD via Fast Alternating Least Squares.

Citation

If you use fancyimpute in your academic publication, please cite it as follows:

@software{fancyimpute,
  author = {Alex Rubinsteyn and Sergey Feldman},
  title={fancyimpute: An Imputation Library for Python},
  url = {https://github.com/iskandr/fancyimpute},
  version = {0.7.0},
  date = {2016},
}

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fancyimpute-0.7.0.tar.gz (26.0 kB view details)

Uploaded Source

Built Distribution

fancyimpute-0.7.0-py3.7.egg (51.4 kB view details)

Uploaded Egg

File details

Details for the file fancyimpute-0.7.0.tar.gz.

File metadata

  • Download URL: fancyimpute-0.7.0.tar.gz
  • Upload date:
  • Size: 26.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.62.2 CPython/3.7.4

File hashes

Hashes for fancyimpute-0.7.0.tar.gz
Algorithm Hash digest
SHA256 76a6919e67ed3ef3364328a6636c86325ca2c9073c40c79df455db12b4285fc9
MD5 f0040aed47d961f1e8421b7f45a0ace2
BLAKE2b-256 2b133c0bf07ebf3e7b232a02867d88a1b322095c89d98dfe4a10ba1cace6022b

See more details on using hashes here.

File details

Details for the file fancyimpute-0.7.0-py3.7.egg.

File metadata

  • Download URL: fancyimpute-0.7.0-py3.7.egg
  • Upload date:
  • Size: 51.4 kB
  • Tags: Egg
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.62.2 CPython/3.7.4

File hashes

Hashes for fancyimpute-0.7.0-py3.7.egg
Algorithm Hash digest
SHA256 783f7b8d2ad7768dbf028168fcabc693b705fe4513ebfaedf58eac5fa129f4e0
MD5 f0612690286d5e894ea0e4d3bac18854
BLAKE2b-256 0c2cfc79a13daec46d66385da84f8ed89aeb6595d252206f4f8d836d73d3bf6e

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page