Skip to main content

FaissImputer

Fast KNN imputation for incomplete numerical data, built for scikit-learn pipelines.

A nearby row is useful for imputation only if it contains the value you need to fill. Queries can observe different features, and each missing feature can require different donors. FaissImputer handles these constraints explicitly, including when the training data is itself incomplete.

Tests PyPI Python License

Why FaissImputer?

  • Choose your donor policy. Like sklearn's KNNImputer, donor_policy="available" can use partially observed training rows, selecting donors that observe each target feature and computing distances from co-observed features. FaissImputer additionally provides donor_policy="complete", which restricts donors to fully observed training rows — a policy with no equivalent KNNImputer option. The main difference is computational: FaissImputer uses missingness-pattern-grouped Faiss search for complete donors and optimized batched matrix distances for available donors, rather than sklearn's chunked all-pairs distance computation.
  • Search with Faiss. Native inner-product search and trainable/approximate Faiss index factories are capabilities that sklearn's KNNImputer does not expose.
  • Prioritize numerical reliability. FaissImputer includes targeted precision and overflow safeguards for distance search and aggregation, with regression tests covering numerical edge cases.
  • Keep familiar workflows. Use fit/transform, scikit-learn pipelines, feature names, missing indicators, and optional pandas output.

Performance

Separate-query benchmark — measured with version 0.3.20

A 2.7× first-transform speedup over scikit-learn KNNImputer on one available-donor workload.

This comparison measured the published FaissImputer 0.3.20 package. It used 20,000 training rows, 300 queries, 20 features, five neighbors, and uniform-weight mean aggregation. Training data had 10% MCAR missingness; each query had four missing features.

First-transform medians, excluding fit, on an AMD EPYC 7763 runner with one native thread:

Input dtype KNNImputer 1.9.1 FaissImputer 0.3.20 Speedup
float32 279.93 ms 103.40 ms 2.71×
float64 334.84 ms 122.48 ms 2.73×

Each method and dtype used three seeds and three fresh workers per seed, with a small untimed warmup. Fit took longer than KNNImputer in these cases; fit plus first transform was still 2.44–2.49× as fast.

The full run covered both donor policies and both dtypes. All benchmark runs passed output and input-preservation checks. Performance depends on workload, configuration, and hardware.

Benchmark run and artifacts · Reports, methodology, and historical results

Same-data OFAT benchmark — source commit 9683d03

A separate 12-run benchmark sweep measures imputation of the same incomplete data used for fitting. It covers row counts, feature counts, missing rates, MCAR/MAR missingness, and neighbor counts using a source build at commit 9683d03.

Results are reported separately for fit_transform and fit_then_transform, and for float32 and float64. Both API measurements include fitting. Timings are median [min–max], and speedups are medians of matched record-level KNNImputer/FaissImputer timing ratios.

FaissImputer's available-donor mode showed similar aggregate reconstruction RMSE and MAE to KNNImputer under the tested conditions. This does not establish identical predictions or algorithmic equivalence. Complete-donor results are reported alongside donor counts and reconstruction error to make the quality trade-off explicit.

The Intel k=30 observation and Intel 50,000-row stress result are presented separately from the AMD sweeps. The report also documents changes in guaranteed complete rows across neighbor settings and the stress run's single repeat per seed.

Full report and methodology · Raw benchmark results · Full-precision analysis summary

Held-out real-data benchmark — source commit ef04b1b

A development build at commit ef04b1b was evaluated on Wine Quality (white) and Abalone, using 1,000 and 3,000 training rows with 1,000 held-out queries, MCAR/MAR missingness, and float32/float64 inputs. Methods include mean and median baselines, KNNImputer, and both FaissImputer donor policies.

Timings include fit plus first transform and are reported as median [min–max] across three seeds and three repeats. Speedups are medians of nine matched record-level KNNImputer/FaissImputer timing ratios. Datasets and dtypes are reported separately.

Available mode was slower than KNNImputer on Wine Quality for both dtypes and on Abalone for float64; it was faster on Abalone for float32. Complete mode was faster on both datasets. Reconstruction error and donor counts are reported alongside timing.

RMSE and MAE measure reconstruction error on masked query entries. Quality summaries use three seed datasets after verifying consistency across repeats. Available-mode output differences from KNNImputer were observed, including with float64 inputs on Abalone; similar aggregate errors do not establish identical predictions.

Full report and methodology · Raw benchmark evidence · Full-precision analysis summary

Installation

Requires Python 3.10 or newer.

python -m pip install --upgrade faiss-imputer

Quick start

import numpy as np
from faiss_imputer import FaissImputer

train = np.array([[0, 10], [2, 20], [4, 40]], dtype=np.float32)
query = np.array([[1.8, np.nan]], dtype=np.float32)

imputer = FaissImputer(n_neighbors=1)
print(imputer.fit(train).transform(query))
# [[ 1.8 20. ]]

For partially observed training data, use FaissImputer(donor_policy="available").

More examples cover partial donors, indicators, numeric missing markers, custom metrics, and pandas pipelines.

When should I use it?

Use FaissImputer when distance-based imputation suits your numerical data and you need control over donor eligibility or reuse of fitted donors across query batches. Scale features appropriately before using distances.

Prefer KNNImputer when its behavior meets your needs and FaissImputer offers no measured advantage for your workload. FaissImputer is not a drop-in replacement: defaults, search precision, ties, and some edge cases differ.

Donor policies

  • "complete" — default: Use training rows observed in every non-empty feature. When non-empty features exist, at least n_neighbors complete donors are required.
  • "available": Use partially observed training rows and select donors separately for each missing feature. Fewer than n_neighbors usable donors are allowed.

In available-donor mode with built-in L2 metrics, a donor must observe the target feature and share an observed feature with the query. Squared L2 distances use those shared coordinates, scaled by the original feature count divided by the shared count. Eligibility therefore varies by target feature, unlike ordinary retrieval from a fixed set of complete vectors.

If no usable donor exists, the fitted column statistic supplies the value. Available mode requires index_factory="Flat".

Comparison with KNNImputer

Behavior FaissImputer KNNImputer
Donors Complete rows by default; feature-specific partial donors in available mode Feature-specific donors; other donor features may be missing
Built-in search Faiss-backed search for complete donors; missing-aware search for available donors Missing-aware pairwise distance computation
Default neighbors 3 5
Aggregation Mean or median; non-uniform weights with mean Mean with uniform, distance, or callable weights

Both support fit/transform workflows, pipelines, missing indicators, feature names, and pandas output through set_output.

For comparable configurations, use available donors and matching neighbor counts, weights, and missing markers. Matching settings does not guarantee identical output. In particular, built-in complete-donor search uses float32 vectors even when stored values and output are float64.

See the API reference for precision, copy behavior, callbacks, and edge cases.

Documentation

Contributing

Bug reports and focused pull requests are welcome. Please include a reproducible example when reporting unexpected behavior through the issue tracker.

Author: ScionKim.

License

MIT License.

Faiss is developed by Meta and distributed under the MIT License. FaissImputer is not affiliated with or endorsed by Meta or the Faiss maintainers.

Metadata

Release files for faiss-imputer 0.3.21

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for faiss-imputer 0.3.21
File Size Uploaded Provenance Actions
faiss_imputer-0.3.21.tar.gz 68.1 kB With provenance Details for faiss_imputer-0.3.21.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for faiss-imputer 0.3.21
File Interpreter ABI Platform Actions
faiss_imputer-0.3.21-py3-none-any.whl Python 3 none any Details for faiss_imputer-0.3.21-py3-none-any.whl

Total release size: 87.7 kB

Release files / faiss_imputer-0.3.21.tar.gz

Download URL faiss_imputer-0.3.21.tar.gz
Size 68.1 kB
Tags Source
SHA-256 checksum
How to use checksums
0fbe4287c047359a10565869533bcb9a4b37a257e5b534a91bccf88227a952e0
BLAKE2b-256 checksum
How to use checksums
ce197f9f5cd068c599dc29a1ceb9c32579a07284c639b2304445c884345d6a0b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 29, 2026.

Transparency log

Release files / faiss_imputer-0.3.21-py3-none-any.whl

Download URL faiss_imputer-0.3.21-py3-none-any.whl
Size 19.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
e8d3bd2d3375027568b8774d0cde5b68d37622471ae7c8efcf8a30ef10a66f58
BLAKE2b-256 checksum
How to use checksums
27debb127f427b0e7e96b326cf43d8314324d9450f211473d9689ed8822bf17e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 29, 2026.

Transparency log

Security for faiss-imputer 0.3.21

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for faiss_imputer-0.3.21.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 29, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.3.21 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page