FaissImputer
Fast KNN imputation for incomplete numerical data, built for scikit-learn pipelines.
A nearby row is useful for imputation only if it contains the value you need to fill. Queries can observe different features, and each missing feature can require different donors. FaissImputer handles these constraints explicitly, including when the training data is itself incomplete.
Why FaissImputer?
- Choose your donor policy. Like sklearn's
KNNImputer,donor_policy="available"can use partially observed training rows, selecting donors that observe each target feature and computing distances from co-observed features. FaissImputer additionally providesdonor_policy="complete", which restricts donors to fully observed training rows — a policy with no equivalentKNNImputeroption. The main difference is computational: FaissImputer uses missingness-pattern-grouped Faiss search for complete donors and optimized batched matrix distances for available donors, rather than sklearn's chunked all-pairs distance computation. - Search with Faiss. Native inner-product search and
trainable/approximate Faiss index factories are capabilities that
sklearn's
KNNImputerdoes not expose. - Prioritize numerical reliability. FaissImputer includes targeted precision and overflow safeguards for distance search and aggregation, with regression tests covering numerical edge cases.
- Keep familiar workflows. Use fit/transform, scikit-learn pipelines, feature names, missing indicators, and optional pandas output.
Performance
Separate-query benchmark — measured with version 0.3.20
A 2.7× first-transform speedup over scikit-learn KNNImputer on one available-donor workload.
This comparison measured the published FaissImputer 0.3.20 package. It used 20,000 training rows, 300 queries, 20 features, five neighbors, and uniform-weight mean aggregation. Training data had 10% MCAR missingness; each query had four missing features.
First-transform medians, excluding fit, on an AMD EPYC 7763 runner with one native thread:
| Input dtype | KNNImputer 1.9.1 | FaissImputer 0.3.20 | Speedup |
|---|---|---|---|
| float32 | 279.93 ms | 103.40 ms | 2.71× |
| float64 | 334.84 ms | 122.48 ms | 2.73× |
Each method and dtype used three seeds and three fresh workers per seed, with a small untimed warmup. Fit took longer than KNNImputer in these cases; fit plus first transform was still 2.44–2.49× as fast.
The full run covered both donor policies and both dtypes. All benchmark runs passed output and input-preservation checks. Performance depends on workload, configuration, and hardware.
Benchmark run and artifacts · Reports, methodology, and historical results
Same-data OFAT benchmark — source commit 9683d03
A separate 12-run benchmark sweep measures imputation of the same
incomplete data used for fitting. It covers row counts, feature counts,
missing rates, MCAR/MAR missingness, and neighbor counts using a source
build at commit 9683d03.
Results are reported separately for fit_transform and
fit_then_transform, and for float32 and float64. Both API measurements
include fitting. Timings are median [min–max], and speedups are medians
of matched record-level KNNImputer/FaissImputer timing ratios.
FaissImputer's available-donor mode showed similar aggregate reconstruction RMSE and MAE to KNNImputer under the tested conditions. This does not establish identical predictions or algorithmic equivalence. Complete-donor results are reported alongside donor counts and reconstruction error to make the quality trade-off explicit.
The Intel k=30 observation and Intel 50,000-row stress result are presented separately from the AMD sweeps. The report also documents changes in guaranteed complete rows across neighbor settings and the stress run's single repeat per seed.
Full report and methodology · Raw benchmark results · Full-precision analysis summary
Held-out real-data benchmark — source commit ef04b1b
A development build at commit ef04b1b was evaluated on Wine Quality
(white) and Abalone, using 1,000 and 3,000 training rows with 1,000
held-out queries, MCAR/MAR missingness, and float32/float64 inputs.
Methods include mean and median baselines, KNNImputer, and both
FaissImputer donor policies.
Timings include fit plus first transform and are reported as median [min–max] across three seeds and three repeats. Speedups are medians of nine matched record-level KNNImputer/FaissImputer timing ratios. Datasets and dtypes are reported separately.
Available mode was slower than KNNImputer on Wine Quality for both dtypes and on Abalone for float64; it was faster on Abalone for float32. Complete mode was faster on both datasets. Reconstruction error and donor counts are reported alongside timing.
RMSE and MAE measure reconstruction error on masked query entries. Quality summaries use three seed datasets after verifying consistency across repeats. Available-mode output differences from KNNImputer were observed, including with float64 inputs on Abalone; similar aggregate errors do not establish identical predictions.
Full report and methodology · Raw benchmark evidence · Full-precision analysis summary
Installation
Requires Python 3.10 or newer.
python -m pip install --upgrade faiss-imputer
Quick start
import numpy as np
from faiss_imputer import FaissImputer
train = np.array([[0, 10], [2, 20], [4, 40]], dtype=np.float32)
query = np.array([[1.8, np.nan]], dtype=np.float32)
imputer = FaissImputer(n_neighbors=1)
print(imputer.fit(train).transform(query))
# [[ 1.8 20. ]]
For partially observed training data, use
FaissImputer(donor_policy="available").
More examples cover partial donors, indicators, numeric missing markers, custom metrics, and pandas pipelines.
When should I use it?
Use FaissImputer when distance-based imputation suits your numerical data and you need control over donor eligibility or reuse of fitted donors across query batches. Scale features appropriately before using distances.
Prefer KNNImputer when its behavior meets your needs and FaissImputer offers no measured advantage for your workload. FaissImputer is not a drop-in replacement: defaults, search precision, ties, and some edge cases differ.
Donor policies
"complete"— default: Use training rows observed in every non-empty feature. When non-empty features exist, at leastn_neighborscomplete donors are required."available": Use partially observed training rows and select donors separately for each missing feature. Fewer thann_neighborsusable donors are allowed.
In available-donor mode with built-in L2 metrics, a donor must observe the target feature and share an observed feature with the query. Squared L2 distances use those shared coordinates, scaled by the original feature count divided by the shared count. Eligibility therefore varies by target feature, unlike ordinary retrieval from a fixed set of complete vectors.
If no usable donor exists, the fitted column statistic supplies the value.
Available mode requires index_factory="Flat".
Comparison with KNNImputer
| Behavior | FaissImputer | KNNImputer |
|---|---|---|
| Donors | Complete rows by default; feature-specific partial donors in available mode | Feature-specific donors; other donor features may be missing |
| Built-in search | Faiss-backed search for complete donors; missing-aware search for available donors | Missing-aware pairwise distance computation |
| Default neighbors | 3 | 5 |
| Aggregation | Mean or median; non-uniform weights with mean | Mean with uniform, distance, or callable weights |
Both support fit/transform workflows, pipelines, missing indicators,
feature names, and pandas output through set_output.
For comparable configurations, use available donors and matching neighbor counts, weights, and missing markers. Matching settings does not guarantee identical output. In particular, built-in complete-donor search uses float32 vectors even when stored values and output are float64.
See the API reference for precision, copy behavior, callbacks, and edge cases.
Documentation
- API reference
- Usage examples
- Benchmark reports and reproduction details
- Migration notes and historical correctness warnings
- Roadmap
- Release history
Contributing
Bug reports and focused pull requests are welcome. Please include a reproducible example when reporting unexpected behavior through the issue tracker.
Author: ScionKim.
License
Faiss is developed by Meta and distributed under the MIT License. FaissImputer is not affiliated with or endorsed by Meta or the Faiss maintainers.
Metadata
Release files for faiss-imputer 0.3.21
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| faiss_imputer-0.3.21.tar.gz | 68.1 kB | With provenance | Details for faiss_imputer-0.3.21.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| faiss_imputer-0.3.21-py3-none-any.whl | Python 3 | none | any | Details for faiss_imputer-0.3.21-py3-none-any.whl |
Total release size: 87.7 kB
Release files / faiss_imputer-0.3.21.tar.gz
| Download URL | faiss_imputer-0.3.21.tar.gz |
|---|---|
| Size | 68.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
0fbe4287c047359a10565869533bcb9a4b37a257e5b534a91bccf88227a952e0
|
|
BLAKE2b-256 checksum How to use checksums |
ce197f9f5cd068c599dc29a1ceb9c32579a07284c639b2304445c884345d6a0b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 29, 2026.
Transparency logRelease files / faiss_imputer-0.3.21-py3-none-any.whl
| Download URL | faiss_imputer-0.3.21-py3-none-any.whl |
|---|---|
| Size | 19.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
e8d3bd2d3375027568b8774d0cde5b68d37622471ae7c8efcf8a30ef10a66f58
|
|
BLAKE2b-256 checksum How to use checksums |
27debb127f427b0e7e96b326cf43d8314324d9450f211473d9689ed8822bf17e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 29, 2026.
Transparency logSecurity for faiss-imputer 0.3.21
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for faiss_imputer-0.3.21.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 29, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.