Project:
Package:
Development:
Contents
Overview
This project provides a modular benchmark pipeline for experimenting with different vector databases (FAISS, Qdrant, …).
It runs end-to-end:
- Download → Hugging Face dataset (optionally export images + manifest)
- Embed → Generate CLIP embeddings for images
- Build → Construct indexes with multiple VectorDBs
- Search → Profile queries (latency + Recall@K vs exact baseline)
- Update → Test insertions & deletions (index maintenance)
All steps are run with uv as the package manager.
How to use iNatInqPerf
# Install uv
curl -LsSf https://astral.sh/uv/install.sh | sh
# Setup environment
uv venv .venv && source .venv/bin/activate
uv sync
# Run an end-to-end benchmark (FAISS IVF+PQ vectordb) on the INQUIRE dataset.
uv run python scripts/run_benchmark.py configs/inquire_benchmark.yaml
# Spin up a 3-node Weaviate cluster (shared Docker network + RAFT) and run the benchmark.
uv run python scripts/run_benchmark.py configs/inquire_benchmark_weaviate_cluster.yaml
# Spin up a 3-node Qdrant cluster (HTTP+gRPC+p2p) and run the benchmark.
uv run python scripts/run_benchmark.py configs/inquire_benchmark_qdrant_cluster.yaml
Distributed VectorDB Deployments
- Benchmark-managed clusters. The
configs/inquire_benchmark_weaviate_cluster.yamlandconfigs/inquire_benchmark_qdrant_cluster.yamlfiles include the container descriptions thatcontainer_contextwill launch automatically before each run. Make sure no identically named containers are already running, otherwise Docker will raise a name-conflict error.
The benchmarking code will
- Download the specified dataset from the HuggingFace website.
- Embed the images using a CLIP model.
- Build a vector database index.
- Perform a search for given queries to obtain query latency, and compute Recall@K vs FAISS Flat baseline..
- Update the index.
Dataset Output Structure
data/raw/
dataset_info.json
state.json
data-00000-of-00001.arrow
images/
00000000.jpg
00000001.jpg
...
images/manifest.csv # [index,filename,label]
Supported Vector Databases
faiss.flat(exact)faiss.ivfpq(IVF + OPQ + PQ)
Profiling Outputs
- Latency statistics (avg, p50, p95)
- Recall@K vs baseline
- JSON metrics in
.results/
Profiling with py-spy
Use py-spy to record flamegraphs during any step:
bash scripts/pyspy_run.sh search-faiss -- python src/inatinqperf/benchmark/benchmark.py search --vectordb faiss.ivfpq --hf_dir data/emb_hf --topk 10 --queries src/inatinqperf/benchmark/queries.txt
Outputs:
.results/search-faiss.svg(flamegraph).results/search-faiss.speedscope.json
Installation
| Installation Method | Command |
|---|---|
| Via uv | uv add inatinqperf |
| Via pip | pip install inatinqperf |
Development
Please visit Contributing and Development for information on contributing to this project.
Additional Information
Additional information can be found at these locations.
| Title | Document | Description |
|---|---|---|
| Code of Conduct | CODE_OF_CONDUCT.md | Information about the norms, rules, and responsibilities we adhere to when participating in this open source community. |
| Contributing | CONTRIBUTING.md | Information about contributing to this project. |
| Development | DEVELOPMENT.md | Information about development activities involved in making changes to this project. |
| Governance | GOVERNANCE.md | Information about how this project is governed. |
| Maintainers | MAINTAINERS.md | Information about individuals who maintain this project. |
| Security | SECURITY.md | Information about how to privately report security issues associated with this project. |
License
iNatInqPerf is licensed under the MIT license.
1 maintainer
Metadata
1 maintainer
Release files for inatinqperf 0.1.108
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| inatinqperf-0.1.108.tar.gz | 307.4 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| inatinqperf-0.1.108-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 345.9 kB
Release files / inatinqperf-0.1.108.tar.gz
| Download URL | inatinqperf-0.1.108.tar.gz |
|---|---|
| Size | 307.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
d34f21fe97f500b6b71e6cde2e5c394c9c27162a799f38d06c0ac4bb77ee736f
|
|
BLAKE2b-256 checksum How to use checksums |
f1dc1717770bf2996c67b042f91f060105d4418b4b2272ef5ba6b58b1a628e3b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.9.17 {"installer":{"name":"uv","version":"0.9.17","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / inatinqperf-0.1.108-py3-none-any.whl
| Download URL | inatinqperf-0.1.108-py3-none-any.whl |
|---|---|
| Size | 38.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
59fd06b9095249d32ef22c6e93d3c34b5cd00d7a4fa195efe33483549b427f37
|
|
BLAKE2b-256 checksum How to use checksums |
6fefd0274068cf5c6b74b68628ece20fcae8ae8308d88ba3f0c6f259263e83f0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.9.17 {"installer":{"name":"uv","version":"0.9.17","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Security for inatinqperf 0.1.108
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
