Pari

Find similar records and deduplicate datasets with Rust, Python, or your terminal.

Pari builds MinHash signatures, finds approximate similarity candidates with LSH, and groups duplicate records. Start in memory, save an index to a versioned .pari file, or process bounded batches from larger datasets through the same Rust engine.

Pari 0.1.1 alpha targets Python 3.10+ and Rust 1.81+ for the public crates. The project is pre-1.0; see the compatibility policy. The Python extension requires Rust 1.83 when built from source.

Installation

Python

python -m pip install "pari-similarity==0.1.1"

The distribution name is pari-similarity; the import name is pari.

Rust

cargo add pari-core@0.1.1 pari-index@0.1.1 pari-store@0.1.1

Use only the layers your application needs. Add pari-format = "0.1.1" when working directly with codecs or persisted-format metadata. Keep public crates on the same release version.

CLI

Download an archive from the 0.1.1 release, verify it against SHA256SUMS, extract it, and put pari on your PATH.

Packages and archives become available when the tagged release workflow completes. See release notes and platform support.

Compare Records

Build two signatures and estimate their Jaccard similarity:

from pari import MinHash

first = MinHash.from_values(
    [b"new york", b"rust", b"search"], num_perm=128, seed=7
)
second = MinHash.from_values(
    [b"new york", b"python", b"search"], num_perm=128, seed=7
)

print(first.jaccard(second))

Use the same permutation count and seed for signatures you compare. The Python guide covers persistent indexes, batch operations, and the explicit MinHash64 and Index64 APIs.

Index and Search

Create documents.jsonl with one record per line:

{"key":1,"values":["new york","rust","search"]}
{"key":2,"values":["new york","python","search"]}

Build a persistent index and verify its contents:

pari index --input documents.jsonl --output documents.pari --json
pari verify --index documents.pari --json
pari search --help

Indexes use safe, versioned persistence. LSH returns approximate candidates; verify candidates against source features when your application needs an exact similarity threshold.

Deduplicate a Dataset

Group duplicate candidates through the native engine:

pari dedup --input documents.jsonl --emit groups --json

Python's DedupeIndex and deduplicate add bounded batches, exact verification, representative selection, and progress callbacks. Optional adapters read PyArrow/Parquet, Polars, and Hugging Face datasets.

• Python deduplication
• Dataset integrations
• Text deduplication and corpus audits
• Code corpus deduplication
• Entity matching

Reference

Use --help to explore commands and options:

pari --help
pari index --help
pari dedup --help

• CLI reference and Python API
• Architecture and storage backends
• Datasketch 2.x interoperability
• LSH planning, observability, and CPU parallelism
• Benchmark methodology and recorded benchmark evidence
• Compatibility, changelog, and release procedure

The committed benchmark scale envelope is 1M records. Performance figures apply to the environment and workload recorded with each result.

Developing

git clone https://github.com/dipeshbabu/pari.git
cd pari

python -m pip install "pre-commit==4.6.2"
pre-commit install
pre-commit run --all-files

cargo fmt --all -- --check
cargo clippy --workspace --all-targets --all-features -- -D warnings
cargo test --workspace --all-targets --all-features

CI remains authoritative. Build the CLI with cargo build --release -p pari-cli. For Python development, install the checkout with python -m pip install ..

Contributing

See CONTRIBUTING.md for setup, checks, and compatibility rules. Report bugs and propose changes through GitHub issues. For security reports, follow SECURITY.md.

License

Pari is released under the MIT license. Third-party attribution is recorded in NOTICE.