Pari

Pari is a fast similarity indexing and deduplication engine for large datasets.

The project is built around a Rust core with Python bindings and a CLI. The goal is to let users move from an in-memory prototype to persistent or shared indexes without rewriting their similarity logic.

Status

Pari 0.2.0 alpha is the current release line for PyPI, crates.io, and GitHub Releases. Published artifacts come only from the signed v0.2.0 tag; until that tag exists, this source tree is a release candidate and 0.1.0 remains the latest published version. Pari is still pre-1.0: supported 0.2.x interfaces follow the v0.x compatibility policy, while experimental surfaces may change in a future minor release with release notes and migration guidance.

Choose the interface that fits your workload:

Interface Install or download Start here
Python 3.10+ python -m pip install "pari-similarity==0.2.0" Python API guide
Rust 1.81+ cargo add pari-core@0.2.0 pari-index@0.2.0 Architecture and crate layers
CLI Download a 0.2.0 archive CLI guide

The project links collect release notes, compatibility and security policies, integrations, workload guides, and benchmark evidence.

Python quick start

Install the published distribution (CPython 3.10 or newer):

python -m pip install "pari-similarity==0.2.0"

The distribution name is pari-similarity; the import name is pari:

from pari import MinHash

first = MinHash.from_values(
    [b"new york", b"rust", b"search"],
    num_perm=128,
    seed=7,
)
second = MinHash.from_values(
    [b"new york", b"python", b"search"],
    num_perm=128,
    seed=7,
)

print(first.jaccard(second))

See the Python API guide for persistent indexes, batch operations, and the 0.2 API surface.

Rust quick start

The four public crates are versioned and released together at 0.2.0:

Crate Use it for
pari-core MinHash signatures and similarity primitives
pari-format Safe codecs and the versioned index format
pari-index In-memory LSH indexing and duplicate grouping
pari-store Crash-safe local persistent indexes

Add only the layers your application uses:

cargo add pari-core@0.2.0 pari-index@0.2.0 pari-store@0.2.0

Or declare them directly:

[dependencies]
pari-core = "0.2.0"
pari-index = "0.2.0"
pari-store = "0.2.0"

Add pari-format = "0.2.0" only when working directly with codecs or format metadata.

CLI quick start

Download the published archive for your platform:

Platform Archive
Linux x86_64 pari-0.2.0-linux.tar.gz
macOS arm64 pari-0.2.0-macos.tar.gz
Windows x86_64 pari-0.2.0-windows.zip

Linux:

curl -LO https://github.com/dipeshbabu/pari/releases/download/v0.2.0/pari-0.2.0-linux.tar.gz
tar -xzf pari-0.2.0-linux.tar.gz
./pari-0.2.0-linux/pari --version

macOS:

curl -LO https://github.com/dipeshbabu/pari/releases/download/v0.2.0/pari-0.2.0-macos.tar.gz
tar -xzf pari-0.2.0-macos.tar.gz
./pari-0.2.0-macos/pari --version

Windows PowerShell:

Invoke-WebRequest -Uri "https://github.com/dipeshbabu/pari/releases/download/v0.2.0/pari-0.2.0-windows.zip" -OutFile "pari-0.2.0-windows.zip"
Expand-Archive -Path "pari-0.2.0-windows.zip" -DestinationPath .
.\pari-0.2.0-windows\pari.exe --version

Verify downloads with the published SHA256SUMS before installing the binary on PATH.

Given JSONL records such as:

{"key":1,"values":["new york","rust","search"]}
{"key":2,"values":["new york","python","search"]}

build and verify a persistent index:

pari index --input documents.jsonl --output documents.pari --json
pari verify --index documents.pari --json

Run native duplicate grouping without sending candidate edges through Python:

pari dedup --input documents.jsonl --emit groups --json

See docs/cli.md for indexing, search, deduplication, inspection, JSONL output, verification, and shell completion.

Source users evaluating shared memory or Redis backends can follow the storage backend guide; those experimental crates are not part of the published 0.2.0 Rust set.

Project links

Design goals

  • Batch-first compute, insert, query, and remote storage paths.
  • Safe, versioned persistence instead of executable object serialization.
  • A persistent local index between RAM-only prototypes and remote databases.
  • Shared backends without coupling LSH code to database-specific commands.
  • A small Rust core shared by Python and CLI frontends.
  • Correctness tests and benchmark evidence for performance-sensitive changes.
  • No merge to main while required CI is failing.

Layers

  • pari-core: signatures and similarity primitives.
  • pari-index: LSH and candidate generation.
  • pari-store: crash-safe local persistent indexes.
  • pari-backend: typed in-process and shared remote storage backends, including Redis.
  • pari-py: PyO3 Python bindings and the pari package.
  • pari-cli: command-line workflows.

See docs/architecture.md, docs/compatibility.md, and the GitHub issues for the implementation roadmap.

Development

cargo fmt --all -- --check
cargo clippy --workspace --all-targets --all-features -- -D warnings
cargo test --workspace --all-targets --all-features

Python wheel development additionally uses maturin:

python -m pip install "maturin>=1.14,<2"
python -m pip install .
maturin build --release

Build the CLI from source with cargo build --release -p pari-cli.

License

Pari is MIT licensed. See LICENSE. Portions derived from or informed by third-party MIT-licensed projects are documented in NOTICE.

Project links

Data verified by PyPI on Sep 1, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for dipesh1701 from gravatar.com dipesh1701

Credits

Author: Dipesh Babu

GitHub Statistics

Data verified by PyPI on Sep 1, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License expression

MIT
View SPDX License List

Requires

Python >=3.10

Provides Extra

datasketch huggingface integrations polars pyarrow

Tags

minhash lsh similarity deduplication rust

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pari_similarity-0.2.0.tar.gz (128.3 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

pari_similarity-0.2.0-cp310-abi3-win_amd64.whl (404.6 kB view details)

Uploaded CPython 3.10+Windows x86-64

pari_similarity-0.2.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (511.2 kB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ x86-64

pari_similarity-0.2.0-cp310-abi3-macosx_11_0_arm64.whl (465.7 kB view details)

Uploaded CPython 3.10+macOS 11.0+ ARM64

File details

Details for the file pari_similarity-0.2.0.tar.gz.

File metadata

  • Download URL: pari_similarity-0.2.0.tar.gz
  • Upload date:
  • Size: 128.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for pari_similarity-0.2.0.tar.gz
Algorithm Hash digest
SHA256 3195923467b821493dafe2858ce71f8d1eabdca75673116e42f3bfc3378c0415
MD5 f7364f66a959b019d29278e0784a7626
BLAKE2b-256 f3df44bfe3a4662da5fe34524afc77c1cffe7ccafdaa15869f3f62f820a4c8b3

See more details on using hashes here.

Provenance

The following attestation bundles were made for pari_similarity-0.2.0.tar.gz:

Publisher: release.yml on dipeshbabu/pari

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file pari_similarity-0.2.0-cp310-abi3-win_amd64.whl.

File metadata

File hashes

Hashes for pari_similarity-0.2.0-cp310-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 7f8000bba3dd771429c1cca332c844fbec949a4300163485c12ec80b13c813e3
MD5 0a87f412bced14c061f22e7e871fb73b
BLAKE2b-256 dd6d39f865b0871c6a5213fbdff0150b0ba2df36b5bb7c2c92bfaa995f8bf28e

See more details on using hashes here.

Provenance

The following attestation bundles were made for pari_similarity-0.2.0-cp310-abi3-win_amd64.whl:

Publisher: release.yml on dipeshbabu/pari

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file pari_similarity-0.2.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for pari_similarity-0.2.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 20f2f3ae587f71f240537cfc598948e6976a18dcd8d6668e659c095ed7731237
MD5 e81b8fdb87ad6f49f8c8d1d13ad63c81
BLAKE2b-256 bd6911fd518a760e2bc8dea70febec2fa387f5eef65d4f085b29b665bbb5cdec

See more details on using hashes here.

Provenance

The following attestation bundles were made for pari_similarity-0.2.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: release.yml on dipeshbabu/pari

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file pari_similarity-0.2.0-cp310-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for pari_similarity-0.2.0-cp310-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 3a7f933a039ee01f8010c9eac694d0718b512c5b92c2f8584420850a2f5b3387
MD5 404ae05660539047f4546fc77d390f9b
BLAKE2b-256 a4658615b23c07a0d39a9fed04fa316761db80fea13c306c66ff732329613b93

See more details on using hashes here.

Provenance

The following attestation bundles were made for pari_similarity-0.2.0-cp310-abi3-macosx_11_0_arm64.whl:

Publisher: release.yml on dipeshbabu/pari

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

This release

0.2.0 This release

4 files

0.1.0

4 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page