Pari

Pari is a fast similarity indexing and deduplication engine for large datasets.

The project is built around a Rust core with Python bindings and a CLI. The goal is to let users move from an in-memory prototype to persistent or shared indexes without rewriting their similarity logic.

Status

Pari is pre-alpha. Public APIs may still evolve, but the supported 0.1 surface, machine-readable CLI contract, signature semantics, and persisted-format guarantees are defined in docs/compatibility.md.

Python quick start

Install from the repository:

python -m pip install .

Build signatures and a persistent index:

from pari import Index, MinHash

first = MinHash.from_values([b"new york", b"rust", b"search"], num_perm=128, seed=7)
second = MinHash.from_values([b"new york", b"python", b"search"], num_perm=128, seed=7)

with Index.create("documents.pari", threshold=0.8, num_perm=128, seed=7) as index:
    index.add_many([(1, first), (2, second)])
    print(index.search(first))

Index.search returns approximate LSH candidates, not exact duplicate decisions. See docs/python.md for the full typed API and docs/persistence.md for durability semantics.

CLI quick start

Build the command-line binary:

cargo build --release -p pari-cli

Given JSONL records such as:

{"key":1,"values":["new york","rust","search"]}
{"key":2,"values":["new york","python","search"]}

build and verify a persistent index:

pari index --input documents.jsonl --output documents.pari --json
pari verify --index documents.pari --json

Run native duplicate grouping without sending candidate edges through Python:

pari dedup --input documents.jsonl --emit groups --json

See docs/cli.md for indexing, search, deduplication, inspection, JSONL output, verification, and shell completion.

Shared Redis indexes

Rust services that need one index shared across processes can use pari-backend with its optional Redis feature. The same BackendIndex32 logic runs against both the in-process memory backend and Redis; Redis details are kept out of pari-index.

use pari_backend::{BackendIndex32, RedisBackend};

let backend = RedisBackend::connect("redis://127.0.0.1:6379/", "documents")?;
let index = BackendIndex32::create(backend, 0.8, 128, 7, None)?;
# Ok::<(), Box<dyn std::error::Error>>(())

See docs/storage-backends.md for the typed backend contract, batching, namespace ownership, TTL semantics, cleanup, security, and benchmark behavior.

Design goals

Batch-first compute, insert, query, and remote storage paths.
Safe, versioned persistence instead of executable object serialization.
A persistent local index between RAM-only prototypes and remote databases.
Shared backends without coupling LSH code to database-specific commands.
A small Rust core shared by Python and CLI frontends.
Correctness tests and benchmark evidence for performance-sensitive changes.
No merge to main while required CI is failing.

Layers

pari-core: signatures and similarity primitives.
pari-index: LSH and candidate generation.
pari-store: crash-safe local persistent indexes.
pari-backend: typed in-process and shared remote storage backends, including Redis.
pari-py: PyO3 Python bindings and the pari package.
pari-cli: command-line workflows.

See docs/architecture.md, docs/compatibility.md, and the GitHub issues for the implementation roadmap.

Development

cargo fmt --all -- --check
cargo clippy --workspace --all-targets --all-features -- -D warnings
cargo test --workspace --all-targets --all-features

Python wheel development additionally uses maturin:

python -m pip install "maturin>=1.14,<2"
maturin build --release

License

Pari is MIT licensed. See LICENSE. Portions derived from or informed by third-party MIT-licensed projects are documented in NOTICE.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pari_similarity-0.1.0.tar.gz (59.7 kB view details)

Uploaded Aug 28, 2026 Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

pari_similarity-0.1.0-cp310-abi3-win_amd64.whl (279.5 kB view details)

Uploaded Aug 28, 2026 CPython 3.10+Windows x86-64

pari_similarity-0.1.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (373.8 kB view details)

Uploaded Aug 28, 2026 CPython 3.10+manylinux: glibc 2.17+ x86-64

pari_similarity-0.1.0-cp310-abi3-macosx_11_0_arm64.whl (340.3 kB view details)

Uploaded Aug 28, 2026 CPython 3.10+macOS 11.0+ ARM64

File details

Details for the file pari_similarity-0.1.0.tar.gz.

File metadata

Download URL: pari_similarity-0.1.0.tar.gz
Upload date: Aug 28, 2026
Size: 59.7 kB
Tags: Source
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for pari_similarity-0.1.0.tar.gz
Algorithm	Hash digest
SHA256	`0f7ac44f8039a5f37e7f85ef830687795c3ef090d13dde8acf0df9e57d268f8d`
MD5	`eccb695426a1acdf16875910043c2632`
BLAKE2b-256	`6024ffbcea3f729926801b6a6f67cf80baccf8cba1233b1d283550b284994a35`

See more details on using hashes here.

Provenance

The following attestation bundles were made for pari_similarity-0.1.0.tar.gz:

Publisher: release.yml on dipeshbabu/pari

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: pari_similarity-0.1.0.tar.gz
- Subject digest: 0f7ac44f8039a5f37e7f85ef830687795c3ef090d13dde8acf0df9e57d268f8d
- Sigstore transparency entry: 2622792629
- Sigstore integration time: Aug 28, 2026
Source repository:
- Permalink: dipeshbabu/pari@6093bed543a14e78e046a44ce1c3182d57d1cc7c
- Branch / Tag: refs/heads/release/v0.1.0-pypi-recovery
- Owner: https://github.com/dipeshbabu
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: release.yml@6093bed543a14e78e046a44ce1c3182d57d1cc7c
- Trigger Event: workflow_dispatch

File details

Details for the file pari_similarity-0.1.0-cp310-abi3-win_amd64.whl.

File metadata

Download URL: pari_similarity-0.1.0-cp310-abi3-win_amd64.whl
Upload date: Aug 28, 2026
Size: 279.5 kB
Tags: CPython 3.10+, Windows x86-64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for pari_similarity-0.1.0-cp310-abi3-win_amd64.whl
Algorithm	Hash digest
SHA256	`b878169c080400d7beca9b673639096536ceb5594066eae2f4fe17558cd129ea`
MD5	`430e24cf7a5a3395054fb7b90b504687`
BLAKE2b-256	`f0982f7556c7f58af72767fd83f178b62192fe515c9efbde3399dee52e6091c7`

See more details on using hashes here.

Provenance

The following attestation bundles were made for pari_similarity-0.1.0-cp310-abi3-win_amd64.whl:

Publisher: release.yml on dipeshbabu/pari

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: pari_similarity-0.1.0-cp310-abi3-win_amd64.whl
- Subject digest: b878169c080400d7beca9b673639096536ceb5594066eae2f4fe17558cd129ea
- Sigstore transparency entry: 2622633929
- Sigstore integration time: Aug 28, 2026
Source repository:
- Permalink: dipeshbabu/pari@722d8695f5bd752bc0b051a05a274d5325a252e1
- Branch / Tag: refs/tags/v0.1.0
- Owner: https://github.com/dipeshbabu
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: release.yml@722d8695f5bd752bc0b051a05a274d5325a252e1
- Trigger Event: push

File details

Details for the file pari_similarity-0.1.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

Download URL: pari_similarity-0.1.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Upload date: Aug 28, 2026
Size: 373.8 kB
Tags: CPython 3.10+, manylinux: glibc 2.17+ x86-64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for pari_similarity-0.1.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm	Hash digest
SHA256	`83b3868c3e1e01b81c164c9b96500d1aa958021f4e4af5bd7b58bd63611d51b5`
MD5	`6c9bde02477e2dded0d0fa286c5e9020`
BLAKE2b-256	`043cea50b3bad56958c2941101eb1256d9f0e00739ebd9f036d07b80a9f818ff`

See more details on using hashes here.

Provenance

The following attestation bundles were made for pari_similarity-0.1.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: release.yml on dipeshbabu/pari

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: pari_similarity-0.1.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
- Subject digest: 83b3868c3e1e01b81c164c9b96500d1aa958021f4e4af5bd7b58bd63611d51b5
- Sigstore transparency entry: 2622633120
- Sigstore integration time: Aug 28, 2026
Source repository:
- Permalink: dipeshbabu/pari@722d8695f5bd752bc0b051a05a274d5325a252e1
- Branch / Tag: refs/tags/v0.1.0
- Owner: https://github.com/dipeshbabu
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: release.yml@722d8695f5bd752bc0b051a05a274d5325a252e1
- Trigger Event: push

File details

Details for the file pari_similarity-0.1.0-cp310-abi3-macosx_11_0_arm64.whl.

File metadata

Download URL: pari_similarity-0.1.0-cp310-abi3-macosx_11_0_arm64.whl
Upload date: Aug 28, 2026
Size: 340.3 kB
Tags: CPython 3.10+, macOS 11.0+ ARM64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for pari_similarity-0.1.0-cp310-abi3-macosx_11_0_arm64.whl
Algorithm	Hash digest
SHA256	`1fbbd92c10f3ed7d23542b5229323b679497581a22807eae53a790f7e2c98299`
MD5	`a780b7d8164b7649249f9a6cfe16ff83`
BLAKE2b-256	`c451354ec659a39c451bc48423f68a30f2897ed0e2dbe0585edb6c3bc6d5de23`

See more details on using hashes here.

Provenance

The following attestation bundles were made for pari_similarity-0.1.0-cp310-abi3-macosx_11_0_arm64.whl:

Publisher: release.yml on dipeshbabu/pari

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: pari_similarity-0.1.0-cp310-abi3-macosx_11_0_arm64.whl
- Subject digest: 1fbbd92c10f3ed7d23542b5229323b679497581a22807eae53a790f7e2c98299
- Sigstore transparency entry: 2622635074
- Sigstore integration time: Aug 28, 2026
Source repository:
- Permalink: dipeshbabu/pari@722d8695f5bd752bc0b051a05a274d5325a252e1
- Branch / Tag: refs/tags/v0.1.0
- Owner: https://github.com/dipeshbabu
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: release.yml@722d8695f5bd752bc0b051a05a274d5325a252e1
- Trigger Event: push

Release history Release notifications | RSS feed

0.2.0

Sep 1, 2026 4 files

This release

0.1.0 This release

Aug 28, 2026 4 files

pari-similarity 0.1.0

Pari

Status

Python quick start

CLI quick start

Shared Redis indexes

Design goals

Layers

Development

License

Project links

Key dates

1 maintainer

Credits

GitHub Statistics

License expression

Requires

Tags

Classifiers

Download files

Source Distribution

Built Distributions

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

Release history Release notifications | RSS feed