Skip to main content

llama-index-redefiner

Personal-data masking and entity extraction for LlamaIndex, using ReDefiNER: an open model (0.8B parameters, Apache 2.0) that finds what a rulebook written in plain English defines. It runs locally on CPU or GPU. English only.

Component Where it goes What it does
ReDefiNERPIINodePostprocessor query engine node_postprocessors Masks personal data in retrieved nodes before the LLM sees them, like LlamaIndex's NERPIINodePostprocessor, and keeps a mapping so answers can be mapped back
ReDefiNERPIIMasker IngestionPipeline transformations Masks personal data in node text before embedding and indexing, so it never reaches the embedding model or the vector store
ReDefiNEREntityExtractor IngestionPipeline transformations Adds the entities a rulebook defines to node metadata, like LlamaIndex's EntityExtractor

The PII components use the rulebook of jungroo-ai/redefiner-pii by default (20 types: names, contact details, addresses, dates, ID and card numbers and more). Pass your own rulebook to mask by your own policy.

Install

pip install llama-index-redefiner

The first call downloads the model weights (about 0.8 GB) from Hugging Face.

Mask retrieved nodes

from llama_index_redefiner import ReDefiNERPIINodePostprocessor

query_engine = index.as_query_engine(node_postprocessors=[ReDefiNERPIINodePostprocessor()])

A node that reads "Hi, this is Meera Nair from Brightwater Dental. Call me on 020 7946 0155." reaches the LLM as "Hi, this is [PERSON_12] from [ORGANIZATION_28]. Call me on [PHONE_59]." The mapping from tags to the original text is kept in node.metadata["__pii_node_info__"], which is excluded from what the LLM and the embedding model see, as with LlamaIndex's own PII postprocessors.

Mask before indexing

from llama_index.core import Document
from llama_index.core.ingestion import IngestionPipeline
from llama_index_redefiner import ReDefiNERPIIMasker

pipeline = IngestionPipeline(transformations=[ReDefiNERPIIMasker()])
nodes = pipeline.run(documents=[Document(text="Hi, this is Meera Nair from Brightwater Dental. Call me on 020 7946 0155.")])
nodes[0].get_content()
# 'Hi, this is [PERSON] from [ORGANIZATION]. Call me on [PHONE].'

Nothing of the masked text is kept. Put a text splitter and an embedding model after it in the same pipeline as usual.

Extract entities into metadata

from llama_index_redefiner import ReDefiNEREntityExtractor

rulebook = """[TYPE] PERSON
[DESCRIPTION] A person's name.

[TYPE] ORGANIZATION
[DESCRIPTION] A company.
"""
pipeline = IngestionPipeline(transformations=[ReDefiNEREntityExtractor(rulebook=rulebook)])
pipeline.run(documents=[Document(text="Hi, this is Meera Nair from Brightwater Dental.")])[0].metadata
# {'entities': ['Meera Nair', 'Brightwater Dental']}

With label_entities=True each type gets its own key ({"PERSON": [...], "ORGANIZATION": [...]}), renamed through entity_map if given.

Rulebooks and settings

A rulebook lists the types you want, with a one-line description each, and can add rules and a list of what must never be found. See the rulebook guide. rulebook takes the text or the path of a rulebook file.

Every component also takes model, min_confidence, none_bias, threshold, max_span_tokens, device, revision and batch_size; see the ReDefiNER README for what the model settings do.

Limits

  • Not the only safeguard. The model misses some personal data and marks some text that is not personal. Check it on your own documents.
  • Speed. On CPU the first call reads the rulebook (several seconds for the PII rulebook) and is then cached; later short texts take under a second. All nodes of a call go to the model in one batch.
  • English only.

Development

pip install -e ".[dev]"
pytest              # fast tests with a stand-in model
pytest -m slow      # the real model on CPU

Licence

Apache License 2.0. See LICENSE. Made by Jungroo AI Labs.

Metadata

Release files for llama-index-redefiner 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for llama-index-redefiner 0.1.0
File Size Uploaded Provenance Actions
llama_index_redefiner-0.1.0.tar.gz 12.3 kB Without provenance Details for llama_index_redefiner-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for llama-index-redefiner 0.1.0
File Interpreter ABI Platform Actions
llama_index_redefiner-0.1.0-py3-none-any.whl Python 3 none any Details for llama_index_redefiner-0.1.0-py3-none-any.whl

Total release size: 23.6 kB

Release files / llama_index_redefiner-0.1.0.tar.gz

Download URL llama_index_redefiner-0.1.0.tar.gz
Size 12.3 kB
Tags Source
SHA-256 checksum
How to use checksums
eb76bb2f12cd4751b7ed3adaf12e364875cfd286814a7746bae0c77e1307fa8c
BLAKE2b-256 checksum
How to use checksums
af5489f6a1828bbd4d94fe0bc7ef745f2d6cac5dc50ceae88b8fff35161ae6dd
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.6

Release files / llama_index_redefiner-0.1.0-py3-none-any.whl

Download URL llama_index_redefiner-0.1.0-py3-none-any.whl
Size 11.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
2e014ced03c1f937a3b328a8aad4c8c08fdbe0f88d12cda55d3e714e7cc48694
BLAKE2b-256 checksum
How to use checksums
d45193874d284aee1b08bd97da1efb0979956b1aa5e51737570ea075369f9636
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.6

Security for llama-index-redefiner 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page