Skip to main content

langchain-redefiner

Personal-data redaction and entity extraction for LangChain, using ReDefiNER: an open model (0.8B parameters, Apache 2.0) that finds what a rulebook written in plain English defines. It runs locally on CPU or GPU, so text is redacted before it leaves your machine. English only.

  • ReDefiNERPIIMiddleware: agent middleware that redacts personal data in user messages (and optionally model output and tool results) before the model sees it. It is LangChain's own PIIMiddleware with ReDefiNER as the detector, so one middleware covers every type in the rulebook.
  • redefiner_detector: the detector on its own, for PIIMiddleware(..., detector=...).
  • ReDefiNERRedactor and ReDefiNEREntityExtractor: document transformers that mask documents, or add the entities they mention to their metadata.

By default the PII rulebook of jungroo-ai/redefiner-pii is used (20 types: names, contact details, addresses, dates, ID and card numbers and more). Pass your own rulebook to redact by your own policy.

Install

pip install langchain-redefiner

The first call downloads the model weights (about 0.8 GB) from Hugging Face.

Agent middleware

from langchain.agents import create_agent
from langchain_redefiner import ReDefiNERPIIMiddleware

agent = create_agent(
    "openai:gpt-5",
    tools=[...],
    middleware=[ReDefiNERPIIMiddleware()],
)
agent.invoke({"messages": [{"role": "user", "content": "Refund Meera Nair, email meera.nair@example.org."}]})
# the model receives: "Refund [REDACTED_PERSON], email [REDACTED_EMAIL]."

Options:

Option Default Meaning
model "jungroo-ai/redefiner-pii" A Hugging Face repo or local folder
rulebook the model's default Rulebook text, a path to a rulebook file, or a dict
strategy "redact" LangChain's strategies: "redact", "mask", "hash" or "block"
types all Only these rulebook types, e.g. ["PERSON", "EMAIL"]
apply_to_input True Redact user messages before the model call
apply_to_output False Redact the model's messages
apply_to_tool_results False Redact tool results
min_confidence 0.0 Ignore spans below this confidence
none_bias, threshold, max_span_tokens model defaults ReDefiNER settings; see its README
device GPU if available "cpu", "cuda" or "cuda:N"

Each match's type is the rulebook type in lower case ([REDACTED_PERSON]). Emails, IP addresses, URLs and card numbers use LangChain's own type names, so strategy="mask" handles them as it does its built-in types (meera.nair@****.org).

To use LangChain's middleware directly:

from langchain.agents.middleware import PIIMiddleware
from langchain_redefiner import redefiner_detector

PIIMiddleware("redefiner", detector=redefiner_detector(types=["PERSON"]), strategy="redact")

Your own policy

Write the types you want masked, with a one-line description each, and anything that must stay:

policy = """[TYPE] CUSTOMER_NAME
[DESCRIPTION] The name of a customer. Our own staff, who sign as "Agent <name>", are not customers.

[TYPE] ACCOUNT_NUMBER
[DESCRIPTION] A customer account number, such as "AC-449120".
"""

middleware = ReDefiNERPIIMiddleware(rulebook=policy)
# "Customer Daniel Brandt (account AC-771204) asked for a refund. Agent Rosa Diaz approved it." reaches the model as
# "Customer [REDACTED_CUSTOMER_NAME] (account [REDACTED_ACCOUNT_NUMBER]) asked for a refund. Agent Rosa Diaz approved it."

See the rulebook guide. Try a policy on a few dozen of your own conversations before relying on it.

Documents

from langchain_core.documents import Document
from langchain_redefiner import ReDefiNEREntityExtractor, ReDefiNERRedactor

docs = [Document(page_content="Hi, this is Meera Nair from Brightwater Dental. Call me on 020 7946 0155.")]

ReDefiNERRedactor().transform_documents(docs)[0].page_content
# 'Hi, this is [PERSON] from [ORGANIZATION]. Call me on [PHONE].'

rulebook = """[TYPE] PERSON
[DESCRIPTION] A person's name.

[TYPE] ORGANIZATION
[DESCRIPTION] A company or institution.
"""
ReDefiNEREntityExtractor(rulebook=rulebook).transform_documents(docs)[0].metadata
# {'entities': {'PERSON': ['Meera Nair'], 'ORGANIZATION': ['Brightwater Dental']}}
  • The redactor records what it masked in metadata["redefiner_redactions"]: type, confidence and offsets, but not the masked text (unless keep_masked_text=True).
  • ReDefiNERRedactor().redact_text masks a single string; wrap it in RunnableLambda to use it in a chain.
  • ReDefiNEREntityExtractor(..., flat=True) writes one key per type ("entity_PERSON": "Meera Nair") for vector stores that only accept flat metadata.

Limits

  • Not the only safeguard. The model misses some personal data and marks some text that is not personal. Check it on your own data.
  • Speed. On CPU the first call reads the rulebook (several seconds for the PII rulebook) and is then cached; later short messages take under a second. Each redacted message is one model call.
  • English only.

Development

pip install -e ".[dev]"
pytest              # fast tests with a stand-in model
pytest -m slow      # the real model on CPU

Licence

Apache License 2.0. See LICENSE. Made by Jungroo AI Labs.

Metadata

Release files for langchain-redefiner 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for langchain-redefiner 0.1.0
File Size Uploaded Provenance Actions
langchain_redefiner-0.1.0.tar.gz 16.1 kB Without provenance Details for langchain_redefiner-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for langchain-redefiner 0.1.0
File Interpreter ABI Platform Actions
langchain_redefiner-0.1.0-py3-none-any.whl Python 3 none any Details for langchain_redefiner-0.1.0-py3-none-any.whl

Total release size: 29.4 kB

Release files / langchain_redefiner-0.1.0.tar.gz

Download URL langchain_redefiner-0.1.0.tar.gz
Size 16.1 kB
Tags Source
SHA-256 checksum
How to use checksums
72b2b239e762f5030bb933f5b97d0e7172fddeb125619576a913088a9b6a5447
BLAKE2b-256 checksum
How to use checksums
7a40d407c949c6ffa6e92b3ad64f79e14832e3e9906310dd5e76c36a4dbf3728
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.6

Release files / langchain_redefiner-0.1.0-py3-none-any.whl

Download URL langchain_redefiner-0.1.0-py3-none-any.whl
Size 13.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums
d4573c7548da173c4b371fe8d188eb38cbbe2df0e94f1b3e6022a8e388b325d8
BLAKE2b-256 checksum
How to use checksums
070ba2c45589ea70f31a0ec69b183f0d70a86994401400ff8570610ac7dcca0b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.6

Security for langchain-redefiner 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page