edgepii

Locate personally identifiable information (PII) as typed character spans on the exact source string.

edgepii returns {type, start, end, text, score} rows. It does not label a whole document and it does not rewrite text with a chat model. Every row proves one invariant:

source[span.start:span.end] == span.text

That check is the contract. If a detector returns an offset that fails it, the library drops the span and counts it. The library never searches the string for a repair.

• Specification · Span contract · API · Examples · Evaluation · Public corpora · First score
• Python 3.11 or newer. Local weights. No cloud PII service.

Contents

• Why this library
• Install
• Quickstart
• Demo
• Presets
• Types
• API
• Masking and the vault
• Make targets
• Tests
• Site
• Evaluation
• Design rules
• Limits
• License

Why this library

Most PII tools answer one question: is this document sensitive? That answer is too coarse for a support ticket, a log line, or a prompt. The team still needs the rest of the string to do the work, and it needs proof of which characters left the system.

edgepii answers a narrower question with a checkable result: which characters, of which type, on this string?

	edgepii	Document classifier	Chat rewrite
Result	Typed character spans	One label per document	New prose
Offset proof	source[start:end] == text	None	None
Keeps non-PII text	Yes	Yes	Often no
Reversible mask	Yes, except secrets	Not applicable	Not applicable
Fixed cost per call	One model pass	One model pass	Token by token

Install

python3.11 -m venv .venv
.venv/bin/pip install edgepii

To work from a checkout instead, use pip install -e . in place of pip install edgepii.

The rule lane needs only phonenumbers and pyyaml. Model lanes are extras:

.venv/bin/pip install "edgepii[hf]"       # transformers plus torch: en-closed, en-bert
.venv/bin/pip install "edgepii[gliner2]"  # gliner2==2.1.0: multi-open
.venv/bin/pip install "edgepii[gliner]"   # gliner: phi-open
.venv/bin/pip install "edgepii[ms]"       # modelscope: zh-schema

A missing extra raises DetectorLoadError with the extra, the repo id, and the revision. It never includes source text.

Quickstart

Rules need no weights and no network:

from edgepii import Locator, redact

source = (
    "Customer Ada Lovelace called from +852 5555 0101. "
    "Email ada@example.com. Card 4242424242424242. Order 48291."
)
result = Locator.preset("rules").locate(source)

for span in result.spans:
    print(span.type, span.start, span.end, repr(span.text), span.score)

PHONE 34 48 '+852 5555 0101' None
EMAIL 56 71 'ada@example.com' None
PAYMENT 78 94 '4242424242424242' 1.0

A None score is a fact, not a gap. The phone and the email matched a grammar. The card passed the Luhn check. A fixed number for the grammar cases would order spans by nothing.

Order 48291 stays unspanned. An internal order id is not in the default type list.

Mask the same result:

masked, vault = redact(source, result.spans)
print(masked)
# Customer Ada Lovelace called from <PHONE_1>. Email <EMAIL_1>. Card <PAYMENT_1>. Order 48291.
print(vault)
# {'<PHONE_1>': '+852 5555 0101', '<EMAIL_1>': 'ada@example.com', '<PAYMENT_1>': '4242424242424242'}

Pick a type subset at call time:

result = Locator.preset("rules").locate(source, types=["EMAIL"])

Demo

The demo is a FastAPI service with one HTML5 page. It shows the span table, the highlighted source, the masked string, and the vault.

make demo          # http://127.0.0.1:8000
make demo-open     # start the server and open a tab

Home	Rules lane

Secret note	Numbered placeholders

Email only	Chinese line

Long text offsets	Copy as JSON

Model lane finds a name	Score floor, caller policy

Narrow viewport

The page calls four endpoints:

Method	Path	Purpose
GET	/api/health	Liveness for the e2e runner
GET	/api/presets	Lanes, plus ready when the extra exists
GET	/api/types	Taxonomy rows with class, rank, and threshold
POST	/api/locate	{source, preset, types} to spans, masked text, and vault

POST /api/locate returns 422 with {ok: false, error, message, hint} for a bad type, an over-long source, or a missing extra. The page shows that message in place, not a stack trace.

All demo text is fake. Do not paste real data.

Presets

A preset is a fixed detector list. locate does not branch on the preset name.

Preset	Weights	Pinned revision	Use when
rules	None	rules	You need email, phone, IBAN, card, IP, or a password cue with no download
en-closed	antoniogr7/pii-identifier-v0.1	3d0f39e	English text and a frozen type list. ModernBERT, Apache-2.0
multi-open	fastino/gliner2-privacy-filter-PII-multi	1cb4166	The caller must change the type list per call
en-bert	ab-ai/pii_model	bbb5e83	BERT control. The owner gates this repo, so you need access
phi-open	nvidia/gliner-PII	bd23e8e	Health-adjacent text. Its health labels are outside taxonomy 2026.11
zh-schema	iic/nlp_structbert_siamese-uie_chinese-base	master	Chinese text through ModelScope schema prompts

en-closed is the default candidate for a closed English lane. en-bert is a control, not the only answer to "is BERT enough?". The full argument is in when the encoder is enough.

Types

Taxonomy 2026.11 lists eleven public types. The YAML file is the only type list. One map folds each model label into a public type.

Type	Class	Rank	Prefer	Notes
SECRET	secret	40	rule	Password, key, token. Masked irreversibly
GOV_ID	gov_id	30	rule	National id, passport, tax id. Patterns stay opt-in
PAYMENT	payment	20	rule	Card after Luhn, IBAN after MOD-97
EMAIL	contact	10	rule	RFC 5321 shape. Score null
PHONE	contact	10	rule	libphonenumber validity or international length
IP_ADDRESS	contact	10	rule	stdlib parse after an RFC 4291 or RFC 3986 match
USERNAME	contact	10	model	A login name, not a display name
PERSON	person	10	model	Given name, surname, full name
ADDRESS	person	10	model	Street through locality
DATE_OF_BIRTH	quasi	0	model	Off by default
ORG	quasi	0	model	Off by default

Rank breaks an overlap. Higher rank wins. A tie then prefers a rule hit, then a longer span, then a higher score, then a smaller start, then the type name. See the taxonomy for the label map and the Chinese schema names.

API

edgepii.__version__

The installed version string. It is read from edgepii/_version.py, so the package and the wheel metadata cannot disagree.

Locator.preset(name, device="cpu", taxonomy="2026.11")

Build a locator with pinned detectors. Raises DetectorLoadError when an extra does not exist.

Locator.from_detectors(detectors, taxonomy="2026.11")

Build a locator from your own objects. A detector exposes locate(text, labels) and returns dicts or Hit rows. This method is the test seam and the eval seam.

locator.locate(source, types=None, taxonomy=None, min_score=None)

Returns LocateResult:

Field	Meaning
spans	Tuple of Span, sorted and non-overlapping
dropped	Count of offsets that failed the slice check
preset	Preset name, or custom
revision	Pinned revision for the preset
taxonomy_version	2026.11

Span freezes these fields: type, start, end, text, score, detector_id, and raw_label. score can be None when no validator exists for the form. The library sets no minimum score. min_score is caller policy.

The merge order does not change. Higher class rank wins. A tie prefers a detector kind the type names, then a longer span, then a higher score, then a smaller start, then the type name. The merge is order independent, so the same inputs give the same result.

locator.locate_many(sources, types=None)

Returns one result per source, in order. The method does not join strings, so offsets stay local to each element.

redact(source, spans, taxonomy="2026.11")

Returns (masked, vault). See below.

Errors

Error	When	Message
TypeError	source is not a str	The type name only
UnknownType	A requested type is not in the taxonomy	The bad name
DetectorLoadError	An extra or a weight file does not exist	extra=… repo=… revision=…
OverlapError	redact sees two spans that overlap	start:end only, no text

No error message contains PII.

Masking and the vault

redact walks spans from the right, so earlier offsets stay valid. A repeat of the same (type, text) pair reuses its placeholder.

Class	Output	In vault
secret	<SECRET>	Never
every other class	<TYPE_N>, left to right	Yes

A vault key maps to the original slice. The caller stores the vault. The function keeps nothing. Secrets stay irreversible because a restore would write a password into a ticket, a log, or a second prompt. See ADR 0004.

redact raises OverlapError if two spans overlap. The locator guarantees non-overlapping output, so the error is a guard, not a second merge pass.

Make targets

make help          # list every target
make install       # venv plus the library, dev tools, and pyarrow
make test          # contract, principle, eval, and demo API tests
make e2e           # Playwright run and screenshots
make demo          # serve on 127.0.0.1:8000
make examples      # run examples/run_local.py
make eval          # score rules on a public corpus, 500 docs by default
make eval-rules    # score the rule lane on both public corpora
make eval-model    # score en-closed on a public corpus
make modal-e2e     # model lanes on Modal
make clean         # remove caches

PORT=9000 make demo changes the port. make install-e2e adds Playwright and its Chromium build.

Tests

make test                      # 23 contract, 18 principle, 24 eval, 8 demo API tests
make e2e                       # 13 browser tests, screenshots to docs/e2e/screenshots
.venv/bin/pytest tests -q      # everything installed

Suite	What it locks
tests/test_acceptance.py	Every documented example, merge ties, window offset shifts, bad-offset drops, label folding, the rule slices, redact order, score and leakage
tests/test_principles.py	The source of every rule: RFC 5321, ISO 7812, ISO 13616, RFC 4291, libphonenumber, the absent score floor, the window invariant, merge determinism
tests/test_eval_corpora.py	The corpus pin, the license allow list, the label fold, the drop reason, the unknown label, the nested gold, and the per-document score
tests/test_demo_api.py	Endpoint shapes, secret exclusion from the vault, clean errors, the over-long guard, the caller floor
tests/test_demo_e2e.py	The page renders, spans appear, the filter works, the masked string is correct, copy works, the floor works, a model lane finds a name, the layout survives a narrow viewport

Screenshots live in docs/e2e/screenshots. make e2e rebuilds them.

Model lanes run on Modal, so a laptop needs no large download:

modal run modal/e2e_models.py

That job loads each pinned checkpoint in a container, checks the structured spans, and reports the result per lane. The job reports a gated checkpoint as skipped, never as a pass.

Site

The documentation site is static HTML, built from the docs in this repository. A build is three steps: record real library output, render the pages, then check them.

make site          # record, build, make images, check
make site-serve    # build, then serve on http://127.0.0.1:8099

Path	Role
site/gen_data.py	Calls the real library on fake strings and records the spans
site/build.py	Renders the Markdown docs and the guide pages into site/dist
site/images.py	Turns the 2 880 pixel e2e PNGs into responsive JPEGs
site/check.py	Gates the build on links, markup, contrast, and the span contract
site/content/	The tutorial pages, in plain English

Two properties are worth keeping:

The demo cannot lie. Every span it shows came from locate. check.py re-verifies source[start:end] == text on the recorded data before the deploy. If a span fails, the build stops.

The docs are not copied. build.py reads docs/*.md directly. An edit to a doc appears on the site without a second edit, and site/links.py rewrites the relative .md links into routes.

GitHub Pages serves the site from .github/workflows/site.yml. The workflow runs the same four commands, so a local build and a deployed build are the same build.

Evaluation

Scoring is exact-span F1 in the CoNLL sense: a prediction counts only when type, start, and end all match gold. The report also counts misses, wrong types, shifted spans, extras, and leakage. Exact match is strict by design, because a one-character shift breaks the mask.

The harness scores a preset on a pinned public PII corpus. A corpus is a repository id, a commit revision, a byte count, and a sha256, so a run replays exactly. Two corpora ship:

Key	Corpus	License	Role
nemotron-pii	nvidia/Nemotron-PII test split	CC-BY-4.0	In domain: the en-closed checkpoint trains on this distribution
gretel-finance-en	gretelai/synthetic_pii_finance_multilingual English test split	Apache-2.0	Out of domain: finance documents and SWIFT messages

make install                          # adds pyarrow for the eval extra
make eval CORPUS=nemotron-pii LIMIT=2000
make eval-model CORPUS=nemotron-pii LIMIT=500     # needs .[hf]

python -m edgepii.eval.cli --preset rules --corpus nemotron-pii --limit 2000 --print

Each external gold label folds into taxonomy 2026.11, or it drops with a written reason. A label that neither map names is counted as unknown, never guessed. Public corpora lists every fold and every drop.

Measured on this host, 4 November 2026. The rules lane finds no name, so PERSON and ADDRESS are zero for that lane by construction. Those two types need a model lane.

Corpus	Preset	Docs	Gold	P	R	F1	F2	Leaks
nemotron-pii	rules	2 000	5 890	0.653	0.266	0.378	0.301	589
nemotron-pii	en-closed	500	1 343	0.710	0.987	0.826	0.916	2
gretel-finance-en	rules	2 891	7 397	0.260	0.119	0.163	0.133	424
gretel-finance-en	en-closed	400	982	0.093	0.255	0.136	0.189	38

Per-type F1 tells the same story, and one row is the reason this harness exists.

Type	en-closed on nemotron-pii	en-closed on gretel-finance-en
PERSON	0.995	0.048
GOV_ID	1.000	0.321
ADDRESS	0.383	0.123
PAYMENT	0.691	0.232
SECRET	0.891	0.291

The card for antoniogr7/pii-identifier-v0.1 reports F1 0.9628 on the Nemotron-PII test split. This run reports 0.826 on 500 documents with the taxonomy folded to eleven public types, so the two numbers are not the same measure: the card scores 55 labels on its own splits. The drop that matters is the other column. PERSON falls from 0.995 to 0.048 on a finance document, because the checkpoint learned a persona template and not a grammar of names. A card figure alone would have shipped that lane.

from edgepii.eval.score import exact_span_f1, leakage_count

report = exact_span_f1(gold=gold_rows, pred=[(s.type, s.start, s.end) for s in result.spans])
print(report.f1, report.miss, report.shift, report.extra)

The scorer compares one document at a time and then sums. A pooled list would let a prediction in one document match gold in another. tests/test_eval_corpora.py locks that case.

Numeric ship floors stay UNCONFIRMED until the sealed freeze exists. Model card scores are not gates. See the evaluation spec and the public corpora.

Design rules

1. One public unit: characters on the source string. BIO and BILOU exist only inside adapters.
2. The taxonomy is data. One YAML file, one version string, one label map.
3. Rules sit beside models. They own structured forms that a standard defines. They never own names.
4. A rule reports 1.0 only when a validator confirms the slice. A shape match reports null. No rule invents a probability.
5. The locator drops a bad offset and counts it. It never searches for a repair.
6. Secrets never enter a vault.
7. import edgepii loads no ML framework. The model imports stay in detector modules.
8. Each preset pins a revision. A floating main is not a preset.
9. The library ships no score floor. min_score is caller policy, measured by the caller.
10. A threshold, a window, or a rank must name its source. A number with no source does not ship.

The full argument, with diagrams, is in first principles, architecture, and AI engineering.

Every rule names its source

A rule earns its place only when a published standard or a named parser decides the answer. This table is the audit record, and tests/test_principles.py locks it.

Rule	Decision made by	Score
Email	RFC 5321 section 4.5.3.1 local part and path limits, dotted domain	null
Phone	libphonenumber is_valid_number, or a leading + with a plan-correct length	1.0 or null
Card	ISO/IEC 7812-1 leading digit, 8 to 19 digits, Luhn	1.0
IBAN	ISO 13616 structure, ISO 7064 MOD-97-10	1.0
IPv4	RFC 3986 section 3.2.2 text form, ipaddress.ip_address	1.0
IPv6	RFC 4291 section 2.2 text form, ipaddress.ip_address	1.0
Secret	Cue word plus an assignment, or a mention whose value holds a digit or a symbol	null
Score floor	The caller, on the caller's own data	no default
Window	The checkpoint max_position_embeddings, at 75 percent, rounded down	derived
Merge rank	The taxonomy class, as the taxonomy records it	data

The audit removed three rules because no source backed them. The first was a fixed 0.80 score for shape matches. The second was per-class score floors of 0.30, 0.40, and 0.60. The third was a "cut in the second half" window rule.

The audit also fixed two rule bugs. A local subscriber number such as 48291 no longer counts as a phone. A card pattern no longer matches inside an IBAN.

Limits

• The rule lane does not find person names. Use a model lane for PERSON.
• English is the tested closed lane. Chinese works through zh-schema and the rule lane.
• The demo caps a source at 20,000 characters. The library splits long text into windows.
• en-bert needs Hugging Face access to ab-ai/pii_model.
• phi-open emits health types that taxonomy 2026.11 does not carry. The locator drops those labels. It does not rename them.
• A model can still miss a span. The measurement in evaluation is how that miss becomes visible.
• Do not send text to a cloud PII API in v1. This library does not.

License

The library is Apache-2.0. Each checkpoint keeps its own license. Check the model card before you ship a lane.

Origin notes, Ideas Lab:

• Locate PII spans in text
• Hide names and phones before the AI sees them
• Span pipeline