hyperscale-pii-scanner

Deterministic, in-process scan for residual personal identifiers in UK English text. Built on Microsoft Presidio with spaCy as the NER engine, with UK-specific recognisers, a UK place-name gazetteer, pinned versions and a small, stable interface.

It exists as a second line of defence for apps that accept documents which are supposed to have been anonymised before upload. The app scans the text, holds the upload while a person replaces or confirms each hit, and only then passes the content on.

This library is best effort. It reports what it found. It never certifies that text is anonymised, and no caller should describe its output that way.

What it detects

What it does not detect

• Anything in a language other than English.
• Bare years, durations ("three weeks"), times.
• Nationality, religion or political group (spaCy's NORP label is ignored).
• Credit cards, bank accounts, IBANs, IP addresses, crypto wallets, UK passport or driving licence numbers, vehicle registrations. Presidio has recognisers for these; they are deliberately not registered.
• Ofcom's reserved fictional phone ranges (07700 900xxx and similar) are not valid numbers and are not reported.
• Names, places and organisations the model does not recognise. NER is statistical. Short alphanumeric tokens and role titles ("GP Partner") are sometimes reported as organisations; a bare surname in a salutation is sometimes missed.
• Place names that are also common words (Street, Hope, Reading, Bath, Wells) or common personal names (Keith, Barry) are excluded from the gazetteer to avoid flagging ordinary prose; the model still reports them when the context makes them a place.

Installation

uv add hyperscale-pii-scanner                # core
uv add "hyperscale-pii-scanner[documents]"   # plus Word and Excel extraction

Python 3.14 or later. The wheel includes the spaCy en_core_web_md model, so nothing is downloaded at install or run time.

Interface

from hyperscale.pii_scanner import Hit, HitKind, redact, scan, scanner_version

hits: list[Hit] = scan(
    text,
    allowlist=["Dr Ian Brown", "IDB Experts Ltd"],
    placeholder_patterns=[r"\b[A-Z]{3,}\d[A-Z]{1,3}\b", r"\([A-Z][A-Z ]+\d?\)"],
    language="en",
)
for hit in hits:
    print(hit.kind, hit.text, hit.start, hit.end, hit.score, hit.recogniser)

redacted = redact(text, hits, token_for=lambda hit: f"[{hit.kind.value.upper()}]")

str(scanner_version())
# 'hyperscale-pii-scanner 0.1.0; presidio-analyzer 2.2.364; spacy 3.8.16; en_core_web_md 3.8.0'

• scan returns hits sorted by position, non-overlapping, each with a score of at least 0.3. Hit is a frozen dataclass; text is always source[start:end].
• HitKind is a closed StrEnum with the thirteen members in the table above.
• redact replaces each hit with a token ([KIND] by default). It raises ValueError if a hit does not line up with the text, which catches hits from a different document.
• scanner_version() returns the library, Presidio, spaCy and model versions for audit logs. It does not load the model.
• language accepts only "en". Any other value raises ValueError.
• allowlist and placeholder_patterns must be iterables of strings; passing a single string raises TypeError rather than silently iterating its characters.
• The model loads once per process on the first non-empty scan (about 1.5 s). scan is safe to call from multiple threads; calls are serialised on a lock, so use processes for parallelism.
• Results are deterministic for the same input, configuration, library versions and CPU architecture.

How overlapping hits are resolved

Presidio can return several results for one span (an NHS number also matches as a phone number; the model may label a town as an organisation). The library keeps one hit per span: a structured kind (date of birth, NHS number, NINO, postcode, case reference, email, URL, phone, age, date, in that order) beats a model kind; otherwise the longer span wins, then the higher score. An exact gazetteer match of a UK place name outranks a same-span model label.

Allowlist and placeholders

Suppression runs after recognition, so it applies the same way whichever recogniser produced a hit. The library finds every whole-word, case-insensitive occurrence of each allowlist entry and every match of each placeholder pattern (a Python regular expression). A hit that touches one of those spans is dropped when every token containing a capital letter or a digit lies inside such a span.

In practice: Ian Brown reported out of Dr Ian Brown is dropped, LOCAL AUTHORITY out of (LOCAL AUTHORITY) is dropped, and a model span that glued several placeholders together with "and" is dropped. A hit that extends beyond a placeholder with a capitalised word (MOTHER2FN Jones) is reported, because the extra text may be a leak. Invalid patterns raise ValueError before scanning.

Document extraction (optional)

from hyperscale.pii_scanner.documents import extract_text

doc = extract_text(path_or_bytes, kind="docx")  # or "xlsx"
hits = scan(doc.text)
for hit in hits:
    print(
        doc.locate(hit.start), hit.kind, hit.text
    )  # "paragraph 14", "sheet Results, cell B20"

Word: every paragraph in the body in document order, including paragraphs inside tables, nested tables and content controls (labelled by their innermost table, row and cell), then each section's headers and footers, including first-page and even-page variants and tables inside them. A header linked to the previous section is read once. Footnotes, endnotes, comments and text boxes are not read. Excel: every non-empty cell, sheet by sheet, using cached values rather than formulas. Files are untrusted input: a 50 MB size cap, a declared-uncompressed-size cap of 250 MB checked before parsing, XML entity resolution disabled in python-docx, an entity-safe parser plus defusedxml in openpyxl. Parsing failures raise ValueError.

Measured behaviour

Measured on 2026-10-12 with make measure against the synthetic fixture corpus in tests/fixtures/corpus/ (five documents, written for this project). The figures describe that corpus, not real documents.

The false positives are all model (spaCy) labels: "NHS", "GP", "GP Partner", "the Family Court", "the Crown Court" as organisations, a first name as a location, and the author's role glued to an allowlisted organisation. Pattern kinds produced no false positives on this corpus.

Timing and memory, Apple M-series laptop, model already loaded:

The test suite asserts the scan stays under 5 s locally and 10 s on the slower CI runner.

Scan time grows with the number of hits and, for spaCy's tokeniser, with pathological punctuation runs (thousands of consecutive brackets). Scans serialise on one lock per process, so cap upload sizes in the calling app and scan large spreadsheets sheet by sheet rather than as one text.

Configuration

There is one configuration point, for the spaCy model:

from hyperscale.pii_scanner import configure

configure(model="en_core_web_lg")  # a package name, or a path to a model directory

Call it before the first scan (for example in a Django AppConfig.ready()); it raises RuntimeError afterwards. A package name that is not installed raises RuntimeError at the first scan; the library never downloads a model. The large model is about 400 MB and is not on PyPI; install it with uv add https://github.com/explosion/spacy-models/releases/download/en_core_web_lg-3.8.0/en_core_web_lg-3.8.0-py3-none-any.whl.

Maintenance

Upgrading the model

1. Change MODEL_VERSION, WHEEL_URL and WHEEL_SHA256 in scripts/vendor_model.py (compute the digest with shasum -a 256 on the downloaded wheel).
2. Change DEFAULT_MODEL_VERSION in src/hyperscale/pii_scanner/_engine.py, the MODEL_META path in the Makefile, and the wheel-content check in ci.yml.
3. python3 scripts/vendor_model.py --force, then make check and make measure.
4. Update the measured numbers above and the CHANGELOG.

Adding a recogniser

1. Add a module under src/hyperscale/pii_scanner/_recognisers/ that subclasses Presidio's PatternRecognizer or LocalRecognizer and produces a new entity type. Presidio compiles patterns case-insensitively; wrap case-sensitive patterns in (?-i:...).
2. Add the entity type to ENTITY_TO_KIND in _postprocess.py and, if it is a structured kind, to STRUCTURED_PRIORITY. Add the kind to HitKind if it is new.
3. Register it in build_recognisers() in _recognisers/__init__.py.
4. Add positive and negative unit tests, plant an example in the corpus, and re-run make measure.

Refreshing the gazetteer

make gazetteer downloads the GeoNames GB extract, applies the rules in scripts/build_gazetteer.py, and rewrites data/uk_places.txt. Review the "dropped by frequency" list it prints; add genuine places to scripts/gazetteer_extra.txt and common-word collisions to scripts/gazetteer_stoplist.txt. The data file header records the extract date.

Licences

Release process

make bump PART=minor   # bump version, promote the CHANGELOG [Unreleased] section, commit
make release           # tag v<version>, push, create the GitHub release; publish.yml uploads to PyPI

Publishing uses PyPI trusted publishing from the pypi GitHub environment; no API tokens.