privyscope-ja

Japanese language pack for privyscope, a two-stage hybrid PII detection & masking engine.

This is a thin plugin: it ships only the Japanese data (regex_rules.yaml, entity_config.yaml) and registers itself with the privyscope core via the privyscope.languages entry point. The engine and the privyscope command live in the core, which is pulled in automatically.

Japanese mixes three scripts — kanji (漢字), hiragana (ひらがな) and katakana (カタカナ) — with no inter-word spaces, so the pack advertises all three scripts and leaves personal-name detection to the NER stage rather than regex.

Install

pip install privyscope-ja      # pulls in the `privyscope` core automatically

Quickstart

privyscope redact "田中さんの電話は090-1234-5678、メールは taro@example.co.jp"
# → "<PER>さんの電話は<PHONE>、メールは <EMAIL>"

from privyscope import Privyscope
eng = Privyscope.from_pretrained(lang="ja")
eng.redact("田中さんの電話は090-1234-5678").masked_text   # "<PER>さんの電話は<PHONE>"

How it works

A two-stage hybrid pipeline, results merged via union:

1. Regex filter — structurally obvious PII: Japanese mobile (0[789]0-xxxx-xxxx, +81), landlines, My Number (個人番号, 12-digit with check digit), passport, driver's license, Corporate Number (法人番号), Japanese bank accounts, email, and credential secrets.
2. ONNX NER — a BIOES token classifier with a constrained Viterbi decoder for contextual PII (names, addresses, private dates). Weights download from Hugging Face Hub (zafrem/privyscope-ja) on first use.

Inference is ONNX Runtime only — no PyTorch at runtime. The base encoder tokenizes with MeCab, but that dependency is needed only for training/fine-tuning.

Entities

8 base types — PER, PHONE, ID_NUM, EMAIL, LOC, BANK, DATE, SECRET — plus extended BIZ_NUM (Corporate Number 法人番号) and DRIVER_LICENSE, caught by regex.

Model & performance

• Base encoder — tohoku-nlp/bert-base-japanese-v3 → BIOES head → Viterbi decoder.
• Runtime artifact — INT8-quantized ONNX, max sequence length 256, downloaded from the Hub with a SHA-256 checksum.txt for integrity.
• Accuracy — run privyscope eval --lang ja your_val.jsonl on your own labelled data; numbers will be published here once the held-out evaluation is finalized.

Limitations

• Not an anonymization/compliance guarantee; use as one layer of privacy-by-design, with human review for sensitive workflows.
• Personal names are contextual (NER only): recall depends on the fine-tuned model, which is trained across kanji/hiragana/katakana renderings.

License

Apache-2.0. See LICENSE.

Stage-1 regex patterns are compiled from the pii-pattern-engine ruleset. Most rules carry a verification function — a checksum or dictionary validator (jp_my_number_valid, luhn) that a match must pass before it is redacted, so a value that merely looks like an ID is left alone.

Regex-only extended entities: BIZ_NUM · DRIVER_LICENSE · CRYPTO · IP · DEVICE · URL.

privyscope_ja/regex_rules.yaml is generated by scripts/gen_regex_rules.py and is overwritten on every build — edit the mapping in that script, not the YAML. See CONTRIBUTING.