Sieve Redact v0.5

A command-line tool that removes sensitive information from documents — deterministically, while preserving byte-for-byte integrity of everything else.

English | 日本語

💡 Concept

Personal information in resumes, reports and logs is converted into a second file that is safe to publish and share.

• Deterministic — Same input plus same rules always produces the same output. Character selection in noise is derived from SHA-256; it depends on nothing else — not on the environment, the clock, or any RNG.
• Byte-for-byte integrity — Bytes outside a match are left untouched, down to the last byte. CRLF line endings survive. Integrity is verified mechanically before output; --strict refuses to write on failure (exit code 3).
• No guessing — The user specifies what is sensitive. The engine never guesses.
• Receipts — The receipt records the facts of processing (positions, lengths, modes, hashes) but never the sensitive body itself. Even "no pattern remains" is reported as a measurement (pattern_appearance_in_output), not a promise.

The name refers to a sieve: drop only what is specified, and leave everything else unharmed, byte for byte.

Companion tool to Sieve Lens (observation of invisible content) — Lens observes, Redact masks.

🧩 The Sieve Series

Sieve is a family of deterministic, zero-dependency engines that share the same design principles (deterministic output, no telemetry, fully local) — each engine observes or masks one specific risk, and they compose:

Observation → masking in one pass: Lens finds what a human can't see, Redact removes it with a receipt that proves nothing else changed.

🔥 Features

Six redaction modes

Built-in matchers (--builtin)

Image masking (--region)

PNG images (8-bit RGB/RGBA, non-interlaced, non-animated) are masked by explicit regions: --region x,y,w,h:MODE[:ARG] (repeatable; overlapping regions are discarded in declaration order; zero-area or out-of-bounds regions are usage errors; --rule cannot be combined with --region).

Image receipts set byte_integrity_verified to null (not applicable — PNG re-encoding changes file bytes structurally) and report pixel integrity outside every region as pixel_integrity_verified; --strict exits 3 when it fails. File metadata (EXIF etc.) is always dropped on save — a privacy-friendly default by design. Output bytes are deterministic across platforms and pinned in CI (with Pillow 12.3.0; future Pillow updates may change encoded bytes, never pixels).

Word-unit matching (+ prefix)

太郎:label also matches the 太郎 inside 田中太郎; 太郎:+label matches only at word boundaries (adjacent characters of the same class, or both alphanumeric, cause rejection).

Config files and Lens integration

• --config rules.cfg: load rules from a file (rule <spec> / builtin <spec> line format).
• --from-lens catalog.json: generate masking rules from an observation catalog (JSON) produced by Sieve Lens. Invisible characters become deletions; look-alike letters (Cyrillic/Greek → Latin) become replacements based on the twin field.

💻 Quick start

git clone https://github.com/neguseatama/sieve-redact.git
cd sieve-redact

# Mosaic a name, label phone numbers
python sieve_redact.py in.txt -o out.txt \
  --rule "田中太郎:mosaic" \
  --builtin phone-jp:label \
  --receipt receipt.json

Example receipt (stdout) — in.txt contains x秘密y, run with --rule "秘密:delete":

Sieve Redact v0.5 — レシート
入力: 8 バイト -> 出力: 2 バイト
redact 件数: 1
  #1 rule=1 matcher=literal mode=delete pos=1..7 len=6B -> 0B sha256=062a2931da68...
他バイト完全性: 検証済み
出力内パターン残留: none

The sensitive body never appears in the receipt — hashes only. In image mode (--region) byte_integrity_verified is null (not applicable) and pixel integrity is measured by pixel_integrity_verified instead.

Integration with Sieve Lens

# (1) Lens produces an observation catalog
python -m sieve_lens_ext.redact_bridge input.txt catalog.json
# (2) Redact masks everything
python sieve_redact.py input.txt -o clean.txt --from-lens catalog.json

🔬 Determinism guarantees

Character selection in noise is fully determined by:

char = POOL[ SHA-256("<pattern>:<byte_offset>:<original char>").digest()[0] % len(POOL) ]

No seed. No environment dependence. Same input, same output, always. The bundled tests verify exact agreement with this formula by recomputation.

Overlapping matches are resolved deterministically as well: rules are tried in declaration order, each rule scans leftmost-first, later matches that overlap a confirmed region are discarded, and confirmed regions are applied in ascending order of position.

🔬 Tests

python -m unittest discover -s tests -v

113 tests (five modes, built-in matchers, word-unit matching, config files, Lens integration, CLI boundaries). No additional dependencies.

⚠️ Known limitations

1. No regex — patterns are literal, exact-match strings only
2. No case-insensitive matching — matching is case-sensitive
3. No automatic sensitive-data detection — the user specifies what is sensitive (by design)
4. Text and PNG images — PDF and Office formats are not supported; image masking is PNG-only (8-bit RGB/RGBA) and JPEG is explicitly out of scope; PDF masking is out of scope (observation is Sieve Lens's domain)
5. Full-width digits and non-ASCII hyphens are out of scope for postal/phone
6. 4-digit area codes starting with 08/09 (except 0800) and short numbers such as 117/104 are out of scope for phone-jp — match them explicitly with --rule

📄 License

MIT. See LICENSE.

👤 Author

• Kai IWASAKI