Sieve Crowd v0.5.0

English | 日本語

Table data anonymization and privacy protection engine

Sieve Crowd is a table-data processing engine that suppresses the risk of identifying individuals in tabular data, so that the data can be shared safely. Building on the principles of k-anonymity and l-diversity, it deterministically performs attribute rounding (generalization) and partial record removal (suppression).

A fully local, zero-dependency, deterministic (100% reproducible) mathematical transformation engine that anonymizes tabular data and disables re-identification. It extends the design philosophy of Sieve-Core into the tabular-data domain.

💡 Concept and design philosophy

Even when personal data is kept safely on your own machine, you sometimes have to provide or publish tabular data that contains it — or you face the risk of leakage. From a statistical standpoint, the combination of attributes such as age, postal code and sex always carries a risk of identifying individuals.

Consider, for example, a dataset of about 600 people. The risk structure becomes clear:

• Before: 574 of the 600 people (96%) were "alone" on the combination of age, postal code and sex — no one else shared their exact values, so an outsider could single them out easily.
• How the transformation works: rather than leaving these isolated records as they are, the engine groups similar records together (by rounding and generalization) so that everyone is hidden inside a safe crowd.
• After: every person belongs to a group of at least 5 (4 look-alikes besides themselves). Even an outsider who tries to identify someone can be right at most 1 in 5 times (20%).

Sieve Crowd does not rely on probabilistic inference or black-box AI. It transforms tabular data into a state in which no one stands out, deterministically, on the basis of the rigorous mathematical models of k-anonymity and l-diversity.

• Deterministic approach: the same input and settings always produce the same output. The result never depends on the execution environment, the clock, or random numbers.
• A measurement, not a promise: after the transformation, the finished table is re-counted to check whether it meets the target safety level. If it falls short, the reason is given on a receipt and the table is not written.
• Explainability: a receipt records the data processing performed for anonymization, so that what was done can be fully traced.

🧩 Sieve series

Sieve is a family of engines that share three design principles — deterministic, zero-dependency, fully local. Each engine handles one observation or one removal, and they connect to each other:

🔥 Main features

• k-anonymity and l-diversity Attributes are generalized so that each record has at least k−1 look-alikes, and the diversity of sensitive values is also guaranteed.
• Zero dependencies, fully local Implemented with the Python standard library only. Data never leaves your computer (no account, offline, no upload).
• Minimal generalization and suppression Data is not rounded more than necessary; resolution is preserved as much as possible while grouping. Records that still cannot form a group are safely removed (suppressed) rather than distorted.
• Pre-flight plan sieve-crowd plan reports, before any search, the exact search-space size (lattice nodes), the suppression budget, the level-0 risk profile and warnings (unreachable k/l, tight limits, many isolated records) — plus, since v0.6.0, the derived keep-vs-qi relations (does a kept column nearly determine a quasi-identifier, in either direction) — so tuning happens before the run, not after.
• Diagnostics in the receipt When records are suppressed, the receipt explains why — crowd sizes, "smaller than k" vs "short of diversity", and a per-column hint of which quasi-identifier one more generalization step would rescue. In aggregates only; never a cell value.
• Custom bins for numeric columns Non-uniform classifications such as 0-19, 20-39, 40-64, 65+ (legal age brackets, income bands) can be given directly as bin edges.
• A policy for missing values missing_value_policy decides what happens to records whose quasi-identifier is empty: star (default) generalizes the value to *; suppress removes the record up front, inside the same counted suppression budget (and refuses with missing_exceeds_budget when they exceed it). No placeholder option — filling values in would turn the missing pattern itself into an identifier.
• Receipt for the record A machine-readable receipt records what was done (rules applied, suppression count, measurements). The receipt contains no personal information.
• Supported formats CSV, JSON and XML (UTF-8). Excel is excluded to keep the tool dependency-free.

🎯 Use cases

• Keeping data on hand for internal use or in-house sharing, stored and managed safely
• Providing survey or customer data to a university, agency, or analytics contractor
• Publishing statistics or open data from hospitals, schools, or local governments
• Feeding realistic test data into a development environment, safely
• Keeping a record of anonymization and processing for legal or audit purposes

📦 Install

pip install git+https://github.com/neguseatama/sieve-crowd.git

Or from a local clone:

git clone https://github.com/neguseatama/sieve-crowd.git
cd sieve-crowd
pip install .

💻 Quick start

1. Inspect the structure

Reads the file and writes a configuration file that proposes a role for each column (name, quasi-identifier, sensitive value, etc.).

sieve-crowd inspect people.csv --config-out config.json

2. Review and edit the configuration

Edit the generated config.json to define the role of each column.

3. Preview the plan (optional)

Before running, check the size of the search, the suppression budget and the level-0 risk profile — no search is performed, nothing is written:

sieve-crowd plan people.csv -c config.json

4. Run anonymization

sieve-crowd run people.csv -c config.json -o people_safe.csv --report-json receipt.json

Input and output example:

people.csv                                       people_safe.csv   (rows shuffled)
name,birth_date,age,zip,sex,diagnosis            age,zip,sex,diagnosis
Takahashi Hina,1986-11-02,40,1500052,M,none     20-24,15100**,F,cold
Kobayashi Haruto,2003-01-03,23,1000042,F,flu    20-24,16000**,M,none
Yamamoto Aoi,1971-10-04,55,1510061,M,flu        55-59,15100**,M,asthma

🧪 Hint packs (built-in on by default)

inspect and plan load hint packs — JSON files that suggest a role and a generalizer for a column and check value shapes, without ever touching the anonymization. Since v0.5.1 the built-in jp-basic pack is loaded by default; --no-builtin-hints turns it off. run never reads a pack: the run path stays byte-for-byte unchanged from v0.4.0 (pass --no-builtin-hints and give no pack, and even the inspect template is the exact v0.4.0 bytes).

sieve-crowd inspect people.csv --config-out config.json              # builtin jp-basic included
sieve-crowd plan   people.csv -c config.json    --findings-json f.json
sieve-crowd inspect people.csv --hint-pack my.json                   # builtin + my.json
sieve-crowd inspect people.csv --config-out t.json --no-builtin-hints  # v0.4.0 byte-exact route

• --hint-pack FILE — load a pack in addition to the built-in (may be given more than once). Load order: the builtin first, then the command-line order; findings.packs / pack_info follow that order. The candidate order is its own contract (v0.5.2), independent of load order and of the order rules are written in a pack: evidence strength (name+value → value → name → stat), then role (id → sensitive → qi, roles outside this list last), then rule id ascending. _suggest follows the first sorted candidate.
• --no-builtin-hints — exclude jp-basic (use only the packs you named; with none named, nothing is loaded and the v0.4.0 byte-exact output survives).
• --findings-json FILE — write the findings as JSON: aggregates and column names only, never a cell value. packs stays a list of pack ids; pack_info adds {id, version, source, sha256} per pack in load order. On plan the file also carries the derived result (derived_summary, v0.6.0) even when no pack is loaded.

Ids are unique across all packs, the builtin included. A user pack reusing jp-basic (or a rule id like jp-phone) is refused with exit code 2; to replace the builtin, give --no-builtin-hints and then reuse its id.

A pack (schema: 1, read as utf-8-sig; optional version string) lists rules. Each rule has a when (matching the column by column_names and/or value_shape), an optional suggest (role + generalizer settings, written into the template under _suggest for that column only), an optional check (set / range / luhn / my_number / email_shape, each three-valued: match / mismatch / cannot-check), and an optional leak flag. The value_shape language is bounded on purpose: # one digit, @ one ASCII letter, \X a literal escape, only {m} / {m,n} quantifiers — no alternation, no nesting, so it cannot describe a catastrophic-backtracking pattern.

Leak checks (v0.5.1). A leak: true rule scans each scanned column for identifier-shaped values as tokens: a cell is split on whitespace and punctuation (dashes, +, @, ., _, % stay inside a token), so a number inside prose is found (03-1234-5678 is also tried without its dashes). Space-separated pieces never combine — 03 1234 5678 is not detected (documented constraint). Tokens are runs of ASCII alphanumerics plus the gluing symbols, so full-width digits and dashes (０９０－１２３４－５６７８) never form a token and are not detected as-is (same kind of documented constraint). Cells longer than 4096 characters are skipped and counted as skipped. On plan only the columns whose declared role is keep, sensitive or qi are scanned (id/drop are never scanned); inspect has no roles and records every column. Counts are reported as leaks[] (count, checked, skipped — plus declared_role on the column, plan only) and, on plan, a warning line on stderr and a code: "leak" element in findings.warnings when a count is 1 or more; inspect prints one candidates-format line instead. The numbers are possibilities, not proof, and warnings never change the exit code.

Derived keep-vs-qi relations (v0.6.0, plan only). For every (keep column X, qi column Q) pair, evaluated in both directions, plan reports when one column is a near-perfect proxy for the other: reveals_qi (X is kept verbatim in the output, so generalizing Q does not hide it — the attacker reads Q off X) and implied_by_qi (the qi value nearly determines the kept value). Every evaluated direction lands in exactly one of four classes: unchecked (fewer than 50 compared records), trivial (a constant column on either side, or every source value unique — the relation carries no information), flagged (the support s is at least 50 and the violations v — records the source value does not pin down to one target value — are at most 1% of s and at most 10% of the excess s−m, where m is the number of distinct source values), and clear (checked, but too many violations). Flagged directions appear as - derived: … lines on stderr (after the hint-pack findings, without a heading of their own) and as code: "derived" elements in findings.warnings (column, declared_role, qi_column, relation, support, violations); the top-level derived_summary counts all four classes and satisfies unchecked + trivial + flagged + clear = 2 × (pairs − skipped_pairs). Values are compared with the engine's own normalization (NFC + strip) and empty cells are excluded. At most 400 pairs are evaluated per plan (skipped_pairs counts the rest). --no-derived skips the whole report; thresholds are code constants, not settings. Column names and counts only — never a cell value.

The built-in jp-basic pack (version 1.1.0) covers Japanese postal codes (prefix length 4), birth dates (#{4}-#{2}-#{2} → suggest qi with type: date; added in v0.6.0), phone / My Number / email / card-number leak checks (Luhn and the official My Number residue-11 check digit), the 47 prefectures as a category hierarchy mapped to the eight-block 地方ブロック grouping (三重 = 近畿, 新潟 = 中部 — one standard fixed inside the pack), plausible ages, and three observation warnings: role_mismatch (a column suggested qi but declared keep / sensitive), mixed_form (full/half-width forms mixed in a column — NFC keeps them as separate crowds), and japanese_era (values in a 和暦 form). Edit a pack to change a standard; hints never affect a run.

The packs/ directory (v0.6.0; municipality edition since v0.6.2). Hint packs that ship separately from the wheel live here — nothing in packs/ or tools/ is packaged. The municipality edition of jp-municipality-2026 (pack 1.1.0) ships with the repository: the ABR prefecture extract packs/mt_pref_all.csv (47 rows), the 総務省 municipality extract packs/soumu_city_r611.csv (1,747 rows, basis date R6.1.1 = 2024-01-01) and the generated pack packs/jp-municipality-2026.json (load with --hint-pack packs/jp-municipality-2026.json), which carries 1,794 codes, 1,794 five-digit prefixes, 47 prefecture names and 1,714 distinct municipality names in four rules. Both sources follow the 公共データ利用規約（第1.0版） — PDL 1.0 (the 総務省 applies the same terms to its site content) — and the required source citations and edited-data statements are in packs/README.md. The generator tools/make_municipality_pack.py (standard library only) auto-detects the input contract from the header — the prefecture master (lg_code / pref) or the 総務省 list (団体コード / 都道府県名 / 市区町村名) — and merges one of each (either order) into the four-rule pack; two inputs of the same contract, or more than two, exit with code 2. It is deterministic (byte-identical output for the same input, independent of the input order), and contract violations exit with code 2 naming the columns. Regenerating from the two committed extracts with --pack-version 1.1.0 reproduces the committed bytes (T29). The town-level master (156 MB) and the 総務省 政令指定都市 sheet are not redistributed.

⚙️ Settings and options

Main command-line options:

• --dry-run: do not write data; only measure anonymization and assess risk
• sieve-crowd plan people.csv -c config.json: pre-flight estimate of the search space, budget and warnings, including the derived keep-vs-qi relations (--no-derived skips them; no search, no output)
• --lang ja: issue a Japanese receipt (default is English; JSON output is also available)
• --in-format / --out-format: explicitly set the input/output format (csv, json, xml)
• sieve-crowd keygen -o key.txt: generate a cryptographic key for pseudonymization

Exit codes:

• 0: success
• 1: I/O error
• 2: configuration error
• 3: refused (the target anonymization level could not be reached; no output written)

🔬 Tests

python -m unittest discover -s tests -v

All test suites run with no external dependencies (standard library only).

The suite also carries a sha256-pinned replica of the Sieve Schema Contract v1 conformance corpus (44 accept/reject cases, maintained in the sieve-lens repository, which hosts the contract text docs/SCHEMA_CONTRACT_v1.md and the canonical engine sieve_lens_ext/schema.py). The replica exercises Crowd's JSON core (sieve_crowd.jsonio): duplicate keys and NaN/Infinity are refused. The UTF-8 BOM case is a declared divergence — Crowd decodes with utf-8-sig on purpose (spreadsheet exports carry a BOM), so the runner pins the acceptance instead of the refusal, as declared in the contract's conformance matrix (§6).

⚠️ Known limitations and caveats

1. Model assumption — Sieve Crowd protects against an attacker who knows the attributes you declared as qi (quasi-identifiers). It is not a universal shield against re-identification through auxiliary information you did not anticipate.
2. Not a substitute for legal or privacy review — It does not guarantee legal or regulatory compliance under privacy laws or guidelines; it is a tool that supports such review technically.
3. Excel is not supported — To keep the dependency-free principle, the binary .xlsx format is out of scope (use CSV / JSON / XML).
4. Row shuffling — To minimize re-identification risk, the output row order is deterministically shuffled.

📄 License

MIT License. See LICENSE for details.

👤 Author

• Kai IWASAKI