profilefoundry 1.0.0

An open-source generator for structured, internally-consistent, linked, temporally coherent synthetic Person Objects.

ProfileFoundry

A structured, internally-consistent, linked, temporally coherent synthetic Person Object - and an open-source generator that produces them at scale.

This repository is the implementation home for ProfileFoundry, which builds on the problem setting exposed by PANORAMA (Selvam & Ghosh, 2025). It provides a deterministic generator, schema, validation suite, release reports, and Hugging Face export tooling for the ProfileFoundry-Core-100K dataset.

The ACL/NeurIPS-style paper draft lives in Paper/. The vetted 100K release package is staged for Hugging Face at srirxml/ProfileFoundry-Core-100K; run python scripts/verify_hf_release_current.py with an HF_TOKEN before calling the remote artifact current. The default HF viewer table is person_objects.parquet, a complete one-row-per-person view with nested sections encoded as JSON strings; the remaining parquet files expose the normalized relational schema.

Status (2026-05-24) — v1.0 release sprint

Phase	Status
0 — Foundation (schema and package scaffolding)	✅ complete
1 — Reference data (manifest + bootstrap + loader)	✅ ref data shipped for 5 validation locales
2 — Generator factory (port + joint constraints + age-gating)	✅ deterministic across processes (set-iteration bug fixed)
3 — Linkage (households, employers, family graph)	✅ household orchestrator + family edges + employer pool
4 — Temporal (event taxonomy, backfill, replay)	✅ replay-valid non-credit timeline + typed payload export
5a — Validation (KS gaps, consistency)	✅ disclosed KS gaps (max 0.124) · 100% consistency
5b — Leakage audit (Wikidata Bloom, reserved-domain email audit, self-collision)	✅ Wikidata Bloom + self-collision + reserved-domain email syntax audit
6 — Scale + publish (100K + HF export)	✅ 100K release · complete viewer table · normalized parquet star schema · MANIFEST.json
7 — Reproducibility (verify fixture, manifest hash)	✅ normalized fixture stable across processes; verify script + fixture
8 — Paper draft	✅ submission draft revised
9 — HF push + tag v1.0.0	⏳ HF upload target documented; remote current check requires HF_TOKEN; tag pending

Quick start

pip install -e ".[dev]"
profilefoundry verify                # smoke-test all 8 locales
profilefoundry person --locale US    # one US Person as JSON
profilefoundry household --locale US # one linked household (multiple Persons)
profilefoundry scale --n 1000 --locale UK --out /tmp/uk.jsonl
profilefoundry scale-households --n 500 --locale CA --out /tmp/ca_households.jsonl
profilefoundry validate --n 300                     # KS gaps + leakage + consistency
profilefoundry export --out /tmp/pf_core --n-per-locale 1000 --generation-date 2026-05-24 --exported-at 2026-05-24 --skip-hibp  # normalized parquet star schema
profilefoundry scale-smoke --sizes 1000,10000,100000           # timings

# Full v1.0 release run (100K profiles in ~4-5 minutes on a laptop):
python scripts/run_full_core.py --generation-date 2026-05-24 --skip-hibp --verbose
# Reproducibility check:
python scripts/verify_reproducibility.py
# Hugging Face upload preflight / push:
python scripts/push_to_hf.py --dry-run
python scripts/push_to_hf.py --repo srirxml/ProfileFoundry-Core-100K
# Confirm the public HF manifest matches the vetted local release:
python scripts/verify_hf_release_current.py
# Build the Wikidata bloom (multi-hour due to WDQS rate limit):
python scripts/ingest_wikidata_persons.py --scope sample --sleep 65 --verbose

Or without installing:

PYTHONPATH=src python3 -m profilefoundry.cli scale --n 1000 --locale US --out /tmp/us.jsonl

CLI reference

All commands are available through profilefoundry after installation, or via PYTHONPATH=src python3 -m profilefoundry.cli from a checkout.

Command	Purpose	Common example
profilefoundry verify	Generate one profile per supported locale as a smoke test.	profilefoundry verify
profilefoundry person	Print one deterministic Person Object as JSON.	profilefoundry person --locale US --seed 4321 --profile-seq 1
profilefoundry household	Print one linked household as JSON.	profilefoundry household --locale UK --seed 4321 --seq 7
profilefoundry scale	Generate flat JSONL profiles for one locale.	profilefoundry scale --n 1000 --locale CA --out /tmp/ca.jsonl
profilefoundry scale-households	Generate linked household members as JSONL.	profilefoundry scale-households --n 500 --locale AU --out /tmp/au_households.jsonl
profilefoundry validate	Run distributional, leakage, replay, and consistency checks.	profilefoundry validate --n 300 --locales US,UK,IN --skip-hibp
profilefoundry export	Write JSONL, parquet tables, manifest, and dataset card.	profilefoundry export --out /tmp/pf_core --n-per-locale 1000 --generation-date 2026-05-24 --exported-at 2026-05-24T00:00:00Z --skip-hibp
profilefoundry scale-smoke	Time the generator at several sizes.	profilefoundry scale-smoke --sizes 1000,10000,100000

Script-level release utilities:

Script	Purpose
python scripts/run_full_core.py --generation-date 2026-05-24 --skip-hibp --verbose	Rebuild the 100K release artifact under data/raw/profilefoundry-core-v1/ and reports under reports/v1_release/.
python scripts/verify_reproducibility.py	Confirm the pinned reproducibility fixture still matches.
python scripts/verify_package_reference_data.py	Confirm reference data is available from an installed wheel.
python scripts/push_to_hf.py --dry-run	Preview the Hugging Face upload.
python scripts/push_to_hf.py --repo srirxml/ProfileFoundry-Core-100K	Upload the vetted release artifact.
python scripts/verify_hf_release_current.py	Compare the local release manifest with the Hugging Face dataset. Requires HF_TOKEN for private or gated repos.
python scripts/export_schema.py --check	Verify the checked-in JSON Schema matches the Pydantic model.
python scripts/ingest_us_acs.py / python scripts/ingest_uk_ons.py	Refresh reference-data inputs from upstream sources.

Project layout

.
├── README.md                  # this file
├── pyproject.toml             # package metadata, deps, ruff/pytest config
├── notes/
│   ├── BUGS.tsv               # resolved/known issue ledger
│   └── RELEASE_PLAN.md        # v1.0 release plan
├── schemas/
│   ├── README.md
│   └── person_v0_1.schema.json  # auto-exported JSON Schema
├── src/profilefoundry/
│   ├── schema/                # Pydantic Person Object v0.1 (source of truth)
│   ├── data/                  # reference-data loader (bootstrap ↔ derived)
│   ├── generate/              # factory + sampling + locale providers
│   ├── linkage/               # (phase 3) households, employers, families
│   ├── validate/              # (phase 5) invariants, distributions, leakage
│   ├── io/                    # (phase 6) Hugging Face export
│   └── cli.py                 # `profilefoundry` entry point
├── data/reference/
│   ├── MANIFEST.md            # every external source enumerated
│   └── bootstrap/             # committed minimum reference marginals
├── scripts/
│   ├── export_schema.py       # regenerate JSON Schema
│   └── ingest_us_acs.py       # pull richer ACS tables live (needs API key)
├── tests/                     # pytest invariants & unit tests
└── Paper/                     # ACL-style paper draft + figures

Testing

PYTHONPATH=src python3 -m pytest tests/

The full pytest suite covers schema invariants (Pydantic model), reference-data loading, factory smoke checks, determinism, plausibility (age, employment, addresses, education), adults-only enforcement (no profiles under 18), linkage (households, employers, family graph), validation (KS gaps, replay, consistency, leakage), release-report freshness, normalized export quality, and reproducibility.

License

• Code and SDK: ProfileFoundry Citation License 1.0 (see LICENSE). Public uses and redistributions must cite the ProfileFoundry paper when available, or this repository until then. Machine-readable citation metadata is in CITATION.cff.
• Generated dataset: CC-BY-4.0.
• Embedded reference data retains its upstream license; see data/reference/MANIFEST.md.