SynthPopCan

SynthPopCan is an early-stage project for building Canadian synthetic population tooling.

SynthPopCan is an independent research-software project. It is not affiliated with, endorsed by, or sponsored by Statistics Canada or the Government of Canada.

Near-term goals:

1. Provide a Python library and CLI that can create synthetic populations through IPF from Statistics Canada margin/control tables.
2. Maintain explicit 2016 and 2021 Census microdata workflows for household- and person-level synthetic populations using tree-based generation plus calibration, while keeping source adapters and metadata appropriately census-vintage-specific.
3. Maintain a local web app for configuring runs, inspecting controls, generating from prepared models, validating outputs, and downloading results.

The 0.7.0 release adds explicit geography contracts, national DA/ADA orchestration, verified prepared-display-boundary retrieval, and a reusable external-data enrichment framework for public, locally supplied, licensed, and restricted sources. Can-FED v2 and ODEF v3 are planned as contrasting area-attribute and facility reference adapters for 0.7.1 and 0.7.2; later integrations remain demand-, authority-, and validation-gated.

Detailed documentation is published at https://synthpopcan.readthedocs.io/. The source files live under docs/; start with docs/index.rst for task-based navigation to the web app, IPF from StatCan margin tables, generated-from-model workflows, the beginner Python API, and advanced microdata/model-training material.

Project planning and research notes are tracked separately:

• PLANS.md: current roadmap, open work, and release sequencing.
• adr/: accepted architecture decisions, alternatives, and consequences.
• NOTES.md: research synthesis from local materials and external literature.
• CHANGELOG.md: public release notes and completed release highlights.
• CORRECTNESS.md: tested correctness claims, evidence, limitations, and reproducibility commands.

Quick Start

These commands use a Unix-style shell. Windows readers should first complete the WSL setup in the Installation guide, then run the same commands inside the Ubuntu terminal.

Install the published package from PyPI:

Network required for installation.

python3 -m pip install synthpopcan

The base installation is enough to generate from portable frequency or CART models. Starting with 0.7.0, only researchers training new CART models need python3 -m pip install "synthpopcan[model-build]".

Then inspect the command line:

synthpopcan --help

Run a small offline smoke test with the bundled fictional model:

Runnable teaching example. Enter both commands in order.

synthpopcan models generate demo-linked-household-person \
  --households 10 \
  --condition "geo=Demo North" \
  --out synthpopcan-quickstart \
  --random-seed 42
synthpopcan validate linked synthpopcan-quickstart

This verifies linked household/person generation, but it does not create a representative Canadian population. The documentation explains how to choose research sources, controls, and model packages.

For a one-off command without installing the tool into the current environment, use uvx:

Network and uv required. See the Installation guide before using this route if uvx --version is not already available.

uvx synthpopcan --help
uvx synthpopcan guide ipf
uvx synthpopcan guide model
uvx synthpopcan guide small-area

From a source checkout for development:

Source checkout, network, Git, and uv required.

git clone https://github.com/dlq/synthpopcan.git
cd synthpopcan
uv sync
uv run synthpopcan --help

For installation details, see docs/installation.md.

Where To Start

Most readers should start in the Sphinx documentation rather than in this README:

Build the documentation locally with:

Source checkout required. Contributor setup is documented in CONTRIBUTING.md.

uv run sphinx-build -W -b html docs docs/_build/html

Developer Benchmarks

Benchmark fixtures are developer tooling, not normal user workflows:

Source checkout required. The province-scale timing run can also require substantial time and memory.

uv run python scripts/benchmarks.py ipf
uv run python scripts/benchmarks.py small-area
uv run python scripts/benchmarks.py small-area --province-scale
uv run python scripts/benchmarks.py ipf-backends

Use --seed-records for smaller or larger IPF runs. The province-scale small-area timing check is opt-in because it depends on the machine. Optional SciPy CSR and Polars comparisons remain benchmark probes rather than runtime backends, and full-data tree-model smoke tests remain outside the default suite. The CLI, beginner Python API, and local web app call the same Python domain implementations; IPF also shares its file-backed workflow orchestration. Browser guidance and durable artifacts do not define a separate computational tier. Scale limits come from the selected workflow, model, output size, and available machine resources.

Data Policy

Large, raw, private, or access-controlled data are not tracked in git.

• data/raw/ is a local ignored cache for authoritative public inputs, organized by provider, product family, and vintage.
• data/derived/ is a local ignored cache for reproducible conversions, subsets, and durable model artifacts; it must not be mistaken for an authoritative source.
• data/work/ is disposable project-local scratch space for model builds, experiments, and other restartable intermediate files.
• data/private/sources/ is a local ignored cache for access-controlled or sensitive source datasets; generated artifacts do not belong there.
• references/ is a local ignored cache for copied papers, proposals, and legacy code references.

Public geography, school, healthcare, road, and environmental layers should generally be fetched from authoritative public sources such as Statistics Canada, open.canada.ca, donneesquebec.ca, and municipal/provincial open-data portals rather than stored in this repository.

Local-only manifests may exist inside ignored data directories to document what is present on a development machine.

Model Packages

Reviewed model packages may be distributed with the project when they are explicitly intended as public research artifacts. The installed package should stay small: only the tiny demo model is bundled. Larger published models are downloaded on demand with synthpopcan models fetch MODEL_ID.

The public catalogue includes parallel 2016 and 2021 linked packages for Canada, supported provinces, and the five CMAs identified in the hierarchical PUMF. List the exact IDs with synthpopcan models list; for example:

synthpopcan models fetch quebec-2021-all-fields
synthpopcan models generate quebec-2021-all-fields \
  --households 1000 --out quebec-2021-population/

Release assets are gzip-compressed to keep downloads small. The CLI handles decompression and stores a normal JSON model package in the local cache.

Bundled model packages are not raw Census microdata. They should still be treated as derived research artifacts with provenance, disclosure-risk checks, and limitations. A model package being marked as a publishable candidate means it passed the project's current checks; it is not a claim of official approval, legal privacy certification, or fitness for every research use.

Before publishing a new model package, review docs/data.md, docs/tree.md, PLANS.md, and CONTRIBUTING.md.

How To Cite

Releases and prepared model packages are archived on Zenodo, so cite whichever matches what your work actually depended on.

For reproducibility, prefer the version DOI for the release and the model package DOI for each package you generated from: together they pin the exact code and the exact artifact, and every model record publishes the checksums needed to verify the file you downloaded.

Citation metadata lives in CITATION.cff, which GitHub renders as a ready-made citation from the sidebar.

Prepared model packages are derived from Statistics Canada public use microdata files under the Statistics Canada Open Licence. Each package carries the attribution notice that licence requires; keep it with the package and with anything you generate from it. See docs/data.md for the full attribution and licensing terms.

Development Acknowledgement

Development of SynthPopCan has been supplemented by the use of large language models for tasks including code generation, review, testing, documentation, and research assistance. All resulting contributions remain subject to human review and the project's automated correctness and quality checks. Responsibility for the project and its releases rests with its maintainer.