statcan-dl

Takes a local snapshot of Statistics Canada's health tables — all 400-odd of them — and leaves you with Parquet files you can open in pandas.

This is not a query API. It mirrors what StatCan publishes to your disk in one shot: no API key, no auth, no rate limits to negotiate. Run it once to get the tables, re-run it later to pick up revisions. Every step skips work already on disk, so an interrupted run resumes where it stopped.

Expect roughly 10 GB and 20–60 minutes on a decent connection for the full health subject.

Install

pip install git+https://github.com/ink-waffle/statcan-dl

Get the data

statcan-dl

That writes to output/statcan/ relative to your current directory. The part you'll actually use is output/statcan/tables/:

import pandas as pd

df = pd.read_parquet("output/statcan/tables/13100489 — Perceived health.parquet")
df.head()

Every table arrives in StatCan's long format — one row per (period, geography, characteristic) combination:

Don't need all of it? Narrow it:

statcan-dl --pids 13100489,13100098       # just these tables
statcan-dl --limit 20                     # first 20, useful for a trial run
statcan-dl --subject 1310                 # a narrower branch of the subject tree
statcan-dl -o ~/data/statcan -w 16        # elsewhere, more parallelism

Subject codes are a hierarchy and match by prefix, so 13 is all of Health and 1310 is a branch within it. List what you'd get without downloading:

python -m statcan_dl.download --list

Dimension metadata

The CSVs give you dimension values but not the structure behind them. output/statcan/table_metadata.json holds that, keyed by product ID — which dimensions a table has, every member of each, and how members nest via parent:

import json

meta = json.load(open("output/statcan/table_metadata.json"))
for dim in meta["13100489"]["dimensions"]:
    print(dim["position"], dim["name"], len(dim["members"]))

That parent field is what lets you tell a total apart from its components — necessary if you're aggregating, since totals sit in the same column as the things they total.

What it produces

output/statcan/
├── csv/<pid>/              raw CSV archives, as extracted
├── tables/                 one Parquet per table  ← use these
└── table_metadata.json     dimensions and members, per table

csv/ is an intermediate; it exists so re-runs don't re-download and so you can go back to the source if a conversion looks wrong. Delete it once you have tables/ and you save most of the disk.

Converting drops seven bookkeeping columns that carry no information once a table stands alone: DGUID, UOM_ID, SCALAR_ID, VECTOR, COORDINATE, TERMINATED, SYMBOL.

Pipeline steps

Run one on its own with --only, or exclude one with --skip-<step>:

statcan-dl --only combine
statcan-dl --skip-metadata

Individual steps are also importable and directly runnable:

python -m statcan_dl.download --limit 5
python -m statcan_dl.combine

Caveats

• VALUE becomes text instead of a float when a table mixes numbers with suppression markers that won't parse. Check the dtype before doing arithmetic.
• Archived tables are excluded. They're still on StatCan's site, but they're superseded and often overlap live tables.
• Titles are truncated to 120 characters in filenames. The full title is in the Parquet metadata under statcan_title, and in table_metadata.json.
• English only — the loader requests -eng.zip archives.

Tests

pip install -e ".[dev]"
pytest                  # runs the real pipeline over 3 small tables
pytest -m "not network" # offline only

Licence

MIT. The data itself is published by Statistics Canada under the Open Licence.