cdc-nhanes-dl

Takes a local snapshot of the entire public NHANES dataset — every table, every survey cycle — and leaves you with Parquet files you can open in pandas.

This is not a query API. It mirrors what CDC publishes to your disk in one shot: no API key, no auth, no rate limits to negotiate. Run it once to get the dataset, re-run it later to pick up new cycles. Every step skips work already on disk, so an interrupted run resumes where it stopped.

Expect roughly 6 GB and 15–40 minutes on a decent connection for the full set.

Install

pip install git+https://github.com/ink-waffle/cdc-nhanes-dl

Get the data

cdc-nhanes-dl

That writes to output/nhanes/ relative to your current directory. The part you'll actually use is output/nhanes/nhanes_tables/:

import pandas as pd

bmx = pd.read_parquet("output/nhanes/nhanes_tables/BMX — Body Measures.parquet")
bmx[["SEQN", "CYCLE", "BMXWT — Weight (kg)", "BMXBMI — Body Mass Index (kg/m**2)"]].head()

One file per table, with all cycles stacked into it. SEQN is the respondent ID (join key across tables), CYCLE tells you which survey wave a row came from.

Don't need everything? Narrow it:

cdc-nhanes-dl --tables BMX,DEMO,GHB          # just these tables, all cycles
cdc-nhanes-dl --cycles 2017-2018,2021-2023   # just these cycles, all tables
cdc-nhanes-dl -o ~/data/nhanes -w 16         # elsewhere, more parallelism

Column names

Raw NHANES variable codes are unreadable, so columns are renamed to CODE — Label using the labels embedded in CDC's SAS files:

BMXWT — Weight (kg)
LBXGH — Glycohemoglobin (%)
RIDAGEYR — Age in years at screening

The bare code stays available in Parquet metadata if you need to map back — per column under the nhanes_code field key, or for the whole table as a JSON map under nhanes_columns.

Typed schemas

The final step generates pandera models under output/nhanes/types/, one module per component, so you get autocomplete over those long column names instead of typing them by hand:

from output.nhanes.types.examination.types import BmxSchema

BmxSchema.validate(bmx)
bmx[BmxSchema.bmxbmi]   # -> "BMXBMI — Body Mass Index (kg/m**2)"

Requires pip install pandera. Pass --import-prefix to match wherever the generated package ends up on your import path (--import-prefix output.nhanes.types for the layout above).

What it produces

output/nhanes/
├── xpt/<cycle>/                 raw SAS transport files, as downloaded
├── parquet/<cycle>/             one-to-one Parquet conversion
├── nhanes_tables/               cycles stacked per table  ← use these
├── types/                       generated pandera schemas
├── table_categories.json        table → component
└── table_descriptions.json      table → plain-English name

xpt/ and parquet/ are intermediates; they exist so re-runs don't re-download and so you can go back to the source if a conversion looks wrong. Delete them once you have nhanes_tables/ and you save about 4 GB.

Pipeline steps

Run one on its own with --only, or exclude one with --skip-<step>:

cdc-nhanes-dl --only download
cdc-nhanes-dl --skip-types

Individual steps are also importable and directly runnable:

python -m cdc_nhanes_dl.download --cycles 2017-2018
python -m cdc_nhanes_dl.combine --tables BMX

Caveats

• Tables with no SEQN column — format lookups, pooled-sample files — are skipped by combine, since they have no respondent to join on. They're still in parquet/.
• Multi-cycle aggregate releases (1999-2004, 1988-2020, …) are downloaded but excluded from stacking; their rows already appear in the individual cycles.
• When a variable is text in one cycle and numeric in another, the stacked column becomes text. Cast it yourself.
• When a label was reworded between cycles, the most recent wording is used for the whole column.
• Limited Access tables are listed in the metadata but their data is not public — CDC gates it behind the RDC, so those downloads fail and are reported as such.

Tests

pip install -e ".[dev]"
pytest                  # runs the real pipeline over 2 small tables
pytest -m "not network" # offline only

Licence

MIT. The data itself is public domain, published by the US CDC / NCHS.