Betteroff

Clean data and charts from the Michigan consumer surveys.

Betteroff finds, downloads, validates, exports and charts the published monthly tables from the University of Michigan Surveys of Consumers. It keeps the question wording, response labels and source details attached to every file it writes, so a CSV or chart can be traced back to the exact download behind it.

betteroff search government
betteroff get government_policy --wide -o government_policy.csv
betteroff chart government_policy --rolling 12 --recessions -o government_policy.svg

Betteroff is an independent project. It is not affiliated with or endorsed by the University of Michigan. See Data use before publishing anything made with it.

Install

Betteroff needs Python 3.11 or newer. The charts are an optional extra because they pull in Altair and vl-convert.

# Command-line tool in its own environment, with charts
uv tool install 'betteroff[charts]'

# Try it without installing
uvx betteroff search housing
uvx --from 'betteroff[charts]' betteroff chart 41

# Python library in an analysis project or notebook environment
uv add 'betteroff[charts]'

pip install 'betteroff[charts]' works too. A uv tool install doesn't make the module importable in a notebook; add it to the notebook's project for that.

Quick start

Find a question. Search and list work offline because they read only the packaged catalog.

$ betteroff search unemployment
Table  Id                           Title
   30  unemployment_expectations    Expected change in unemployment during the next year
   29  business_conditions_5_years  Business conditions expected during the next 5 years
   34  government_policy            Opinions about the government's economic policy

Read the question's wording, response categories and notes. Once data is cached, describe also shows the latest three months and a 24-month sparkline per response.

betteroff describe unemployment_expectations

Get the data. In a terminal you see the latest 24 months; piped or written to a file you get every month.

betteroff get 30                                   # table in the terminal
betteroff get 30 > unemployment.csv                # tidy CSV, one row per month and response
betteroff get 30 --wide -o unemployment_wide.csv   # one row per month, for Datawrapper
betteroff get 30 --measure index -f json           # Michigan's index score as JSON records

Chart it.

betteroff chart 30 --open                                   # unemployment_expectations.html
betteroff chart 30 --rolling 12 --recessions -o unemp.svg   # editable vector file
betteroff chart 30 --start 2000 -o unemp.png                # 2x PNG

Table numbers and ids are interchangeable: betteroff get 34 and betteroff get government_policy return the same data.

Python

import betteroff as bo

bo.search('housing')
ds = bo.get('government_policy', refresh=True)
ds.data          # tidy pandas DataFrame
ds.wide()        # one row per month
ds.metadata      # wording, source, snapshot, coverage, notes
chart = ds.chart(rolling=12, recessions=True)   # a plain altair.LayerChart
chart.save('government_policy.html', inline=True)

See the Python API docs.

Supported tables

Every table in Michigan's catalog that uses the historical three-category layout, with one positive, one neutral and one negative response plus nonresponse where reported:

Tables with other layouts (the sentiment indexes, income and price distributions, probabilities, reasons for opinions and demographic breakdowns) aren't supported yet. All series cover all households.

How it works

1. Betteroff reads Michigan's tables page, finds the current link to the table's historical Excel file and downloads it. Links carry a per-release key, so they are discovered fresh instead of guessed.
2. It checks the file is a real Excel workbook, not an access page or error text sent with HTTP 200, then parses it and checks the column headers against the catalog. Any unexpected or missing column stops the run with exit code 4.
3. It validates the values and saves the raw bytes in a content-addressed cache with a manifest. A failed download or validation never replaces the last good snapshot.
4. Later runs reuse the cache for 24 hours. --refresh forces a check; --offline never touches the network.
5. When a refresh brings new bytes, Betteroff compares them with the previous snapshot and reports new months, revised months and removed months.

Latest month is not the same as retrieval time. Michigan's detailed-data site updates after an embargo that ends four weeks after each month's final release, so the newest month here trails the headline number by a month or more. A refresh that finds no new month is working as intended.

More in the data docs.

Outputs

Every file gets a metadata sidecar with the question wording, response mapping, source URL, retrieval time, snapshot hash, coverage, notes, source quirks and package versions. Betteroff refuses to overwrite a file unless you pass --force.

Recipes

Save a workflow in TOML and rerun it with one command:

schema_version = 1
output_dir = 'output'

[[jobs]]
question = 'government_policy'
start = '2000-01-01'
data_formats = ['csv', 'json']
chart_formats = ['html', 'svg']
rolling = 12
recessions = true

betteroff run betteroff.toml --refresh

See the recipe docs.

Exit codes

Errors print one line and a suggested fix. Add --debug before the command, or set BETTEROFF_DEBUG=1, for a full traceback.

Environment

Data use

Michigan's usage agreement lets you display, reformat and print the data for your organization's use with the attribution "University of Michigan, Survey Research Center, Surveys of Consumers." It bars reproducing, redistributing, publishing or broadcasting the data without written consent from the university.

Betteroff's MIT license covers the code only, not the data. Betteroff fetches data to your machine and never hosts or bundles it. Its tests use synthetic files. Getting permission for any publication is up to you; charts include the required attribution in their footer.

Documentation

• Command-line reference
• Python API
• Data, metadata and caching
• Charts
• Recipes
• Releasing
• Changelog

Development

git clone https://github.com/stiles/betteroff && cd betteroff
uv sync --all-extras
uv run pytest                # offline tests
uv run pytest -m live        # fetch every table from Michigan; run sparingly
uv run ruff check src tests && uv run ruff format --check src tests
uv run mypy

Normal tests never touch the network. A scheduled GitHub Action runs the live check weekly and opens an issue if Michigan's files change.

License

MIT for the code. The data belongs to the University of Michigan; see Data use.