subsetsio

The subsets.io public data library, local-first. One pip package that clones the catalog of published Delta tables to your machine, searches it with the production retrieval engine, queries it with guarded local SQL, and serves it all over HTTP — no accounts, no API keys, no hosted service.

pip install subsetsio            # library + CLI into the active environment
subsets sync                     # clone the small tier (~3.5k tables, ~200 MB)
subsets serve                    # http://127.0.0.1:8080 over your mirror

Python 3.10 or newer; wheels are published for Linux, macOS, and Windows, so there is nothing to compile.

Want the subsets command globally, isolated from your projects? Use uv tool install subsetsio or pipx install subsetsio instead — same package, same command; you'll still pip install subsetsio in any environment where you want import subsetsio. If subsets isn't on your PATH after a plain pip install, python -m subsetsio is the same CLI.

Data comes anonymously from the public bundles host over plain HTTPS (SUBSETS_BUNDLES_URL, default https://bundles.subsets.io): the catalog manifest, the metadata document, and tiered tar bundles rebuilt hourly from whatever changed. Everything after subsets sync works offline.

The CLI

subsets sync                   # clone / refresh the mirror (small tier)
subsets add --bundle medium    # widen the selection (--bundle large: the widest tier)
subsets add                    # print what this mirror is set to sync
subsets sync --dry-run         # see what would change
subsets sync --prune           # drop local tables outside the selection
subsets list                   # browse the catalog
subsets search "malaysia inflation"      # ranked search, in-process engine
subsets info dosm-cpi-headline-inflation # schema, license, attribution
subsets query 'SELECT * FROM "dosm-cpi-headline-inflation" LIMIT 10'
subsets serve [--host 127.0.0.1] [--port 8080]
subsets mcp                    # MCP server over stdio (Claude Desktop etc.)

Sync is bundle-based: sources are packed into three cumulative size tiers (one tar per tier) at the natural breaks of their size distribution, so a cold clone is a handful of GETs instead of tens of thousands. Re-syncing is incremental — a bundle is only re-downloaded when its content changed, and tables above the per-table size cap are not distributed.

large is the widest tier, not the whole database. We do not yet serve the entire index — the published bundles are a slice of it, filled in usefulness-adjusted size order (the most useful data per byte first), and that slice widens over time. So --bundle large means "everything we currently bundle", and re-syncing it later gets you more than it did before.

The mirror remembers its tier. subsets add --bundle {small,medium,large} sets it (persisted at the mirror root, per mirror) and subsets add alone prints it; every later subsets sync keeps that tier, so a bare sync never silently narrows back to small. sync --bundle X still works standalone and is remembered the same way — except under --dry-run, which previews without choosing. Narrowing the selection deletes nothing on its own: the tables from the wider tier stay until you ask for sync --prune.

--prune removes local tables that are no longer in the catalog selection. It refuses to run when the catalog selection is empty, or when it would take more than half the synced tables with it — both look like a degraded catalog rather than housekeeping; --force says you meant it (a narrower tier, a different SUBSETS_BUNDLES_URL).

search/info/query answer entirely from your machine and take --json to emit the raw payload (the same shapes the local server speaks). subsets search prints ranked dataset ids one per line, best hit first (default 60) — ids are semantic slugs, so the bare list is the cheapest candidate set an agent can consume. --matched annotates each id with its match evidence: values:myr@currency_code (6.7k) means the literal myr sits in that column's cells in that many rows (the WHERE clause writes itself, and the CLI prints it as a next: suggestion), title:/desc:/ text: mark metadata hits, ~x marks a token with no lexical trace — a semantic match. Ids that explain themselves stay bare. Cell evidence covers synced tables only (and only columns passing the value gates), so its absence never means the value is absent. subsets info <id...> has the detail (schema, license, attribution — several ids at once, and --query to include the same match evidence). A typo-corrected query is reported via a corrected: line (stderr in plain mode). No matches → empty stdout, exit 1. SQL is DuckDB dialect; dataset ids are the table names (double-quote them — hyphens otherwise parse as subtraction), and the referenced datasets must be synced first.

Search — the production engine, in-process

subsets search (and GET /search) runs a compiled multi-lane retrieval engine (Rust, shipped inside the wheel — see rust/) fused with weighted reciprocal-rank fusion over a typo-corrected query: lexical lanes (curated keywords and synthetic queries from the published metadata, IDF-weighted text over titles and columns), two semantic lanes (static embeddings over dataset descriptions and synthetic queries, so conceptual queries match without exact words), and an exact-match lane over code-shaped values (tickers, ISO codes, category labels) extracted from your synced tables — so subsets search AAPL finds the table whose cells mention AAPL even though its metadata never does. The whole ~26k-dataset catalog is searchable regardless of how much is synced. The index persists under the mirror (a value index plus the engine's state cache), so warm rebuilds touch only what changed. The embedding model (~30 MB, minishlab/potion-base-8M) downloads from HuggingFace on the first index build and is cached locally — everything after that is offline.

The local server

subsets serve binds 127.0.0.1:8080 by default and exposes five endpoints over the mirror — no auth, errors as clean JSON ({"detail": "..."}):

SQL runs in an in-process DuckDB via delta_scan over your synced tables, behind three guards: read-only statements only (writes, DDL, table functions, file/URL access are refused with a 400), a 100,000-row result cap (413 — aggregate or LIMIT for more), and a 60s execution timeout (504).

The MCP server

subsets mcp speaks the Model Context Protocol over stdio — the same engine as serve, exposed as three tools (search, get_dataset, execute_sql) so MCP clients can use your mirror directly. No auth, no network, stdlib-only transport. For Claude Desktop, add to claude_desktop_config.json:

{"mcpServers": {"subsets": {"command": "subsets", "args": ["mcp"]}}}

Tool failures (bad SQL, unknown dataset, nothing synced yet) come back as isError tool results carrying the same messages the CLI prints, so the model can read them and recover.

The Python library

import subsetsio

subsetsio.list_tables()                  # the catalog, as a DataFrame
subsetsio.search("refugee flows")        # ranked search, local engine
subsetsio.info("unhcr-population")       # schema, license, attribution (a dict)
df = subsetsio.read("unhcr-population")  # one table, as a DataFrame
df = subsetsio.query('SELECT * FROM "unhcr-population" LIMIT 10')  # local SQL

read serves the table from the local mirror (a pinned Delta table) — repeated reads are milliseconds, and anything synced keeps working offline.

For agents — the Claude skill

This repo ships a Claude skill that teaches agents the whole flow above (install → sync → search/info/query → cite the upstream source). It lives at plugins/subsets/skills/subsets/SKILL.md and versions with the package, enforced by tests/test_skill_parity.py.

• Claude Code: add this repo as a plugin marketplace (.claude-plugin/marketplace.json at the repo root).
• claude.ai: scripts/build-skill-zip.sh builds the uploadable dist/subsets-skill.zip.

How it works

The sync contract is the catalog manifest: every published table with its pinned Delta version, path, and size, fetched in one GET. Bundles are the only fetch path — the client downloads the tiers it asked for, extracts exactly the tables whose catalog pin the bundle carries, and reports anything the bundles don't cover (tables over the size cap, or newer than the last bundle build). Cloned tables are valid Delta tables at the pinned version — the log plus only the parquet files that version references.

Beside the catalog, subsets sync stores the host's metadata.json: per dataset, the title, description, column schema, attribution (upstream source, source_url, license), and the curated retrieval signals (keywords, synthetic queries) that feed the search index. Both documents are parsed strictly against a versioned contract (subsetsio/models.py) before anything is persisted: every cataloged dataset must have a metadata entry, required fields must be present and well-typed, and a violation fails the sync loudly naming the document, dataset, and field — a bad publish never lands on the mirror. Only the enrichment fields (description, keywords, queries, tags) plus columns, license, and source_url may be absent, and they are still type-checked when present.

Configuration

Environment variables are the whole user-facing configuration surface. The one thing written to disk beside the data is the mirror's own selection (<mirror>/.selection.json — the tier subsets add chose); deleting it just resets the selection to small.

License

MIT — see LICENSE. The datasets it distributes carry their own upstream licenses; subsets info reports each one, and info --json gives you the field to cite.