sciharvest

Zero-key, reproducible harvesting of public scientific datasets with auditable provenance.

sciharvest turns a tiny JSON "descriptor" (which source, which query) into a portable, self-describing dataset: every harvest writes a normalized payload plus a provenance manifest that records the request, the fetch time, the tool version, the source license, and a SHA-256 of the stored data. Re-run verify and reproduce at any time to prove the data hasn't drifted.

It has zero runtime dependencies (Python standard library only), which makes it trivial to install, audit, and run in air-gapped or long-term research environments.

Why

Reproducible research starts at the data boundary. All too often a figure's data lives in a file with no record of where it came from, when it was pulled, or under what license it was reused. sciharvest encodes that boundary:

• Re-run nobody's logs — a descriptor fully names the query.
• Prove provenance — the manifest hash means verifiable, self-describing data.
• Detect drift — upstream data changes are caught by reproduce, not discovered in review.

Features

• No API keys needed for any built-in source (USGS, OpenAlex, PubMed, Crossref).
• Zero runtime dependencies: pure urllib + unittest.
• Deterministic payload hashing (SHA-256) for offline re-verification.
• fetch / verify / reproduce lifecycle that treats data as code.
• Polite-pool etiquette: identifies itself (User-Agent, NCBI tool/email, Crossref mailto) and rate-limits NCBI calls.
• Recorded offline fixtures so the test suite never needs the network.

Install & Quickstart

$ git clone https://github.com/kyoai-zhao/sciharvest.git
$ cd sciharvest
$ python -m pip install -e .
$ sciharvest --version
sciharvest 0.1.0

Fetch an example dataset (USGS daily discharge, Potomac River):

$ sciharvest fetch examples/usgs-potomac-discharge.json --out ./data
potomac-daily-discharge-60d: 59 rows from usgs
payload_sha256: f3f615a02bb978ba28cef96d208e4658d33a5db1d35bcc73557aa4896cf54244
manifest:       ./data/manifests/potomac-daily-discharge-60d.json

Verify that stored data still matches its manifest:

$ sciharvest verify examples/usgs-potomac-discharge.json --out ./data
potomac-daily-discharge-60d: OK: stored data matches the manifest (59 rows)

Re-fetch and detect upstream drift:

$ sciharvest reproduce examples/usgs-potomac-discharge.json --out ./data

The dataset is now ./data:

data/
├── raw/
│   └── potomac-daily-discharge-60d.json     # normalized rows + columns
└── manifests/
    └── potomac-daily-discharge-60d.json     # provenance manifest (hash, request, license, time)

CLI reference

Dataset descriptors

A descriptor is the reproducible recipe — a small JSON object:

{
  "name": "potomac-daily-discharge-60d",
  "source": "usgs",
  "request": {
    "endpoint": "dv",
    "params": { "sites": "01646500", "parameterCd": "00060", "period": "P60D" }
  },
  "notes": "Daily-mean discharge at Potomac River near Washington, DC.",
  "license": "Public domain (U.S. Government work)"
}

Provenance manifest

Every payload is paired with an auditable manifest:

{
  "schema": "sciharvest/manifest.v1",
  "name": "potomac-daily-discharge-60d",
  "source": "usgs",
  "request": { "endpoint": "dv", "params": { "period": "P60D", ... } },
  "fetched_at": "2026-08-13T20:10:00Z",
  "tool_version": "0.1.0",
  "license": "Public domain (U.S. Government work)",
  "source_homepage": "https://waterdata.usgs.gov/",
  "source_documentation": "https://waterservices.usgs.gov/rest/",
  "record_count": 59,
  "columns": ["site_no", "variable", "unit", "datetime", "value", "qualifier"],
  "unit": "ft3/s",
  "hash_algorithm": "sha256",
  "payload_sha256": "f3f615a0..."
}

unit is read from the source payload (never assumed) and echoed into every row — a defensive check against silent unit mismatches in later analysis.

Built-in sources

Development

$ python -m unittest discover -s tests -t .       # offline fixture-based suite
$ python -m pip install -e .
$ sciharvest list-sources

The suite never touches the network; fixtures in tests/fixtures/ are pruned, real API responses. To add a source, see CONTRIBUTING.md.

Roadmap

• High-water-mark de-duplication and dataset diff output
• CSV/Parquet exporters with a stable column contract
• Descriptor validation schema convention ($schema)
• More keyless sources (e.g., US Census, Open-Meteo, UNdata)

License

MIT — see LICENSE. Upstream data retains its own license, which is recorded in every manifest.