sourceledger

Build a dataset from public sources where every row says which download it came from.

A project lists its inputs in sources.toml: each source's URL, the publisher's reuse terms, and whether the raw file may be republished. sourceledger runs four stages over it:

• fetch downloads each source into data/raw/ and writes a record beside it: the URL, the time, the SHA-256 and size, the vintage, and the server's ETag. The next fetch sends the ETag back, so an unchanged source isn't downloaded again. Each new version is also added to data/fetches.jsonl.
• build re-hashes each download, reads it, and writes data/processed/<id>.csv with five provenance columns in front: source_id, source_url, retrieved_at, vintage and raw_sha256. It also writes datapackage.json, a Frictionless Data Package that describes every output. The same records always give the same files.
• validate re-reads the outputs and lists every problem it finds, such as an output from an older fetch, an edited raw file, a record made for another URL, vintage or pin, a source without reuse terms, or a datapackage.json that no longer matches.
• revisions OLD_LOG lists the sources whose bytes changed since an earlier copy of the log. A weekly CI run can use it to report a publisher's revision.

init, status, attribution and verify help around them: see Other commands.

There are no dependencies. Excel files need the optional xlsx extra. The docs page walks through a fetch and a publisher's silent revision.

Install

uv add sourceledger            # or 'sourceledger[xlsx]' to read Excel files

Quick start

Write a sources.toml in the project root. sourceledger init writes one to start from, with this example commented out:

[[source]]
id = "co2"
status = "implemented"
url = "https://raw.githubusercontent.com/datasets/co2-ppm/main/data/co2-annmean-mlo.csv"
vintage = "main branch, unversioned"
terms = "ODC-PDDL-1.0 (datasets/co2-ppm package; original data from NOAA GML)"
terms_url = "https://opendatacommons.org/licenses/pddl/"
license = "PDDL-1.0"
redistribute = true

Then run the stages from that directory:

sourceledger fetch
sourceledger build
sourceledger validate

sourceledger reads CSV (gzipped or not), JSON and Excel, by each source's format or its URL's extension. For other formats, see Your own readers.

What to commit

Commit sources.toml and data/fetches.jsonl. The log holds checksums and URLs, not data: it's the history of every version you have fetched.

Keep data/raw/ and data/processed/ out of git, so the repository never redistributes a publisher's data. The .fetch.json records live in data/raw/ too, so a fresh clone runs fetch before build and validate.

sources.toml

Each [[source]] table takes these keys:

validate reports a missing terms, terms_url, license or redistribute. The other keys are checked when the manifest loads. Keys not listed here are kept, for a project's own reader.

To have an editor with a TOML language server (Tombi, Taplo) check the manifest as you type, put this line at the top of sources.toml:

#:schema https://raw.githubusercontent.com/simonvanlierde/sourceledger/main/src/sourceledger/sources.schema.json

The schema accepts exactly what sourceledger accepts. A test keeps the two in step.

How fetch behaves

• It tries four times, waiting 5, 10 and 20 s between attempts, on 429, 500, 502, 503, 504, network errors and a body cut short. A Retry-After header, in seconds or as a date, raises the wait to at most 120 s. Any other error status fails at once.
• Each connect and read times out after 30 s, and one source's whole fetch after 300 s.
• It asks for the file uncompressed, and refuses a compressed reply. The checksum is always of the file as published.
• It refuses a redirect from a public host to a loopback, private or link-local address. When a redirect is followed, the record keeps the final URL too.
• Every source is tried. When some fail, the others are still written, and the run exits 1 with one line per failed source.
• Each download streams to a temporary file, hashed on the way, and is renamed into place. Memory stays small, and a crash never leaves half a file. A log line cut off by a crash is dropped on the next run.
• A second fetch in the same project, while one runs, exits at once (a lock on data/raw/.lock, Linux and macOS only).
• Each line it prints says new or unchanged, against the last version in the log.
• For a source with archive = true, each connect and read of the Internet Archive request times out after 15 s. When it fails, fetch prints a warning, leaves archive_url empty, and still succeeds.

Other commands

• init writes a starter sources.toml in the current directory: the schema line and one example source, commented out. It refuses to overwrite a sources.toml that exists.
• status prints one line per source: its id, status and vintage, when its last logged version was fetched (or never), the start of its SHA-256, its licence, and whether the raw file may be republished. It reads the manifest and the log, never the network.
• attribution prints a Markdown table of the implemented sources, in manifest order: name, terms in the publisher's words, licence, terms URL, and whether the raw file may be republished. Paste it into a dataset's README or release notes.
• verify DIR is for someone who downloaded a published dataset. It reads DIR/datapackage.json and checks that each file it lists exists, with the size and SHA-256 it gives. It prints one line per problem and exits 1 if there are any. It needs no sources.toml and no network.

fetch, build and validate take --only ID, as often as needed, to work on just those sources. An id that is not in sources.toml is an error. build --only keeps the other sources' entries in datapackage.json as the last build wrote them, and validate --only skips the checks that cover the whole output.

Your own readers

sourceledger reads every source with parse_source, which hands it to parse_csv, parse_json or parse_xlsx. For another format, a project writes its own reader, a function from a source's manifest table and raw bytes to columns and rows, and its own command:

# src/my_dataset/pipeline.py
import sourceledger
from sourceledger import Source


def parse(source: Source, payload: bytes) -> tuple[list[str], list[dict[str, str]]]:
    """Branch on source["id"] for a source in a format sourceledger doesn't read."""
    return sourceledger.parse_source(source, payload)


def main(argv: list[str] | None = None) -> int:
    return sourceledger.main(argv, parse=parse, prog="my-dataset", user_agent="my-dataset")

# pyproject.toml
[project.scripts]
my-dataset = "my_dataset.pipeline:main"

my-dataset fetch, build and validate then use that reader, and publishers see my-dataset as the User-Agent. The project-templates data-pipeline kind sets this up for you, with CI and a weekly revision report. The stages are also functions: load_manifest, fetch, build, validate, revisions, status, attribution and verify.

Compared with

Other tools cover parts of this. Use one of them when that part is all you need.

Checked against each project's code and docs on 2026-10-18.

Stability

Until 1.0, the public API is sourceledger's __all__, the keys of sources.toml, and the format of fetch records and the log. A breaking change to any of them comes only in a minor version, listed in CHANGELOG.md, with a warning first where that's cheap. Older fetch logs stay readable.

Development

uv sync
uv tool install prek   # once per machine
prek install           # git hooks
just check             # lint, types and tests, as CI runs them

License

MIT. See LICENSE.