acadfetch

English · 简体中文

Batch-download files from academic data repositories — resumably, rate-limited, and checked against the checksums the repository published.

$ acadfetch fetch 10.5281/zenodo.23612969 -o ./data
title   : Dinler Tarihi Alanında Güncel Konular ve Akademik Değerlendirmeler
doi     : 10.5281/zenodo.23612969
license : cc-by-4.0
files   : 1, 1.6 MiB
landing : https://zenodo.org/records/23612969
Dinler Tarihi ... .pdf  1.2 MiB/1.6 MiB  312.4 KiB/s  ETA 0:02
ok   Dinler Tarihi ... .pdf 1.6 MiB (resumed at 512.0 KiB)

1/1 file(s) in ./data

What it is not: a search engine, a scraper for paywalled publishers, or a citation manager. It is the boring part in the middle — the part that gets killed by a flaky VPN at 80%, and the part that nobody writes down.

Why another downloader

Because curl -O in a loop loses three things that matter for research data:

The manifest is the point. A dataset you cannot re-verify is a dataset you cannot cite.

Install

Python 3.10–3.15. No runtime dependencies.

Try it without installing anything:

uvx acadfetch fetch 10.5281/zenodo.23612969   # uv

Or keep it on the PATH:

pipx install acadfetch

From a checkout, for development:

git clone https://github.com/young-ski/acadfetch && cd acadfetch
python -m pip install -e . pytest mypy && python -m pytest -q

Supported platforms

Usage

One target per run:

acadfetch fetch 10.5281/zenodo.23612969        # a DOI
acadfetch fetch arxiv:1706.03762               # an arXiv id
acadfetch fetch https://ndownloader.figshare.com/files/70600233/report.pdf  # any URL

The first two are answered by Zenodo and arXiv and work from any network those are reachable on. The third is a direct URL, so the host decides: figshare's CDN answers 403 with a WAF block page from some networks — the same behaviour that made its adapter not worth keeping in 0.2.0. acadfetch says so and keeps nothing, rather than saving the block page as a PDF. If you hit that, the file itself is fine; see the 403 section of docs/troubleshooting.md.

figshare was removed in 0.2.0: pass the article's download URL instead, and --checksum keeps the verification the API used to give you.

The rest of the surface:

acadfetch fetch 10.5281/zenodo.23612969 -n     # dry run: which dir, which files
acadfetch fetch 10.5281/zenodo.23612969 --only raw.tar.gz
acadfetch fetch 10.5281/zenodo.23612969 --max-size 2GB   # refuse anything bigger
acadfetch fetch 10.5281/zenodo.23612969 -j 1 --rate 0.5  # be a good citizen
acadfetch fetch 10.5281/zenodo.23612969 --retries 2 --retry-403 0  # give up sooner
acadfetch fetch 10.5281/zenodo.23612969 --manifest-out build/m.json  # a second copy
acadfetch manifest 10.5281/zenodo.23612969 -o m.json     # metadata only, no bytes
acadfetch fetch zenodo:123456 --header "Authorization: Bearer $TOKEN"   # restricted

acadfetch verify ./data/manifest.json          # re-hash what you downloaded
acadfetch --version                            # what is installed
acadfetch sources                              # adapters, with an example each

10.5281/zenodo.123456, zenodo:123456 and arxiv:2401.12345 are placeholders; use any real id.

In a pipeline

--json puts one report document on stdout and moves progress to stderr, so the two never interleave:

acadfetch fetch zenodo:123456 --json | jq -r '.files[] | select(.state=="failed") | .name'

To hand the bytes to another downloader, -w/--urls writes an aria2-compatible list (- is stdout) and transfers nothing:

acadfetch fetch zenodo:123456 -w - --glob '*.csv' | aria2c -i -

--glob takes shell patterns and combines with --only (both must pass).

Exit codes are unchanged, and a failure before any transfer still produces a document with error set. report is the document's schema version (now 2, field reference in docs/json-report.md): fields are added, never removed or given a new meaning, and state may gain values — treat an unknown one as non-terminal. verify --json reports per-file MISMATCH rows the same way, and its summary counts skipped rows separately. At the end of a failed transfer, part names the .part file that was kept, so a retry can resume instead of starting over. A 403 that carries a block page is retried; see docs/troubleshooting.md.

Progress is one aggregate line for the whole run (3 file(s) 12.4/44.1 MiB 1.8 MiB/s ETA 0:17), redrawn in place on a TTY. When stdout is a pipe it becomes a new line at most every 5 s (one extra when a file starts or stops), so a CI log stays greppable without becoming a wall. -q/--quiet turns it off.

-o/--out defaults to acadfetch-out/<source>-<id> (a URL adds its host and a short hash), and -n prints it before you spend bandwidth. Files land in <name>.part and are renamed only after the checksum matches, so an interrupted or failed run never leaves something that looks finished -- and never throws away the bytes it already paid for: rerun and it resumes. The one case where a run must start over is a server that ignores Range; the bytes already on disk are then moved aside and kept until a whole new copy has arrived, so a restart that dies early costs nothing. acadfetch-out/ is worth adding to a project's .gitignore; this repository already has it.

The default directory names the target, not only its filename — data.csv from two hosts is two datasets, and they do not share a directory. A file already there is reused when its checksum matches; with no checksum, the size plus one more tie (an explicit -o, or a manifest in that directory that recorded the same URL) is required, and --force skips the question entirely.

A DOI that resolves to a web page instead of a file is refused, naming the page: fetching the landing page as if it were the dataset is how a repository's front door ends up being cited as a data file.

A run that cannot fit is refused before the first byte is transferred, naming the shortfall: needs 41.2 GiB but 12.0 GiB is free. What is already on disk is subtracted, so a resume is judged on the remainder. Each finished file is flushed before its rename (--no-fsync skips that; it is faster and weaker against a power cut).

Exit codes: 0 success, 1 failure, 2 partial, 64 bad usage, 130 interrupted (rerun to resume).

Tokens

ZENODO_TOKEN is read from the environment and sent only to Zenodo, so a token never travels to a host that has no business seeing it -- including across a redirect: a hop to another host (a storage CDN, a download mirror) goes out without it, a hop from https to http drops it with a warning (--allow-http keeps it), and a presigned URL gets no Authorization at all, because S3 rejects a request that carries both. Cookies and proxy credentials are dropped on any cross-origin hop as well. --header still overrides, --no-token ignores the environment, and the manifest records the variable name plus the last four characters only:

export ZENODO_TOKEN=...            # restricted records, higher rate limits
acadfetch fetch zenodo:123456      # prints: auth    : token from $ZENODO_TOKEN (…9f2c)

Verification and the manifest

Every completed run writes manifest.json:

{
  "schema": 3,
  "doi": "10.5281/zenodo.123456",
  "license": "cc-by-4.0",
  "files": [
    { "name": "raw.tar.gz", "size": 8123, "checksum": "md5:9f2c…",
      "sha256": "b41d…", "state": "downloaded", "selected": true,
      "verified": true, "verified_reason": "matched-md5" },
    { "name": "notes.txt", "size": 12, "state": "not-selected", "selected": false,
      "sha256": null, "verified": null, "verified_reason": "not-selected" }
  ]
}

The manifest describes the record, not the run: --only still writes a row for every file, so verify can tell "not fetched" from "fetched and then lost", and a successful --only run verifies green. Add --require-all when you want the whole record accounted for. Each fetched row also records final_url, etag, http_status and retrieved_at, and credential query parameters are stripped from every URL (--record-full-url keeps them). Full field and state reference: docs/manifest.md.

name is the file on disk, which can differ from the repository's name when two files collide. verified_reason separates the two kinds of verified: null: no-checksum-published (arXiv) from unparseable-checksum (a repository that published something we cannot name an algorithm for). Schema 1 and 2 manifests still read, and verify says so for schema 1, because their names were recorded before duplicates were resolved.

acadfetch verify recomputes SHA-256 for each entry and exits non-zero on any mismatch or missing file; rows the run never selected are reported skipped. Add --against-remote to ask the source again and compare what it publishes now: a manifest someone rewrote stops agreeing even though the bytes still match it. --require-remote makes an unreachable source a failure. Commit the manifest next to your analysis; it is what makes the download reproducible a year later.

Names are resolved against the manifest's own directory, and a name that points outside it — ../sibling/file, an absolute path, a symlink leading out — is reported as escaped rather than quietly hashed, because a manifest is a document someone can hand you. --allow-escape follows them; --root changes what "inside" means.

When a source publishes no checksum (arXiv), verification degrades to a byte count and verified is null rather than a false sense of safety.

Supported sources

figshare (figshare:N, 10.6084/…, article URLs) was removed in 0.2.0; it answers with the conversion instead of a generic "cannot tell what this is". See docs/migration-0.2.md.

A CDN in front of a direct URL can answer 403 with a block page. Those are retried twice with backoff, while a 403 carrying the application's own JSON is terminal on the first answer; --retry-403 N changes the count.

Networking

Proxies, timeouts and a private CA are flags, not luck:

acadfetch fetch zenodo:123456 --proxy http://proxy.campus:3128   # overrides the environment
acadfetch fetch zenodo:123456 --no-proxy                        # ignore HTTP(S)_PROXY
acadfetch fetch zenodo:123456 --timeout 15                      # idle/connect budget, seconds
acadfetch fetch zenodo:123456 --ca-bundle corp-tls.pem          # trust this PEM instead of the store

--proxy takes http:// or https:// and wins over --no-proxy; without either, HTTP_PROXY / HTTPS_PROXY / ALL_PROXY are honoured by urllib as usual. A timeout is per request, connect to first byte.

Debugging

--verbose (or ACADFETCH_DEBUG=1) traces each request, each retry and the traceback of a failure to stderr; stdout, --json documents and exit codes are unchanged. Credential header values and presigned query parameters are masked, so the trace can be pasted into an issue.

Politeness

Defaults: 3 parallel transfers (-j/--concurrency), 2 requests/second, 4 retries with jittered exponential backoff, Retry-After honoured up to 120 s. A block page is retried twice (--retry-403 N); Zenodo and arXiv are donation-funded — --rate 1 -j 1 costs you minutes and costs them less.

Respect the license in the manifest. --dry-run prints it before you spend bandwidth; unknown means check before redistributing.

Adding a source

1. Read acadfetch/sources/zenodo.py — an adapter is match(target) -> id | None plus resolve(client, target) -> Record.
2. Fill File(checksum=...) whenever the API publishes one.
3. Never hardcode rate limits; take the shared Client.
4. Add a match test and, if possible, a fixture of the real API payload.

scripts/probe_resume.py interrupts a live transfer and resumes it — the only honest way to test this code path.

Getting involved

A 30-second tour shows the whole loop, and docs/integrations.md has Make, CI, aria2, a Snakemake storage plugin and a pooch downloader.

A nightly job downloads three fixed public records and verifies them, so a rotted adapter opens an issue before a user reports one. It never blocks a merge; the next good night closes the issue.

• Stuck on something → docs/troubleshooting.md first; it is short and indexed by symptom
• Usage questions → Discussions
• Bugs → the bug template; say whether verify reports MISMATCH
• Security (bad bytes, path escapes, token leaks) → SECURITY.md, privately
• Adapters, tests, docs → PRs welcome, read CONTRIBUTING.md first

Known limits

• figshare was removed in 0.2.0; existing figshare manifests still verify, and new targets convert with one --checksum command (docs/migration-0.2.md).
• Tokens come from the environment only; no interactive login flow and no per-file token refresh.
• No search/pagination; you bring the target.
• arXiv PDFs are fetched by constructed URL, so a withdrawn ID gives a 404 after resolution.
• Windows ARM is untested.
• macOS is best effort: it is in CI but not release-blocking, and nobody has run it by hand yet.
• A source that publishes neither a size nor a checksum (bare direct URLs without a Content-Length on HEAD) can still be truncated silently by a server that lies about Content-Length, because HTTP gives no in-band way to notice the extra bytes.

中文

学术数据仓库的批量下载器：断点续传、全局限速、按仓库公布的校验和验证，并落 manifest.json（DOI、许可协议、每文件 SHA-256）供日后复现。

它不是搜索器、不抓付费墙、也不是引文管理器 —— 它只管"取数据"这一步，并让这一步 事后可证明。

安装

Python 3.10–3.15，零运行时依赖。

uvx acadfetch fetch 10.5281/zenodo.23612969   # 不安装，直接跑

pipx install acadfetch                        # 或者装到 PATH 上

快速开始

10.5281/zenodo.23612969 是真实记录，可以直接复制；其余 id 都是占位符。

acadfetch fetch 10.5281/zenodo.23612969 -n     # 先看：落到哪、传多少、许可是什么
acadfetch fetch 10.5281/zenodo.23612969 -o ./data
acadfetch verify ./data/manifest.json          # 事后复验；对不上就非零退出

目标也可以是 arxiv:1706.03762、任意 DOI，或任意 https:// 直链。

直链成败由对方主机决定：figshare 的 CDN 在某些网络会回 403 拦截页（当年删掉 figshare 适配器正是因为这个）。acadfetch 会明说这是拦截页、而不是把 HTML 存成 PDF，并把下一条命令给你。

它替你保住的东西

默认 3 并发、2 请求/秒、4 次重试。Zenodo 和 arXiv 靠募捐运营，--rate 1 -j 1 多花你几分钟，对面少受点罪。

进流水线

acadfetch fetch 10.5281/zenodo.23612969 --json    # 一个 JSON 文档给 stdout，进度走 stderr
acadfetch fetch 10.5281/zenodo.23612969 -w -      # 输出 aria2 可用 URL 列表，不传字节

--json 的字段契约见 docs/json-report.md；report 是文档 版本号，只加字段、不改语义。

退出码

0 成功、1 失败、2 部分成功、64 用法错误、130 被中断（.part 已保留， 重跑即可续）。

令牌

ZENODO_TOKEN 从环境读取，只发给 Zenodo；跨主机跳转时剥掉，https 降级到 http 时也会剥掉并告警。manifest 只记变量名和后四位。

export ZENODO_TOKEN=...
acadfetch fetch zenodo:123456

支持的源

figshare 适配器在 0.2.0 已移除；源不公布校验和时用 --checksum md5:... 补上， 旧 manifest 仍可 verify。见 docs/migration-0.2.md。

已知限制

• manifest.json 没有签名：verify 证明磁盘上的字节与 manifest 一致，不能证明 manifest 本身没被改写。补这一层用 verify --against-remote（会重新问源站）， 细节见 SECURITY.md。
• arXiv 不公布校验和，验证降级为字节数，verified 是 null 而不是假的安心。
• 没有搜索与分页，目标得你自己给。
• Windows ARM 未测；macOS 只做尽力而为（在 CI 里，但不阻塞发布）。

更多

• 30 秒演示 docs/demo.md · 按症状排障 docs/troubleshooting.md · 接进流水线 docs/integrations.md（Make、CI、aria2、Snakemake 插件、pooch 下载器）
• 加源、动传输、写测试的规矩见 CONTRIBUTING.md

许可

工具是 MIT。数据本身的许可见 manifest 的 license 字段；unknown 表示转发前先 确认，而 -n 会在花掉带宽之前就把它打出来。

Citing

acadfetch --cite prints how to cite the tool; the machine-readable form is CITATION.cff. For the data, cite the repository record — the manifest is the part that lets a reader re-check it.

Audit trail

How the 3.15 support claim was reached, including what it did not prove — the free-threaded build, Windows ARM, and every place the probe ran short: docs/py315-windows-report.md.