Skip to main content

acadfetch

CI PyPI License: MIT

English · 简体中文

Batch-download files from academic data repositories — resumably, rate-limited, and checked against the checksums the repository published.

$ acadfetch fetch 10.5281/zenodo.23612969 -o ./data
title   : Dinler Tarihi Alanında Güncel Konular ve Akademik Değerlendirmeler
doi     : 10.5281/zenodo.23612969
license : cc-by-4.0
files   : 1, 1.6 MiB
landing : https://zenodo.org/records/23612969
Dinler Tarihi ... .pdf  1.2 MiB/1.6 MiB  312.4 KiB/s  ETA 0:02
ok   Dinler Tarihi ... .pdf 1.6 MiB (resumed at 512.0 KiB)

1/1 file(s) in ./data

What it is not: a search engine, a scraper for paywalled publishers, or a citation manager. It is the boring part in the middle — the part that gets killed by a flaky VPN at 80%, and the part that nobody writes down.

Why another downloader

Because curl -O in a loop loses three things that matter for research data:

Problem What happens with a plain loop What acadfetch does
Connection drops at 80% restart from zero resumes with Range + If-Range, keeps the partial
Silent corruption nobody notices until analysis breaks verifies the repository's md5/sha256, re-downloads on mismatch
"Which version did I actually use?" untraceable writes manifest.json with DOI, license, per-file SHA-256
Getting rate-limited into a 403 hammering, then blocked one global token bucket, honours Retry-After

The manifest is the point. A dataset you cannot re-verify is a dataset you cannot cite.

Install

Python 3.10–3.15. No runtime dependencies.

Try it without installing anything:

uvx acadfetch fetch 10.5281/zenodo.23612969   # uv

Or keep it on the PATH:

pipx install acadfetch

From a checkout, for development:

git clone https://github.com/young-ski/acadfetch && cd acadfetch
python -m pip install -e . pytest mypy && python -m pytest -q

Supported platforms

Platform Status
Linux tested — CI, every supported Python
Windows tested — CI on 3.10, 3.13 and 3.15
macOS best effort — CI runs it on 3.12 and 3.15, a failure is a bug to fix but does not hold a release
Windows ARM untested
Python 3.15 tested — CI, on Linux and Windows

Usage

One target per run:

acadfetch fetch 10.5281/zenodo.23612969        # a DOI
acadfetch fetch arxiv:1706.03762               # an arXiv id
acadfetch fetch https://ndownloader.figshare.com/files/70600233/report.pdf  # any URL

Or a whole list, with the failures kept for the next run:

acadfetch batch identifiers.txt                # one identifier per line
acadfetch batch identifiers.txt -o ./data --failed-out todo.txt

Blank lines and # comments are skipped, so a list can carry notes. Every identifier still gets a directory of its own under -o (default acadfetch-out); one bad line does not stop the rest, and the identifiers that failed land in failed.txt (or --failed-out), verbatim and one per line — so the retry is acadfetch batch failed.txt. The file is replaced even when nothing failed, so a stale list never lies. Exit codes keep their meaning: 0 all ok, 1 all failed, 2 partial, 130 interrupted, and an interrupt puts the lines that were never attempted on the list too. Records are attempted one after another; -j still applies inside a record.

The first two are answered by Zenodo and arXiv and work from any network those are reachable on. The third is a direct URL, so the host decides: figshare's CDN answers 403 with a WAF block page from some networks — the same behaviour that made its adapter not worth keeping in 0.2.0. acadfetch says so and keeps nothing, rather than saving the block page as a PDF. If you hit that, the file itself is fine; see the 403 section of docs/troubleshooting.md.

Picking a few files out of many

A record with 70 files rarely wants all of them. --list-files prints the names and transfers nothing; edit the list, then feed it back:

acadfetch fetch 10.5281/zenodo.19665762 --list-files > want.txt
acadfetch fetch 10.5281/zenodo.19665762 --only-file want.txt --failed-out failed.txt
acadfetch fetch 10.5281/zenodo.19665762 --only-file failed.txt   # retry the misses

--only-file takes one exact name per line (blank lines and # comments are skipped, - reads stdin, names are case-insensitive); it combines with --only and is narrowed by --glob. --failed-out writes the selected names that could not be fetched, verbatim and in selection order, replacing the file even when nothing failed. A name that matches nothing is a failure, not a quietly smaller download: it is named on stderr, written to the list, and the run exits 2.

figshare was removed in 0.2.0: pass the article's download URL instead, and --checksum keeps the verification the API used to give you.

The rest of the surface:

acadfetch fetch 10.5281/zenodo.23612969 -n     # dry run: which dir, which files
acadfetch fetch 10.5281/zenodo.23612969 --only raw.tar.gz
acadfetch fetch 10.5281/zenodo.23612969 --max-size 2GB   # refuse anything bigger
acadfetch fetch 10.5281/zenodo.23612969 -j 1 --rate 0.5  # be a good citizen
acadfetch fetch 10.5281/zenodo.23612969 --retries 2 --retry-403 0  # give up sooner
acadfetch fetch 10.5281/zenodo.23612969 --manifest-out build/m.json  # a second copy
acadfetch manifest 10.5281/zenodo.23612969 -o m.json     # metadata only, no bytes
acadfetch fetch zenodo:123456 --header "Authorization: Bearer $TOKEN"   # restricted

acadfetch verify ./data/manifest.json          # re-hash what you downloaded
acadfetch --version                            # what is installed
acadfetch sources                              # adapters, with an example each

10.5281/zenodo.123456, zenodo:123456 and arxiv:2401.12345 are placeholders; use any real id.

In a pipeline

--json puts one report document on stdout and moves progress to stderr, so the two never interleave:

acadfetch fetch zenodo:123456 --json | jq -r '.files[] | select(.state=="failed") | .name'

To hand the bytes to another downloader, -w/--urls writes an aria2-compatible list (- is stdout) and transfers nothing:

acadfetch fetch zenodo:123456 -w - --glob '*.csv' | aria2c -i -

--glob takes shell patterns and combines with --only (both must pass).

Exit codes are unchanged, and a failure before any transfer still produces a document with error set. report is the document's schema version (now 2, field reference in docs/json-report.md): fields are added, never removed or given a new meaning, and state may gain values — treat an unknown one as non-terminal. verify --json reports per-file MISMATCH rows the same way, and its summary counts skipped rows separately. At the end of a failed transfer, part names the .part file that was kept, so a retry can resume instead of starting over. A 403 that carries a block page is retried; see docs/troubleshooting.md.

Progress is one aggregate line for the whole run (3 file(s) 12.4/44.1 MiB 1.8 MiB/s ETA 0:17), redrawn in place on a TTY. When stdout is a pipe it becomes a new line at most every 5 s (one extra when a file starts or stops), so a CI log stays greppable without becoming a wall. -q/--quiet turns it off.

-o/--out defaults to acadfetch-out/<source>-<id> (a URL adds its host and a short hash), and -n prints it before you spend bandwidth. Files land in <name>.part and are renamed only after the checksum matches, so an interrupted or failed run never leaves something that looks finished -- and never throws away the bytes it already paid for: rerun and it resumes. The one case where a run must start over is a server that ignores Range; the bytes already on disk are then moved aside and kept until a whole new copy has arrived, so a restart that dies early costs nothing. acadfetch-out/ is worth adding to a project's .gitignore; this repository already has it, and names the default failed.txt there as well.

The default directory names the target, not only its filename — data.csv from two hosts is two datasets, and they do not share a directory. A file already there is reused when its checksum matches; with no checksum, the size plus one more tie (an explicit -o, or a manifest in that directory that recorded the same URL) is required, and --force skips the question entirely.

A DOI that resolves to a web page instead of a file is refused, naming the page: fetching the landing page as if it were the dataset is how a repository's front door ends up being cited as a data file.

A run that cannot fit is refused before the first byte is transferred, naming the shortfall: needs 41.2 GiB but 12.0 GiB is free. What is already on disk is subtracted, so a resume is judged on the remainder. Each finished file is flushed before its rename (--no-fsync skips that; it is faster and weaker against a power cut).

Exit codes: 0 success, 1 failure, 2 partial, 64 bad usage, 130 interrupted (rerun to resume).

Tokens

ZENODO_TOKEN is read from the environment and sent only to Zenodo, so a token never travels to a host that has no business seeing it -- including across a redirect: a hop to another host (a storage CDN, a download mirror) goes out without it, a hop from https to http drops it with a warning (--allow-http keeps it), and a presigned URL gets no Authorization at all, because S3 rejects a request that carries both. Cookies and proxy credentials are dropped on any cross-origin hop as well. --header still overrides, --no-token ignores the environment, and the manifest records the variable name plus the last four characters only:

export ZENODO_TOKEN=...            # restricted records, higher rate limits
acadfetch fetch zenodo:123456      # prints: auth    : token from $ZENODO_TOKEN (…9f2c)

Verification and the manifest

Every completed run writes manifest.json:

{
  "schema": 3,
  "doi": "10.5281/zenodo.123456",
  "license": "cc-by-4.0",
  "files": [
    { "name": "raw.tar.gz", "size": 8123, "checksum": "md5:9f2c…",
      "sha256": "b41d…", "state": "downloaded", "selected": true,
      "verified": true, "verified_reason": "matched-md5" },
    { "name": "notes.txt", "size": 12, "state": "not-selected", "selected": false,
      "sha256": null, "verified": null, "verified_reason": "not-selected" }
  ]
}

The manifest describes the record, not the run: --only still writes a row for every file, so verify can tell "not fetched" from "fetched and then lost", and a successful --only run verifies green. Add --require-all when you want the whole record accounted for. Each fetched row also records final_url, etag, http_status and retrieved_at, and credential query parameters are stripped from every URL (--record-full-url keeps them). Full field and state reference: docs/manifest.md.

name is the file on disk, which can differ from the repository's name when two files collide. verified_reason separates the two kinds of verified: null: no-checksum-published (arXiv) from unparseable-checksum (a repository that published something we cannot name an algorithm for). Schema 1 and 2 manifests still read, and verify says so for schema 1, because their names were recorded before duplicates were resolved.

acadfetch verify recomputes SHA-256 for each entry and exits non-zero on any mismatch or missing file; rows the run never selected are reported skipped. Add --against-remote to ask the source again and compare what it publishes now: a manifest someone rewrote stops agreeing even though the bytes still match it. --require-remote makes an unreachable source a failure. Commit the manifest next to your analysis; it is what makes the download reproducible a year later.

Names are resolved against the manifest's own directory, and a name that points outside it — ../sibling/file, an absolute path, a symlink leading out — is reported as escaped rather than quietly hashed, because a manifest is a document someone can hand you. --allow-escape follows them; --root changes what "inside" means.

When a source publishes no checksum (arXiv), verification degrades to a byte count and verified is null rather than a false sense of safety.

Supported sources

Source Target forms Checksums
Zenodo DOI 10.5281/zenodo.N, zenodo:N, record URL md5
arXiv arxiv:ID, bare ID, abs/pdf URL none (size only)
doi.org any DOI, resolved then dispatched depends
direct any http(s) URL headers, if present

figshare (figshare:N, 10.6084/…, article URLs) was removed in 0.2.0; it answers with the conversion instead of a generic "cannot tell what this is". See docs/migration-0.2.md.

A CDN in front of a direct URL can answer 403 with a block page. Those are retried twice with backoff, while a 403 carrying the application's own JSON is terminal on the first answer; --retry-403 N changes the count.

Networking

Proxies, timeouts and a private CA are flags, not luck:

acadfetch fetch zenodo:123456 --proxy http://proxy.campus:3128   # overrides the environment
acadfetch fetch zenodo:123456 --no-proxy                        # ignore HTTP(S)_PROXY
acadfetch fetch zenodo:123456 --timeout 15                      # idle/connect budget, seconds
acadfetch fetch zenodo:123456 --ca-bundle corp-tls.pem          # trust this PEM instead of the store

--proxy takes http:// or https:// and wins over --no-proxy; without either, HTTP_PROXY / HTTPS_PROXY / ALL_PROXY are honoured by urllib as usual. A timeout is per request, connect to first byte.

Debugging

--verbose (or ACADFETCH_DEBUG=1) traces each request, each retry and the traceback of a failure to stderr; stdout, --json documents and exit codes are unchanged. Credential header values and presigned query parameters are masked, so the trace can be pasted into an issue.

Politeness

Defaults: 3 parallel transfers (-j/--concurrency), 2 requests/second, 4 retries with jittered exponential backoff, Retry-After honoured up to 120 s. A block page is retried twice (--retry-403 N); Zenodo and arXiv are donation-funded — --rate 1 -j 1 costs you minutes and costs them less.

Respect the license in the manifest. --dry-run prints it before you spend bandwidth; unknown means check before redistributing.

Adding a source

  1. Read acadfetch/sources/zenodo.py — an adapter is match(target) -> id | None plus resolve(client, target) -> Record.
  2. Fill File(checksum=...) whenever the API publishes one.
  3. Never hardcode rate limits; take the shared Client.
  4. Add a match test and, if possible, a fixture of the real API payload.

scripts/probe_resume.py interrupts a live transfer and resumes it — the only honest way to test this code path.

Getting involved

A 30-second tour shows the whole loop, and docs/integrations.md has Make, CI, aria2, a Snakemake storage plugin and a pooch downloader.

A nightly job downloads three fixed public records and verifies them, so a rotted adapter opens an issue before a user reports one. It never blocks a merge; the next good night closes the issue.

  • Stuck on something → docs/troubleshooting.md first; it is short and indexed by symptom
  • Usage questions → Discussions
  • Bugs → the bug template; say whether verify reports MISMATCH
  • Security (bad bytes, path escapes, token leaks) → SECURITY.md, privately
  • Adapters, tests, docs → PRs welcome, read CONTRIBUTING.md first

Known limits

  • figshare was removed in 0.2.0; existing figshare manifests still verify, and new targets convert with one --checksum command (docs/migration-0.2.md).
  • Tokens come from the environment only; no interactive login flow and no per-file token refresh.
  • No search/pagination; you bring the target.
  • arXiv PDFs are fetched by constructed URL, so a withdrawn ID gives a 404 after resolution.
  • Windows ARM is untested.
  • macOS is best effort: it is in CI but not release-blocking, and nobody has run it by hand yet.
  • A source that publishes neither a size nor a checksum (bare direct URLs without a Content-Length on HEAD) can still be truncated silently by a server that lies about Content-Length, because HTTP gives no in-band way to notice the extra bytes.

中文

学术数据仓库的批量下载器:断点续传、全局限速、按仓库公布的校验和验证,并落 manifest.json(DOI、许可协议、每文件 SHA-256)供日后复现。

它不是搜索器、不抓付费墙、也不是引文管理器 —— 它只管"取数据"这一步,并让这一步 事后可证明。

安装

Python 3.10–3.15,零运行时依赖。

uvx acadfetch fetch 10.5281/zenodo.23612969   # 不安装,直接跑
pipx install acadfetch                        # 或者装到 PATH 上

快速开始

10.5281/zenodo.23612969 是真实记录,可以直接复制;其余 id 都是占位符。

acadfetch fetch 10.5281/zenodo.23612969 -n     # 先看:落到哪、传多少、许可是什么
acadfetch fetch 10.5281/zenodo.23612969 -o ./data
acadfetch verify ./data/manifest.json          # 事后复验;对不上就非零退出

目标也可以是 arxiv:1706.03762、任意 DOI,或任意 https:// 直链。

直链成败由对方主机决定:figshare 的 CDN 在某些网络会回 403 拦截页(当年删掉 figshare 适配器正是因为这个)。acadfetch 会明说这是拦截页、而不是把 HTML 存成 PDF,并把下一条命令给你。

批量清单则一条一条来,失败的留下:

acadfetch batch identifiers.txt                # 每行一个目标
acadfetch batch identifiers.txt -o ./data --failed-out todo.txt

空行与 # 注释会被跳过;每个目标仍在 -o(默认 acadfetch-out)下拿到自己 的目录,一行坏掉不影响其余。失败的目标原样写进 failed.txt(或 --failed-out),重跑就是 acadfetch batch failed.txt;没失败也会清空该文件, 不会留下上次的假清单。退出码含义不变:0 全成、1 全败、2 部分、130 被中断 —— 中断时还没轮到的行也会进清单。

单个记录里只挑几个文件时,用 --list-files 取名单(只打印名字、不传字节),再 用 --only-file 喂回去:

acadfetch fetch 10.5281/zenodo.19665762 --list-files > want.txt
acadfetch fetch 10.5281/zenodo.19665762 --only-file want.txt --failed-out failed.txt
acadfetch fetch 10.5281/zenodo.19665762 --only-file failed.txt   # 重试漏掉的

名单一行一个文件名,空行与 # 注释跳过,- 读标准输入,名字不区分大小写;可与 --only 合并、再被 --glob 收窄。写错的名单算失败:stderr 会点名、会进清单、退出 码是 2,不会被悄悄跳过。注意 batch 的 --failed-out 记的是目标,fetch 的记的 是文件名。

它替你保住的东西

情况 它怎么做
传到 80% 断了 .part 保留,重跑用 Range/If-Range 接着传;服务器若无视 Range,旧字节先挪到一边,整份新文件到齐才替换
悄悄损坏 按仓库公布的 md5/sha256 校验,不符就重下;仍不符则 verified: false 并删掉坏字节
"我到底下的是哪个版本" manifest.json 记 DOI、许可、每文件 SHA-256、最终 URL、HTTP 状态与时间
被限流吃 403 全局令牌桶限速,遵守 Retry-After,拦截页按次数重试

默认 3 并发、2 请求/秒、4 次重试。Zenodo 和 arXiv 靠募捐运营,--rate 1 -j 1 多花你几分钟,对面少受点罪。

进流水线

acadfetch fetch 10.5281/zenodo.23612969 --json    # 一个 JSON 文档给 stdout,进度走 stderr
acadfetch fetch 10.5281/zenodo.23612969 -w -      # 输出 aria2 可用 URL 列表,不传字节

--json 的字段契约见 docs/json-report.md;report 是文档 版本号,只加字段、不改语义。

退出码

0 成功、1 失败、2 部分成功、64 用法错误、130 被中断(.part 已保留, 重跑即可续)。

令牌

ZENODO_TOKEN 从环境读取,只发给 Zenodo;跨主机跳转时剥掉,https 降级到 http 时也会剥掉并告警。manifest 只记变量名和后四位。

export ZENODO_TOKEN=...
acadfetch fetch zenodo:123456

支持的源

源 目标形式 校验和
Zenodo DOI 10.5281/zenodo.N、zenodo:N、记录 URL md5
arXiv arxiv:ID、裸 ID、abs/pdf URL 无(只比字节数)
doi.org 任意 DOI,解析后再分派 视源而定
直链 任意 http(s) URL 响应头里有就用

figshare 适配器在 0.2.0 已移除;源不公布校验和时用 --checksum md5:... 补上, 旧 manifest 仍可 verify。见 docs/migration-0.2.md。

已知限制

  • manifest.json 没有签名:verify 证明磁盘上的字节与 manifest 一致,不能证明 manifest 本身没被改写。补这一层用 verify --against-remote(会重新问源站), 细节见 SECURITY.md。
  • arXiv 不公布校验和,验证降级为字节数,verified 是 null 而不是假的安心。
  • 没有搜索与分页,目标得你自己给。
  • Windows ARM 未测;macOS 只做尽力而为(在 CI 里,但不阻塞发布)。

更多

许可

工具是 MIT。数据本身的许可见 manifest 的 license 字段;unknown 表示转发前先 确认,而 -n 会在花掉带宽之前就把它打出来。

Citing

acadfetch --cite prints how to cite the tool; the machine-readable form is CITATION.cff. For the data, cite the repository record — the manifest is the part that lets a reader re-check it.

Audit trail

How the 3.15 support claim was reached, including what it did not prove — the free-threaded build, Windows ARM, and every place the probe ran short: docs/py315-windows-report.md.

Metadata

Release files for acadfetch 0.9.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for acadfetch 0.9.0
File Size Uploaded Provenance Actions
acadfetch-0.9.0.tar.gz 293.0 kB With provenance Details for acadfetch-0.9.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for acadfetch 0.9.0
File Interpreter ABI Platform Actions
acadfetch-0.9.0-py3-none-any.whl Python 3 none any Details for acadfetch-0.9.0-py3-none-any.whl

Total release size: 367.5 kB

Release files / acadfetch-0.9.0.tar.gz

Download URL acadfetch-0.9.0.tar.gz
Size 293.0 kB
Tags Source
SHA-256 checksum
How to use checksums
6a11f97e5473db4cb59b3178ac8f060e872e75f4464e6e915df602cedeca8344
BLAKE2b-256 checksum
How to use checksums
75b2043580dff877403900b696fd1206c80fbe72b31c730379a72680992999f8
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 18, 2026.

Transparency log

Release files / acadfetch-0.9.0-py3-none-any.whl

Download URL acadfetch-0.9.0-py3-none-any.whl
Size 74.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
f73da374d460135510f544a72592211b8016e49a9e82c41496fd2089949ca992
BLAKE2b-256 checksum
How to use checksums
ff0789b0a077293add0a76128abc09568495545894d11e93ce1f1f461b034497
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 18, 2026.

Transparency log

Security for acadfetch 0.9.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for acadfetch-0.9.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 18, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page