pixls-preprints

Access and store metadata from different APIs offering preprint publications.

pixls-preprints is an asynchronous command line harvester for preprint metadata. It queries bioRxiv, medRxiv and OSF for a range of dates and writes the results as JSON Lines: one document per line, one file per source.

$ python -m pixls_preprints --start 2024-07-14 --dest data --servers biorxiv osf
$ wc -l data/biorxiv/2024-07-14.json
49 data/biorxiv/2024-07-14.json

Why use it

• Fast. Everything is asynchronous: all pages of a query are fetched concurrently, and so are the sources. You get as much speed as your rate limit allows, and an OSF token raises that ceiling.
• Several sources, one command. bioRxiv, medRxiv and OSF sit behind a single uniform CLI, each writing into its own output file.
• Polite by default. Every source has its own rate limit (requests per time window) and concurrency cap, so you stay within what the upstream API grants you.
• Survives flaky APIs. 429, 502, 503, 504 and dropped connections are retried with a back-off, honouring Retry-After. Whatever still fails is logged, counted and reported at the end of the run, not fatal.
• Streams to disk. Each page is written as soon as it arrives, so a long harvest does not accumulate in memory and partial results survive an interruption.
• JSON Lines output. Results can be appended, read line by line without loading a whole file, and fed straight into a downstream indexing pipeline.
• Layered configuration. Config file, then environment variables, then defaults. Tokens are read only from the environment, never from the config file.
• Easy to extend. A new preprint server means one new subclass of Source. The HTTP client, rate limiting, retries and file writing are inherited.

Requirements

• Python 3.11 or newer (the project is developed against 3.14, see .python-version)
• uv for dependency management
• An OSF API token, which is optional but raises the rate limit OSF grants you

Installation

git clone https://codeberg.org/pixls/pixls-preprints.git
cd pixls-preprints
uv sync

Then create the two local files. Both are optional and both are ignored by git, so every clone needs its own copy:

cp config.toml.example config.toml
cp .env.example .env          # then put your OSF token in it

Without config.toml the tool falls back to the defaults shipped inside the package, which hold the same values as config.toml.example. .env is loaded automatically at startup, so there is no need to source it, and variables already set in your shell take precedence over it.

Usage

Run the package as a module:

uv run python -m pixls_preprints -d data -srv biorxiv osf

Installing the package instead provides a pixls-preprints command, which takes the same options and needs no config file to start:

uv pip install pixls-preprints
pixls-preprints -d data -srv biorxiv osf

Without a date, yesterday is harvested. The destination directory gets one subdirectory per source, and the output file is named after the requested range.

A single day from bioRxiv:

uv run python -m pixls_preprints -s 2024-07-14 -d data -srv biorxiv
# -> data/biorxiv/2024-07-14.json

A whole month from two sources, into one file each:

uv run python -m pixls_preprints -s 2024-07-01 -e 2024-07-31 -d data -srv medrxiv osf
# -> data/medrxiv/2024-07-01_-_2024-07-31.json
# -> data/osf/2024-07-01_-_2024-07-31.json

The same month, but one file per day, overwriting what is already there:

uv run python -m pixls_preprints -s 2024-07-01 -e 2024-07-31 -S -F -d data -srv biorxiv
# -> data/biorxiv/2024-07-01.json … data/biorxiv/2024-07-31.json

A custom file name and a config file from somewhere else:

uv run python -m pixls_preprints -s 2024-07-14 -d data -f july.json -c /etc/pixls/config.toml

Existing output files are never silently overwritten: the source is skipped with a note unless you pass --force. Log records are appended to pixls_preprints_cli.log in the directory you ran the command from.

If requests are given up on after their retries, the output file is short. The run still finishes and still exits 0, but prints one warning per source saying how many requests were lost:

$ uv run python -m pixls_preprints -s 2024-07-14 -d data -srv biorxiv osf
WARNING: biorxiv: 3 requests failed permanently, output may be incomplete

Options

Option	Description
-s, --start	Start date (YYYY-MM-DD), defaults to yesterday
-e, --end	End date (YYYY-MM-DD), defaults to the start date
-d, --dest	Required. Destination directory, gets one subdirectory per source
-srv, --servers	Required. Sources to fetch from: biorxiv, medrxiv, osf (one or more)
-f, --file	Output file name, defaults to a name built from the dates
-S, --split-daily	Write one output file per day instead of one for the whole range
-F, --force	Overwrite already existing output files
-c, --config	Path to the config file, defaults to config.toml in the directory you run from

An end date without a start date is an error. Note that bioRxiv and medRxiv select by publication date, while OSF selects by modification date.

Configuration

Per-source settings are resolved from three places, in this order of precedence:

1. the TOML config file
2. environment variables named PIXLS_PREPRINTS_<SOURCE>_<SETTING>
3. the defaults shipped with the package, in pixls_preprints/config_defaults.toml

The config file is the one given with -c, or config.toml in the directory you run from. A path given with -c has to exist, otherwise the run stops with Config file not found. Without a config file the run uses the packaged defaults, and says so in the log.

# config.toml
[biorxiv]
max_rate = 1          # requests allowed per time_period
time_period = 0.4     # length of the rate limit window, in seconds
max_concurrency = 7   # requests in flight at most

[osf]
max_rate = 1
time_period = 1.5
max_concurrency = 10
page_size = 50        # documents requested per page

Tokens are the exception: they are read only from the environment. A token entry in the config file is ignored and produces a warning.

# .env
export PIXLS_PREPRINTS_OSF_TOKEN=your-token-here

See config.toml.example and .env.example for the full set of keys.

Output format

One JSON document per line, in whatever shape the upstream API returns it. bioRxiv and medRxiv records look like this (abridged):

{"doi": "10.1101/001891", "title": "Population genomics of …", "authors": "Carlotta De Filippo;Monica Di Paola;…", "date": "2014-01-17", "version": "1", "type": "New Results", "license": "cc_by_nc_nd", "category": "Evolutionary Biology", "abstract": "…", "server": "biorxiv"}

OSF records follow the JSON:API shape of the OSF v2 API, with bibliographic contributors and license embedded so no extra request per preprint is needed.

Development

The Makefile wraps the common tasks:

make test              # run the test suite, skipping tests that need network access
make test-online       # run only the tests that talk to the real APIs
make check-formatting  # black, in check mode
make formatting        # black, rewriting files
make check-code        # flake8
make check-coverage    # pytest with coverage
make audit             # formatting + lint + tests, the pre-merge gate

Tests are marked with medium, large and online (see pytest.ini), so you can deselect the slow or network-dependent ones, for example uv run pytest -m "not online" tests/.

Project layout

pixls_preprints/
├── cli.py            # argument parsing, date handling, task orchestration
├── config.py         # config file + environment + defaults
├── fetcher.py        # rate-limited async HTTP client with retries
└── sources/
    ├── source.py     # abstract base class shared by all sources
    ├── biorxiv.py    # bioRxiv and medRxiv (cursor paging)
    └── osf.py        # OSF (page-number paging, optional token)

To add a source, subclass Source, implement url, fetch() and fetch_preprints(), and wire it into _gather_tasks() and the --servers choices in pixls_preprints/cli.py.

Getting help

• Something broken, or a source missing? Open an issue at codeberg.org/pixls/pixls-preprints/issues.
• Questions about the upstream APIs: see the bioRxiv/medRxiv API summary and the OSF API documentation.
• Runtime detail: every run appends to pixls_preprints_cli.log, which records the arguments used, each request and every retry. It is the first place to look when a harvest comes back short.

Contributing

Contributions are welcome. Please read CONTRIBUTING.md for the workflow, coding conventions and what make audit expects before you open a pull request.

Maintainers

Developed and maintained by Benjamin Wolff (wolff@zbmed.de) at ZB MED – Information Centre for Life Sciences, as part of the PIXLS project.

Funding

Developed in PIXLS – Preprint Information eXtraction for Life Sciences, funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation).

License

Source code is licensed under the MIT License. Documentation and other content are licensed under CC BY 4.0, and configuration and generated files under CC0 1.0. See LICENSING.md for details.