Skip to main content
Documentation Status

Data Gatherer

Data Gatherer is a Python library for automatically extracting dataset references from scientific publications. It processes full-text articles—whether in HTML or XML format—and uses both rule-based and LLM-based methods to identify and structure dataset citations.

What It Does

  • Parses scientific articles from open-access sources like PubMed Central (PMC).

  • Extracts dataset mentions from structured sections (e.g., Data Availability, Supplementary Material).

  • Supports two main strategies:

    • Retrieve-Then-Read (RTR): First retrieves relevant sections using hand-crafted rules, then applies LLMs.

    • Full-Document Read (FDR): Applies LLMs to the full text without section filtering.

  • Outputs structured results in JSON format.

  • Includes support for known repositories (e.g., GEO, PRIDE, MassIVE) via a configurable ontology.

Flowchart illustrating the main passages in the process_url function.

Use Cases

  • Helping data curators and librarians identify datasets cited in publications.

  • Supporting meta-analysis and secondary data discovery.

  • Enabling dataset indexing and retrieval across the open-access literature.

Citation

If you use Data Gatherer in your research or project, please cite our paper:

@inproceedings{marini_data_2025,
        title = {Data {Gatherer}: {LLM}-{Powered} {Dataset} {Reference} {Extraction} from {Scientific} {Literature}},
        url = {https://aclanthology.org/2025.sdp-1.10},
        doi = {10.18653/v1/2025.sdp-1.10},
        booktitle = {Proceedings of the {Fifth} {Workshop} on {Scholarly} {Document} {Processing} ({SDP} 2025)},
        publisher = {Association for Computational Linguistics},
        author = {Marini, Pietro and Santos, Aécio and Contaxis, Nicole and Freire, Juliana},
        year = {2025},
        pages = {114--123},
}

Metadata

Release files for data-gatherer 0.2.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for data-gatherer 0.2.2
File Size Uploaded
data_gatherer-0.2.2.tar.gz 175.7 kB Details

Built distribution (wheel)

Table of built distributions (wheels) for data-gatherer 0.2.2
File Interpreter ABI Platform
data_gatherer-0.2.2-py3-none-any.whl Python 3 none any Details

Total release size: 382.4 kB

Release files / data_gatherer-0.2.2.tar.gz

Download URL data_gatherer-0.2.2.tar.gz
Size 175.7 kB
Tags Source
SHA-256 checksum
How to use checksums
3020bccab9f167d4a10e8ed638592af8bd77813ec8a49c4eb164d129ba70ca17
BLAKE2b-256 checksum
How to use checksums
e61122b21c0735cc439338c2f133ee42ebcc02deba438feb7178e35a82526bf2
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.7

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on May 26, 2026.

Transparency log

Release files / data_gatherer-0.2.2-py3-none-any.whl

Download URL data_gatherer-0.2.2-py3-none-any.whl
Size 206.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
b9e3563bd5cef644b1216bb179188d564e844dcdd2622e42336ddf2d918a16cd
BLAKE2b-256 checksum
How to use checksums
4d13e0b8b9bd11b7bba79d0e5a39d6b6d09284e0ea76e54f78f33836fe631ef8
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.7

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on May 26, 2026.

Transparency log

Release history Release notifications | RSS feed

This release

0.2.2 This release

2 release files

0.2.1

2 release files

0.2.0

2 release files

0.1.8

2 release files

0.1.7

2 release files

0.1.6

2 release files

0.1.5

2 release files

0.1.3

2 release files

0.1.2

2 release files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page