Skip to main content

Get and analyze all PLOS articles

Project description

Build Status

All of Plos (allofplos)

Copyright (c) 2017-2022, Public Library of Science. MIT License, see LICENSE.txt for more information.

Why allofplos?

This is for downloading/updating/maintaining a repository of all PLOS XML article files. This can be used to have a copy of the PLOS text corpus for further analysis. Use this program to download all PLOS XML article files instead of doing web scraping.

Installation instructions

This program requires Python 3.8+.

Using pip:

$ pip install allofplos

This should install allofplos and requirements. At this stage you are ready to go.

If you want to manually install from source (for example for development purposes), first clone the project repository:

$ git clone git@github.com:PLOS/allofplos.git

Install Python dependencies inside the newly created virtual environment:

$ pipenv install

How to run the program

Execute the following command.

$ python -m allofplos.update

or, if running from source:

$ pipenv run python -m allofplos.update

The first time it runs it will download a >7 Gb zip file (allofplos.zip) with all the XML files inside. Note: Make sure that you have enough space in your device for the zip file and for its content before running this command (at least 30Gb). After this file is downloaded, it will extract its contents into the allofplos_xml directory inside your installation of allofplos.

If you want to see the directory on your file system where this is installed run

python -c "from allofplos import get_corpus_dir; print(get_corpus_dir())"

If you ever downloaded the corpus before, it will make an incremental update to the existing corpus. The script:

  • checks for and then downloads to a temporary folder individual new

    articles that have been published

  • of those new articles, checks whether they are corrections (and

    whether the linked corrected article has been updated)

  • checks whether there are VORs (Versions of Record) for uncorrected

    proofs in the main articles directory and downloads those

  • checks whether the newly downloaded articles are uncorrected proofs

    or not after all of these checks, it moves the new articles into the main articles folder.

Here’s what the print statements might look like on a typical run:

147 new articles to download.
147 new articles downloaded.
3 amended articles found.
0 amended articles downloaded with new xml.
Creating new text list of uncorrected proofs from scratch.
No new VOR articles indexed in Solr.
17 VOR articles directly downloaded.
17 uncorrected proofs updated to version of record. 44 uncorrected proofs remaining in uncorrected proof list.
9 uncorrected proofs found. 53 total in list.
Corpus started with 219792 articles.
Moving new and updated files...
164 files moved. Corpus now has 219939 articles.

How to run the tests

To run the tests, you will need to install allofplos with its testing dependencies. These testing dependencies include pytest, which we will use to run the tests.

$ pipenv run python -m pytest

It should return something like this:

collected 20 items

allofplos/tests/test_corpus.py ............                       [ 60%]
allofplos/tests/test_unittests.py ........                        [100%]

==================== 20 passed in 0.36 seconds =========================

Community guidelines

If you wish to contribute to this project please open a ticket in the GitHub repo at https://github.com/PLOS/allofplos/issues. For support requests write to mining@plos.org

Citing This Library

allofplos is published in the proceedings of the SciPy 2018. DOI 10.25080/Majora-4af1f417-009 refers to all versions of allofplos.

If you want to cite allofplos using Bibtex:

@InProceedings{ elizabeth_seiver-proc-scipy-2018,
  author    = { Elizabeth Seiver and M Pacer and Sebastian Bassi },
  title     = { Text and data mining scientific articles with allofplos },
  booktitle = { Proceedings of the 17th Python in Science Conference },
  pages     = { 61 - 64 },
  year      = { 2018 },
  editor    = { Fatih Akici and David Lippa and Dillon Niederhut and M Pacer },
  doi       = { 10.25080/Majora-4af1f417-009 }
}

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

allofplos-1.1.1.tar.gz (63.9 kB view details)

Uploaded Source

Built Distribution

allofplos-1.1.1-py3-none-any.whl (62.4 kB view details)

Uploaded Python 3

File details

Details for the file allofplos-1.1.1.tar.gz.

File metadata

  • Download URL: allofplos-1.1.1.tar.gz
  • Upload date:
  • Size: 63.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.12.9

File hashes

Hashes for allofplos-1.1.1.tar.gz
Algorithm Hash digest
SHA256 680d182a8e43ace24365b9e6c1184f474ceb40af84d08d4de680086d6a32246b
MD5 67daebb6681cea6a3570021bda33130e
BLAKE2b-256 e770169f008e89028e46f9935bd3741b8143262ffc5a5c59570bd13ae0c71d26

See more details on using hashes here.

Provenance

The following attestation bundles were made for allofplos-1.1.1.tar.gz:

Publisher: python-publish.yml on PLOS/allofplos

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file allofplos-1.1.1-py3-none-any.whl.

File metadata

  • Download URL: allofplos-1.1.1-py3-none-any.whl
  • Upload date:
  • Size: 62.4 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.12.9

File hashes

Hashes for allofplos-1.1.1-py3-none-any.whl
Algorithm Hash digest
SHA256 b034961d485f4c7e0679a6a3c815f58544596832c774f2074df54567567d4453
MD5 0ad3ec175cca1153407c93b7c48ff4a7
BLAKE2b-256 24796b4e47d468fa27a24d8ccaebe11238f0baeb33129889f7724a6936d624a5

See more details on using hashes here.

Provenance

The following attestation bundles were made for allofplos-1.1.1-py3-none-any.whl:

Publisher: python-publish.yml on PLOS/allofplos

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page