Auto-CORPus

Requires Python 3.10+

The Automated pipeline for Consistent Outputs from Research Publications (Auto-CORPus) is a tool for the standardisation and conversion of publication HTML to three convenient machine-interpretable outputs to support biomedical text analytics. Firstly, Auto-CORPus can be configured to convert HTML from various publication sources to BioC format. Secondly, Auto-CORPus transforms publication tables to a JSON format to store, exchange and annotate table data between text analytics systems. Finally, Auto-CORPus extracts abbreviations declared within publication text and provides an abbreviations JSON output that relates an abbreviation with the full definition.

We present a JSON format for sharing table content and metadata that is based on the BioC format. The JSON schema for the tables JSON can be found within the keyFiles directory.

The documentation for Auto-CORPus is available on our GitHub Pages site.

Installation

Install with pip:

pip install autocorpus

If you want to be able to process PDF files (only available with Auto-CORPus >v1.1.0), you will need to install (large!) additional dependencies. To install Auto-CORPUS with PDF processing support, run:

pip install autocorpus[pdf]

Usage

You can run Auto-CORPus on a single file like so:

auto-corpus -b PMC -t "output" -f "path/to/html/file" -o JSON

Auto-CORPus can also process whole directories:

auto-corpus -b PMC -t "output" -f "path/to/directory/of/html/files" -o JSON

Available arguments

Config files

If you wish to contribute or edit a config file then please follow the instructions in the config guide.

Auto-CORPus is able to parse HTML from different publishers, which utilise different HTML structures and naming conventions. This is made possible by the inclusion of config files which tell Auto-CORPus how to identify specific sections of the article/table within the source HTML. We have supplied a config template along with example config files for PubMed Central, PLOS Genetics and Nature Genetics in the configs directory. Users of Auto-CORPus can submit their own config files for different sources via the issues tab.

Auto-CORPus recognises 2 types of input file which are:

• Full text HTML documents covering the entire article
• HTML files which describe a single table

Auto-CORPus does not provide functionality to retrieve input files directly from the publisher. Input file retrieval must be completed by the user in a way which the publisher permits.

Auto-CORPus relies on a standard naming convention to recognise the files and identify the correct order of tables. The naming convention can be seen below:

Full article HTML: {any_name_you_want}.html

• {any_name_you_want} is how Auto-CORPus will group articles and linked tables/image files

Linked table HTML: {any_name_you_want}_table_X.html

• {any_name_you_want} must be identical to the name given to the full text file followed by_table_X where X is the table number

If passing a single file via the file path then that file will be processed in the most suitable manner, if a directory is passed then Auto-CORPus will first group files based on common elements in their file name {any_name_you_want} and process all related files at once. Related files in separate directories will not be processed at the same time. Files processed at the same time will be output into the same files, an example input and output directory can be seen below:

Input:

PMC1.html
PMC1_table_1.html
PMC1_table_2.html
/subdir
    PMC1_table_3.html
    PMC1_table_4.html

Output:

PMC1_bioc.json
PMC1_abbreviations.json
PMC1_tables.json (contains table 1 & 2 and any tables described within the main text)
/subdir
    PMC1_tables.json (contains tables 3 & 4 only)

A log file is produced in the output directory providing details of the day/time Auto-CORPus was run, the arguments used and information about which files were successfully/unsuccessfully processed with a relevant error message.

For developers

This is a Python application that uses poetry for packaging and dependency management. It also provides pre-commit hooks for various linters and formatters and automated tests using pytest and GitHub Actions.

To get started:

1. Download and install Poetry following the instructions for your OS.

2. Clone this repository and make it your working directory

3. (Optionally) download private test data for additional regression tests. This uses data which cannot be redistributed publicly (only available to members of the omicsNLP organisation).

   git submodule update --init
   

4. Set up the virtual environment:

   poetry install --all-extras
   

   Note: The --all-extras flag is because of the additional dependencies required for analysing extra file types (PDF, Word, Excel, etc).

5. Activate the virtual environment (alternatively, ensure any Python-related command is preceded by poetry run):

   poetry shell
   

6. Install the git hooks:

   pre-commit install
   

7. Run the main app for a single file example:

   auto-corpus -c "autocorpus/configs/config_pmc.json" -t "output" -f "path/to/html/file" -o JSON
   

8. Run the main app for a directory of files example

   auto-corpus -c "autocorpus/configs/config_pmc.json" -t "output" -f "path/to/directory/of/html/files" -o JSON