Skip to main content

ScienceBeam Utils

Project description

ScienceBeam Utils

License: MIT

Provides utility functions related to the ScienceBeam project.

Please refer to the development documentation if you wish to contribute to the project.

Most tools are not yet documented. Please feel free to browse the code or tests, or raise an issue.


Apache Beam may be used to for preprocessing but also its transparent FileSystems API which makes it easy to access files in the cloud.


pip install apache_beam[gcp]
pip install sciencebeam-utils

CLI Tools

Find File Pairs

The preferred input layout is a directory containing a gzipped pdf (.pdf.gz) and gzipped xml (.nxml.gz), e.g.:

  • manuscript_1/
    • manuscript_1.pdf.gz
    • manuscript_1.nxml.gz
  • manuscript_2/
    • manuscript_2.pdf.gz
    • manuscript_2.nxml.gz

Using compressed files is optional but recommended to reduce file storage cost.

The parent directory per manuscript is optional. If that is not the case then the name before the extension must be identical (which is recommended in general).


python -m \
--data-path <source directory> \
--source-pattern *.pdf.gz --xml-pattern *.nxml.gz \
--out <output file list csv/tsv>


python -m \
--data-path gs://some-bucket/some-dataset \
--source-pattern *.pdf.gz --xml-pattern *.nxml.gz \
--out gs://some-bucket/some-dataset/file-list.tsv

That will create the TSV (tab separated) file file-list.tsv with the following columns:

  • source_url
  • xml_url

That file could also be generated using any other preferred method.

Split File List

To separate the file list into a training, validation and test dataset, the following script can be used:

python -m \
--input <csv/tsv file list> \
--train 0.5 --validation 0.2 --test 0.3 --random --fill


python -m \
--input gs://some-bucket/some-dataset/file-list.tsv \
--train 0.5 --validation 0.2 --test 0.3 --random --fill

That will create three separate files in the same directory:

  • file-list-train.tsv
  • file-list-validation.tsv
  • file-list-test.tsv

The file pairs will be randomly selected (--random) and one group will also include all remaining file pairs that wouldn't get include due to rounding (--fill).

As with the previous step, you may decide to use your own process instead.

Note: those files shouldn't change anymore once you used those files

Get Output Files

Since ScienceBeam is intended to convert files, there will be output files. To make it specific what the filenames are, the output files are also kept in a file list. This tool will generate the file list (it doesn't matter whether the files actually exist for this purpose).


python -m \
  --source-file-list path/to/source/file-list-train.tsv \
  --source-file-column=source_url \
  --output-file-suffix=.xml \
  --output-file-list path/to/results/file-list.lst

By adding the --check argument, it will check whether the output files exist (see below).

Check File List

After generating an output file list, this tool can be used whether the output files exist or are complete.


python -m \
  --file-list path/to/results/file-list.lst \
  --file-column=source_url \

This will check the first 100 output files and report on it. The command will fail if none of the output files exist.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

sciencebeam_utils-0.1.5.tar.gz (30.4 kB view hashes)

Uploaded source

Built Distribution

sciencebeam_utils-0.1.5-py3-none-any.whl (44.9 kB view hashes)

Uploaded py3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page