parscival 0.7.0

modular framework for parsing, mapping and transforming STS data

Parscival

Description

Parscival is a modular framework for ingesting, parsing, mapping, curating, validating and storing textual data. It is originally designed to process STS inputs and export them to any arbitrary format.

Data parsing and transforming is performed according to an experimental specification described in a YAML file. For an example see here.

The output data is saved by default using the HDF5 binary data format. HDF5 is an open source file format that supports large, complex, heterogeneous data. It is designed for fast I/O processing and storage.

To enable parallel (on-the-fly) access to the HDF5 data produced, Parscival uses klepto, a python library that provides fast and flexible access to large amounts of storage.

In order to define how to transform the data into an arbitrary output format, Parscival implements a lightweight plugin architecture. For example, by using the render-template plugin, the output result can be simple described as a Jinja template. For an example on how to transform the data into json see here.

Install

pip install parscival

Usage

usage: parscival [-h] [--job-id JOB_ID] [--version] [-v] [-vv] FILE_PARSER_SPEC FILE_OUTPUT FILE_DATASET [FILE_DATASET ...]

A modular framework for ingesting, parsing, mapping, curating, validating and storing heterogeneous data

positional arguments:
  FILE_PARSER_SPEC     parscival specification
  FILE_OUTPUT          processed data output
  FILE_DATASET         input dataset

options:
  -h, --help           show this help message and exit
  --job-id JOB_ID      job identifier for logging
  --version            show program's version number and exit
  -v, --verbose        set loglevel to INFO
  -vv, --very-verbose  set loglevel to DEBUG

Examples

# converts documents from pesticides-s.nbib into pesticides.cortext.json as described by pubmed-nbib.yaml
parscival -v src/parscival_specs/pubmed/pubmed-nbib.yaml /tmp/pesticides.cortext.json tests/datasets/pesticides-s.nbib

# converts documents from both pesticides-s.nbib and hetercat-s.nbib into pesticides.db as described by pubmed-nbib.yaml
parscival -v src/parscival_specs/pubmed/pubmed-nbib.yaml /tmp/pesticides.cortext.db tests/datasets/pesticides-s.nbib tests/datasets/hetercat-s.nbib

Supported formats

Sources

• PubMed (.nbib) : PubMed is a free search engine accessing primarily the MEDLINE database of references and abstracts on life sciences and biomedical topics. The parsing spec is avalaible here. You can find a more detailed description in the related documentation.

• Europresse (.html) : Europresse is a comprehensive database providing access to a vast range of news and information from various sources. The parsing specification allows for extracting structured data from Europresse HTML files.

Intermediate data

The intermediate data is stored usign the CorText Graph format:

Field	Value	Type	Description
file	sourceFile(fieldName)	text	source file for the data
id	fieldName.doc[0,n-1]	integer	ID of each document
rank	fieldName.doc[id][0,m-1]	integer	field cardinal index
parserank	fieldName.doc[id][rank][0,p-1]	integer	parsed cardinal index
data	fieldName.doc[id][rank][parserank]	[text,integer]	parsed data

Output

• cortext.json: intermediate data is converted to json using the cortext.json template

• cortext.sqlite: intermediate data is converted to a sqlite script using the cortext.sqlite template. If requested by the processing spec, the resulting sqlite script can be intepreted and thus converted to a binary database.

Requirements

Parscival has been set up using PyScaffold, a project generator for bootstrapping high-quality Python packages. For details and usage information on PyScaffold see https://pyscaffold.org.

This project uses PyScaffold in combination with Tox, a generic virtualenv management and test command line tool acting as frontend to Continuous Integration servers. A list with all the available tasks is obtained via the tox -av command.

To prepare your environment you will need to install the following dependencies:

pip install -U pip setuptools
pip install -U tox

Deployment

virtualenv .venv
source .venv/bin/activate
# ... edit setup.cfg to add dependencies ...
pip install -e .
tox

# to build distribution
tox -e build

Documentation

In order to compile the Parscival documentation you must type:

tox -e docs

Dependences

• libhdf5-dev: Provides the development files for the HDF5 (Hierarchical Data Format version 5) library. HDF5 is designed to store and organize large amounts of data, making it suitable for high-performance data processing applications.

Environment variables

• PARSCIVAL_PLUGINS_PATHS: Specifies the directories where Parscival should look for plugins.

• PARSCIVAL_PLUGIN_RENDER_TEMPLATE_DIR: Specifies the directory where Parscival should look for default rendering templates used by plugins.

• PARSCIVAL_LOG_PATH Specifies the directory where Parscival should keep the logging activity.

Learn more

To learn more about Parscival, compile the documentation by executing the following command: tox -e docs

Alternatively, you may directly refer to some raw documentation pages linked below:

General

• How to process HTML documents with Parscival
• How to process plain text key-value documents with Parscival

Plugins

• Parsing
• Mapping
• Curating
• Storing

Parscival specification examples

• Europresse (.html)
• Pubmed (.nbib)

Credits

Parscival is being developed by the CorTexT Platform and Cogniteva SAS.