osc-transformer-presteps 0.1.9

Install via PyPi

You can simply install the package via:

$ pip install osc-transformer-presteps

Afterwards, you can use the tooling as a CLI tool by typing:

$ osc-transformer-presteps

We are using Typer to provide a user-friendly CLI. All details and help will be shown within the CLI itself and are not described here in more detail.

Example 1: Extracting Data from PDFs

Assume the folder structure is as follows:

project/
├-input/
│ ├-file_1.pdf
│ ├-file_2.pdf
│ └─file_3.pdf
├-logs/
└─output/

Now, after installing osc-transformer-presteps, run the following command to extract data from the PDFs to JSON:

$ osc-transformer-presteps extraction run-local-extraction 'input' --output-folder='output' --logs-folder='logs' --force

Note: The --force flag overcomes encryption. Please check if this is a legal action in your jurisdiction.

Example 2: Curating a New Training Data Set for Relevance Detector

To perform curation, you will need a KPI mapping file and an annotations file. Here are examples of such files:

KPI Mapping File:

• kpi_id: The unique identifier for each KPI.

• question: The specific question being asked to extract relevant information.

• sectors: The industry sectors to which the KPI applies.

• add_year: Indicates whether to include the year in the extracted data (TRUE/FALSE).

• kpi_category: The category of the KPI, typically specifying the data type (e.g., TEXT).

Annotation File:

• company: The name of the company being analyzed.

• source_file: The document from which data is extracted.

• source_page: The page number(s) containing the relevant information.

• kpi_id: The ID of the KPI associated with the data.

• year: The year to which the data refers.

• answer: The specific data or text extracted as an answer.

• data_type: The type of the extracted data (e.g., TEXT or TABLE).

• relevant_paragraphs: The paragraph(s) or context where the data was found.

• annotator: The person or tool that performed the annotation.

• sector: The industry sector the company belongs to.

You can find demo files in the demo/curation/input folder.

Assume the folder structure is as follows:

project/
├-input/
│ ├-data_from_extraction/
│ │ ├-file_1.json
│ │ ├-file_2.json
│ │ └─file_3.json
│ ├-kpi_mapping/
│ │ └─kpi_mapping.csv
│ ├-annotations_file/
│ │ └─annotations_file.xlsx
├-logs/
└─output/

Now, you can run the following command to curate a new training data set:

$ osc-transformer-presteps relevance-curation run-local-curation 'input/-data_from_extraction/file_1.json' 'input/annotations_file/annotations_file.xlsx' 'input/kpi_mapping/kpi_mapping.csv'

Note: The previous comment may need some adjustment when running on different machine like windows due to the slash.

Example 3: Curating a New Training Data Set for KPI Detector

To perform curation, you will need the extracted json files and kpi mappinf file and annotations file (the same as described above).

Assume the folder structure is as follows:

project/
├-input/
│ ├-data_from_extraction/
│ │ ├-file_1.json
│ │ ├-file_2.json
│ │ └─file_3.json
│ ├-kpi_mapping/
│ │ └─kpi_mapping.csv
│ ├-annotations_file/
│ │ └─annotations_file.xlsx
│ ├-relevance_detection_file/
│ │ └─relevance_detection.csv
├-logs/
└─output/

Now, you can run the following command to curate a new training data set:

$ osc-transformer-presteps kpi-curation run-local-kpi-curation  'input/annotations_file/' 'input/data_from_extraction/' 'output/' 'kpi_mapping/kpi_mapping_file.csv' 'relevance_detection_file/relevance_file.xlsx'  --val-ratio 0.2 --agg-annotation "" --find-new-answerable --create-unanswerable

Note: The previous comment may need some adjustment when running on different machine like windows due to the slash.

Important Note on Annotations

When performing curation, it is crucial that all JSON files used for this process are listed in the demo/curation/input/test_annotation.xlsx file. Failure to include these files in the annotation file will result in corrupted output.

Ensure that every JSON file involved in the curation process is mentioned in the annotation file to maintain the integrity of the resulting output.

Use Code Directly Without CLI via Github Repository

First, clone the repository to your local environment:

$ git clone https://github.com/os-climate/osc-transformer-presteps

We are using pdm to manage the packages and tox for a stable test framework. First, install pdm (possibly in a virtual environment) via:

$ pip install pdm

Afterwards, sync your system via:

$ pdm sync

You will find multiple demos on how to proceed in the demo folder.

pdm

To add new dependencies, use pdm. For example, you can add numpy via:

$ pdm add numpy

For more detailed descriptions, check the PDM project homepage.

tox

For running linting tools, we use tox. You can run this outside of your virtual environment:

$ pip install tox
$ tox -e lint
$ tox -e test

This will automatically apply checks on your code and run the provided pytests. See more details on tox.