ohcrn-lei 0.3.0

The ohcrn-lei tool takes a PDF or TXT file of a clinical report and extracts desired information from it, which will be output in json format.

🌺🌺🌺🌺🌺🌺 OHCRN-LEI 🌺🌺🌺🌺🌺🌺

The Ontario Hereditary Cancer Research Network - LLM-based Extraction of Information

Overview • Getting started • Contributing • Citation

Status

Overview

The ohcrn-lei processes PDF or TXT files of clinical reports and extracts desired information from it, which will be output in json format.

LEI currently supports the following built-in extraction tasks:

• Report: Extracts the following data:
  1. "report_date" (Collected On, Received On, etc in "YYYY-MM-DD" format).
  2. "report_type": Type of report.
  3. "testing_context": Purpose of the testing (clinical or research).
  4. "ordering_clinic": Name of the lab that ordered the test.
  5. "testing/laboratory": Name of the lab that conducted the test.
• Molecular test:
  1. Sequencing Scope: Gene panel, Targeted variant testing, Whole exome sequencing (WES), Whole genome sequencing (WGS), Whole transcriptome sequencing (WTS).
  2. Tested Genes
  3. Sample Type: Amplified DNA, ctDNA, Other DNA enrichments, Other RNA fractions, polyA+ RNA, Ribo-Zero RNA, Total DNA, Total RNA.
  4. Analysis Type: Variant analysis, Microarray, Repeat expansion analysis, Karyotyping, Fusion analysis, Methylation analysis.
• Variants:
  1. Variant Identifiers: e.g OMIM, Clinvar, dbSNP, etc
  2. Gene Symbol
  3. Transcript ID: NCBI or LRG.
  4. Variant descriptor in genomic context (gHGVS):
  5. Variant descriptor in coding sequence context (cHGVS)
  6. Amino Acid Change (pHGVS)
  7. Chromosome: The chromosome identifier.
  8. Exon: The exon number
  9. Reference genome build: ("GRCh37","GRCh38")

In addition to the built-in extraction tasks, additional tasks can also be provided via a task definition file (via the -t option). See below for the task definition file format requirements.

Getting started

Prerequisites

OpenAI account

OHCRN-LEI uses GPT4o and thus requires an OpenAI account token. If you do not have an account, please create one on the OpenAI webpage. After verifying your account, log in, navigate to the API Keys section in your dashboard, and click on "Create new secret key" to generate your key.

Copy the key and store it in an environment variable named OPENAI_API_KEY. To do so, open your .bashrc (or .zshenv if you use zsh) and add the following entry:

export OPENAI_API_KEY=your-api-key-here

Poppler

OHCRN-LEI requires poppler to be installed. If you don't have poppler installed already, you can do so as follows:

MacOS (via homewbrew):

$ brew install poppler

Ubuntu/Debian:

$ sudo apt install poppler-utils

UV

We also recommend uv for a faster and easier installation process. (It only takes seconds!)

MacOS (via homewbrew):

$ brew install uv

Linux:

$ curl -LsSf https://astral.sh/uv/install.sh | sh

Installation

Preliminarily: Install directly from github via uv:

$ uv tool install "ohcrn-lei @ git+https://github.com/courtotlab/ohcrn_lei"

Usage

After installation, you can run the ohcrn-lei in your command line. For example, to run the report extraction task on the file example.pdf, run:

ohrcn-lei --task report -outfile output.json example.pdf

Currently, the report, molecular_test and variant task are supported out-of-the-box. However, you can also create your own custom tasks.

The full set of parameters can be found below:

usage: ohcrn-lei [-h] [-b PAGE_BATCH] [-t TASK] [-o OUTFILE]
                 [--mock-LLM] [--no-ocr]
                 filename

Extract data from report file.

positional arguments:
  filename              Path to the report file to process.

options:
  -h, --help            show this help message and exit
  -b, --page-batch PAGE_BATCH
                        Number of pages to be processed at a given
                        time. Default=2
  -t, --task TASK       Specify the extraction task. This can either be
                        a pre-defined task
                        ('report','molecular_test','variant')or a plain
                        *.txt file with a task definition. See
                        documentationfor the task definition file
                        format specification.Default: report
  -o, --outfile OUTFILE
                        Output file or '-' for stdout (default)
  --mock-LLM            Don't make real LLM call, produce mock output
                        instead.
  --no-ocr              Disable OCR processing.

Creating custom tasks

To create a new extraction task from scratch, you can create a new task definition file. The task definition file follows the following format

##### START PROMPT #####
Enter your LLM prompt here. Must instruct the LLM to generate a JSON dictionary output.
##### END PROMPT #####
##### START PLUGINS #####
json_key=plugin_name
...
##### END PLUGINS #####

The following plugins are supported:

• trie_hgnc : Extracts HGNC gene symbols and aliases using a Trie search algorithm.
• regex_hgvsg : Extracts genomic HGVS strings using a regular expression search.
• regex_hgvsc : Extracts coding sequence HGVS strings using a regular expression search.
• regex_hgvsp : Extracts protein-level HGVS strings using a regular expression search.
• regex_variants : Extracts variant IDs (OMIM,dbSNP,etc.) using a regular expression search.
• regex_chromosome : Extracts chromosome identifiers using a regular expression search.

Contributing

Build process

To build the project from source, use uv

$ git clone https://github.com/courtotlab/ohcrn_lei.git
$ cd ohcrn_lei
$ uv build

The resulting .whl file will be n the dist/ directory.

Code quality

Unit tests are located in the tests/ directory. To run them use

$ uv run pytest

You can lint and format the code with ruff:

$ uv run ruff check
$ uv run ruff format

Pull requests

If you are an outside contributor, you'll want to fork the repo first. Members of the Courtot Lab can create a branch on this repo instead.

1. Create your feature branch (git checkout -b feature/fooBar)
2. Commit your changes (git commit -am 'Add some fooBar')
3. Push to the branch (git push origin feature/fooBar)
4. Test
5. Open a pull request.

Authors

• Andres Melani de la Hoz
• Jochen Weile

Citation

Coming soon.