runrex 0.5.0

Library to aid in organizing, running, and debugging regular expressions against large bodies of text.

Runrex

Library to aid in organizing, running, and debugging regular expressions against large bodies of text.

Table of Contents

• About the Project
• Getting Started
  • Prerequisites
  • Installation
• Usage
• Roadmap
• Contributing
• License
• Contact
• Acknowledgements

About the Project

The goal of this library is to simplify the deployment of regular expression on large bodies of text, in a variety of input formats.

Getting Started

To get a local copy up and running follow these simple steps.

Prerequisites

• Python 3.8+
• runrex package: https://github.com/kpwhri/runrex

Installation

1. Clone the repo

   git clone https://github.com/kpwhri/runrex.git
   

2. Install requirements (requirements-dev is for test packages)

   pip install -r requirements.txt -r requirements-dev.txt
   

3. If you wish to read text from SAS or SQL, you will need to install additional requirements. These additional requirements files may be of use:
   • ODBC-connection: requirements-db.txt
   • Postgres: requirements-psql.txt
   • SAS: requirements-sas.txt
4. Run tests.

   set/export PYTHONPATH=src
   pytest tests
   

Usage

Example Implementations

• Social Isolation
• Acute Pancreatitis
• Anaphylaxis
• PCOS

Build Customized Algorithm

• Create 4 files:
  • patterns.py: defines regular expressions of interest
    • See examples/example_patterns.py for some examples
  • test_patterns.py: tests for those regular expressions
    • Why? Make sure the patterns do what you think they do
  • algorithm.py: defines algorithm (how to use regular expressions); returns a Result
    • See examples/example_algorithm.py for guidance
  • config.(py|json|yaml): various configurations defined in schema.py
    • See example in examples/example_config.py for basic config

Input Data

Accepts a variety of input formats, but will need to at least specify a document_id and document_text. The names are configurable.

Sentence Splitting

By default, the input document text is expected to have each sentence on a separate line. If a sentence splitting scheme is desired, it will need to be supplied to the application.

Schema/Examples

For more details, see the example config or consult the schema

Output Format

• Recommended output format is jsonl
  • The data can be extracted using python:

import json
with open('output.jsonl') as fh:
    for line in fh:
         data = json.loads(line)  # data is dict

• Output variables are configurable and can include:

  • id: unique id for line
  • name: document name
  • algorithm: name of algorithm with finding
  • value
  • category: name of category (usually the pattern; multiple categories contribute to an algorithm)
  • date
  • extras
  • matches: pattern matches
  • text: captured text
  • start: start index/offset of match
  • end: end index/offset of match

• Scripts to accomplish useful tasks with the output are included in the scripts directory.

Versions

Uses SEMVER.

See https://github.com/kpwhri/runrex/releases.

Roadmap

See the open issues for a list of proposed features (and known issues).

Contributing

Any contributions you make are greatly appreciated.

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

License

Distributed under the MIT License.

See LICENSE or https://kpwhri.mit-license.org for more information.

Contact

Please use the issue tracker.

Acknowledgements