Text processing utils

Small helpers for NLP tasks.

Installation

Create and activate a virtual environment.
Then, install the package via pip and download the spaCy pipeline.

pip install text-processing-utils
python3 -m spacy download en_core_web_md

Contents

• batches
  • get_batches_of_strict_size_with_remainder
  • get_n_batches
  • get_batches_of_roughly_equal_size
• boolean_checks
  • is_gibberish
  • is_plural
  • an_vs_a
• char_offsets
  • is_inside
  • get_span_distance_sorted
  • get_span_distance
  • remove_whitespace_from_annotation
  • merge_annotation_offsets
• bio_tags
  • bio_tags_to_spans
  • remove_overlapping_bio_tags
  • transform_into_char_offsets_and_readable_tag
  • token_spans_to_char_annotations
• locate
  • get_sent_idx
  • locate_span_in_context
• highlight_context
  • enclose_with_special_symbol
• sentences
  • lower_first_letter_if_sent_start
  • correct_sentence_boundary_detection
• regex
  • make_named_group_unique
• types
  • Offset
  • Annotations

About Us

We are the Institute of Climate and Energy Systems (ICE) - Jülich Systems Analysis belonging to the Forschungszentrum Jülich. Our interdisciplinary department's research is focusing on energy-related process and systems analyses. Data searches and system simulations are used to determine energy and mass balances, as well as to evaluate performance, emissions and costs of energy systems. The results are used for performing comparative assessment studies between the various systems. Our current priorities include the development of energy strategies, in accordance with the German Federal Government’s greenhouse gas reduction targets, by designing new infrastructures for sustainable and secure energy supply chains and by conducting cost analysis studies for integrating new technologies into future energy market frameworks.

Acknowledgements

The authors would like to thank the German Federal Government, the German state governments, and the Joint Science Conference (GWK) for their funding and support as part of the NFDI4Ing consortium. Funded by the German Research Foundation (DFG) – project number: 442146713. Furthermore, this work was supported by the Helmholtz Association under the program "Energy System Design".