data-preprocessors 0.16.0

An easy to use tool for Data Preprocessing specially for Text Preprocessing

An easy to use tool for Data Preprocessing specially for Text Preprocessing

Table of Contents

• Installation
• Quick Start
• Functions
  • Split Textfile
  • Parallel Corpus Builder
  • Remove Punc

Installation

Install the latest stable release
For windows
$ pip install -U data-preprocessors

For Linux/WSL2
$ pip3 install -U data-preprocessors

Quick Start

from data_preprocessors import text_preprocessor as tp
sentence = "bla! bla- ?bla ?bla."
sentence = tp.remove_punc(sentence)
print(sentence)

>> bla bla bla bla

Functions

Split Textfile

from data_preprocessors import text_preprocessor as tp
tp.split_textfile(
    main_file_path="example.txt",
    train_file_path="splitted/train.txt",
    val_file_path="splitted/val.txt",
    test_file_path="splitted/test.txt",
    train_size=0.6,
    val_size=0.2,
    test_size=0.2,
    shuffle=True,
    seed=42
)

# Total lines:  500
# Train set size:  300
# Validation set size:  100
# Test set size:  100