MordinezNLP 0.1.0

Powerfull python tool for modern NLP processing

MordinezNLP

Useful toolkit for NLP projects

MordinezNLP provides tools to download the data from the web, CommonCrawl and ElasticSearch using multiprocessing and custom file processing functions

MordinezNLP has is a powerful tool to clean up dirty texts to make use of them in Neural Networks with better performance.

Use MordinezNLP to extract text data from PDFs (tables ommiting) and from HTMLs.

MordinezNLP is build on top of the SpaCy and Stanza.

Quick tour

Text cleaning and POS tagging

from MordinezNLP.processors import BasicProcessor
from MordinezNLP.pipelines import PartOfSpeech
from MordinezNLP.tokenizers import spacy_tokenizer
import spacy

nlp = spacy.load("en_core_web_sm")
nlp.tokenizer = spacy_tokenizer(nlp)

bp = BasicProcessor()
post_process = bp.process("this is my text to process by a funcion", language='en')

pos_tagger = PartOfSpeech(
    nlp,
    'en'
)

pos_output = pos_tagger.process(
    [post_process],
    4,
    30,
)

CommonCrawl downloader

from MordinezNLP.downloaders import CommonCrawlDownloader

ccd = CommonCrawlDownloader(
    [
        "reddit.com/r/space/*",
        "reddit.com/r/spacex/*",
    ]
)
ccd.download('./test_data')

PDF parser

from io import BytesIO
from MordinezNLP.parsers import process_pdf

with open("my_pdf_doc.pdf", "rb") as f:
       pdf = BytesIO(f.read())
   output = process_pdf(pdf)
   print(output)

Installation

With pip

pip install MordinezNLP

URLs

• Docs
• GitHub