textpipe 0.3.0

textpipe: clean and extract metadata from text

textpipe: clean and extract metadata from text

textpipe is a Python package for converting raw text in to clean, readable text and extracting metadata from that text. Its functionalities include transforming raw text into readable text by removing HTML tags and extracting metadata such as the number of words and named entities from the text.

Vision: the zen of textpipe

• Designed for use in production pipelines without adult supervision.
• Rechargeable batteries included: provide sane defaults and clear examples to adapt.
• A uniform interface with thin wrappers around state-of-the-art NLP packages.
• As language-agnostic as possible.
• Bring your own models.

Features

• Clean raw text by removing HTML and other unreadable constructs
• Identify the language of text
• Extract the number of words, number of sentences, named entities from a text
• Calculate the complexity of a text
• Obtain text metadata by specifying a pipeline containing all desired elements

Usage example

>>> from textpipe import doc, pipeline
>>> sample_text = 'Sample text! <!DOCTYPE>'
>>> document = doc.Doc(sample_text)
>>> print(document.clean)
'Sample text!'
>>> print(document.language)
'en'
>>> print(document.nwords)
2

>>> pipe = pipeline.Pipeline(['CleanText', 'NWords'])
>>> print(pipe(sample_text))
{'CleanText': 'Sample text!', 'NWords': 2}

Contributing

See CONTRIBUTING for guidelines for contributors.