Skip to main content

Sanitise text while keeping your sanity

Project description

# Saniti

Sanitise lists of text documents quickly, easily and whilst maintaining your sanity

The aim was to streamline processing lists of documents into the same outputs into simply specifying the list of texts and defining the sanitization pipeline.

### Usage:

As a function-ish

>original_text = [“I like to moves it, move its”, “I likeing to move it!”, “the of”]

>text = saniti(original_text, [“token”, “destop”, “depunct”, “unempty”, “stem”, “out_corp_dict”]) #sanitise the text while initalising the class

>print(text.text)

As a class

>sani1 = saniti() # initialise the santising class

>text = sani1.process(original_text, [“token”, “destop”, “depunct”, “unempty”, “lemma”, “out_tag_doc”]) # sanitise the text

>print(text)

## Pipeline Components

  • “token” - tokenise texts

  • “depunct” - remove punctuation

  • “unempty” - remove empty words within documents

  • “lemma” - lemmatize text

  • “destop” - remove stopwords

  • “stem” - stem texts

  • “out_tag_doc” - turns the texts into gensim tagged documents for Doc2Vec

  • “out_corp_dict” - turns the texts into gensim corpus and dictionary

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

saniti-0.1.4.tar.gz (3.0 kB view details)

Uploaded Source

File details

Details for the file saniti-0.1.4.tar.gz.

File metadata

  • Download URL: saniti-0.1.4.tar.gz
  • Upload date:
  • Size: 3.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for saniti-0.1.4.tar.gz
Algorithm Hash digest
SHA256 f908d4706dc78e1a03e48e55043f82e0b43c5bf175b2079dcb72b97321009f86
MD5 876d426dbbeaf43958da1d0c5f114f22
BLAKE2b-256 335190d0cb8a0bbc99b9bb2a4165ff9046ebc1a31465bd6bd4b6d35ec5d67479

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page