Skip to main content
Help the Python Software Foundation raise $60,000 USD by December 31st!  Building the PSF Q4 Fundraiser

Text preprocessing, representation and visualization from zero to hero.

Project description

Texthero is a python toolkit to work with text-based dataset quickly and effortlessly. Texthero is very simple to learn and designed to be used on top of Pandas. Texthero has the same expressiveness and power of Pandas and is extensively documented. Texthero is modern and conceived for programmers of the 2020 decade with little knowledge if any in linguistic.

You can think of Texthero as a tool to help you understand and work with text-based dataset. Given a tabular dataset, it's easy to grasp the main concept. Instead, given a text dataset, it's harder to have quick insights into the underline data. With Texthero, preprocessing text data, map it into vectors and visualize the obtained vector space takes just a couple of lines.

Texthero include tools for:

  • Preprocess text data: it offers both out-of-the-box solutions but it's also flexible for custom-solutions.
  • Natural Language Processing: keyphrases and keywords extraction, named entity recognition and much more.
  • Text representation: TF-IDF, term frequency, pre-trained and custom word-embeddings.
  • Vector space analysis: clustering (K-means, Meanshift, DBSAN and Hierarchical), topic modelling (LDA and LSI) and interpretation.
  • Text visualization: keywords visualization, vector space visualization, place localization on maps.

Texthero is free, open source and well documented (and that's what we love most by the way!).

We hope you will find pleasure working with Texthero as we had during his development.

Install texthero via pip:

pip install texthero

☝️Under the hoods, Texthero makes use of multiple NLP and machine learning toolkits such as Gensim, NLTK, SpaCy and scikit-learn. You don't need to install them all separately, pip will take care of that.

For fast performance, make sure you have installed Spacy version >= 2.2. Also, make sure you have a recent version of python, the higher, the best.

The best way to learn Texthero is through the Getting Started docs.

In case you are an advanced python user, then help(texthero) should do the work.

import texthero as hero
import pandas as pd

df = pd.read_csv(
   "https://github.com/jbesomi/texthero/raw/master/dataset/bbcsport.csv"
)

df['pca'] = (
   df['text']
   .pipe(hero.clean)
   .pipe(hero.tfidf)
   .pipe(hero.pca)
)
hero.scatterplot(df, 'pca', color='topic', title="PCA BBC Sport news")
import texthero as hero
import pandas as pd

df = pd.read_csv(
    "https://github.com/jbesomi/texthero/raw/master/dataset/bbcsport.csv"
)

df['tfidf'] = (
    df['text']
    .pipe(hero.clean)
    .pipe(hero.tfidf)
)

df['kmeans_labels'] = (
    df['tfidf']
    .pipe(hero.kmeans, n_clusters=5)
    .astype(str)
)

df['pca'] = df['tfidf'].pipe(hero.pca)

hero.scatterplot(df, 'pca', color='kmeans_labels', title="K-means BBC Sport news")
>>> import texthero as hero
>>> import pandas as pd
>>> text = "This sèntencé    (123 /) needs to [OK!] be cleaned!   "
>>> s = pd.Series(text)
>>> s
0    This sèntencé    (123 /) needs to [OK!] be cleane...
dtype: object

Remove all digits:

>>> s = hero.remove_digits(s)
>>> s
0    This sèntencé    (  /) needs to [OK!] be cleaned!
dtype: object

Remove digits replace only blocks of digits. The digits in the string "hello123" will not be removed. If we want to remove all digits, you need to set only_blocks to false.

Remove all type of brackets and their content.

>>> s = hero.remove_brackets(s)
>>> s 
0    This sèntencé    needs to  be cleaned!
dtype: object

Remove diacritics.

>>> s = hero.remove_diacritics(s)
>>> s 
0    This sentence    needs to  be cleaned!
dtype: object

Remove punctuation.

>>> s = hero.remove_punctuation(s)
>>> s 
0    This sentence    needs to  be cleaned
dtype: object

Remove extra white-spaces.

>>> s = hero.remove_whitespace(s)
>>> s 
0    This sentence needs to be cleaned
dtype: object

Sometimes we also wants to get rid of stop-words.

>>> s = hero.remove_stopwords(s)
>>> s
0    This sentence needs cleaned
dtype: object

Texthero is composed of four modules: preprocessing.py, nlp.py, representation.py and visualization.py.

Scope: prepare text data for further analysis.

Full documentation: preprocessing

Scope: provide classic natural language processing tools such as named_entity and noun_phrases.

Full documentation: nlp

Scope: map text data into vectors and do dimensionality reduction.

Supported representation algorithms:

  1. Term frequency (count)
  2. Term frequency-inverse document frequency (tfidf)

Supported clustering algorithms:

  1. K-means (kmeans)
  2. Density-Based Spatial Clustering of Applications with Noise (dbscan)
  3. Meanshift (meanshift)

Supported dimensionality reduction algorithms:

  1. Principal component analysis (pca)
  2. t-distributed stochastic neighbor embedding (tsne)
  3. Non-negative matrix factorization (nmf)

Full documentation: representation

Scope: summarize the main facts regarding the text data and visualize it. This module is opinionable. It's handy for anyone that needs a quick solution to visualize on screen the text data, for instance during a text exploratory data analysis (EDA).

Supported functions:

  • Text scatterplot (scatterplot)
  • Most common words (top_words)

Full documentation: visualization

Sometimes we just want things done, right? Texthero help with that. It helps makes things easier and give to the developer more time to focus on his custom requirements. We believe that start cleaning text should just take a minute. Same for finding the most important part of a text and same for representing it.

In a very pragmatic way, texthero has just one goal: make the developer spare time. Working with text data can be a pain and in most cases, a default pipeline can be quite good to start. There is always the time to come back and improve the preprocessing steps for instance.

Pull requests are amazing and most welcome. Start by fork this repository and open an issue.

Texthero is also looking for maintainers and contributors. In case of interest, just drop a line at jonathanbesomi__AT__gmail.com

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for texthero, version 1.0.7
Filename, size File type Python version Upload date Hashes
Filename, size texthero-1.0.7-py3.7.egg (17.9 kB) File type Egg Python version 3.7 Upload date Hashes View
Filename, size Texthero-1.0.7-py3-none-any.whl (16.4 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size Texthero-1.0.7.tar.gz (13.0 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page