textwiser 2.0.0

TextWiser: Text Featurization Library

TextWiser: Text Featurization Library

TextWiser (AAAI'21) is a research library that provides a unified framework for text featurization based on a rich set of methods while taking advantage of pretrained models provided by the state-of-the-art libraries.

The main contributions include:

• Rich Set of Embeddings: A wide range of available embeddings and transformations to choose from.
• Fine-Tuning: Designed to support a PyTorch backend, and hence, retains the ability to fine-tune featurizations for downstream tasks. That means, if you pass the resulting fine-tunable embeddings to a training method, the features will be optimized automatically for your application.
• Parameter Optimization: Interoperable with the standard scikit-learn pipeline for hyper-parameter tuning and rapid experimentation. All underlying parameters are exposed to the user.
• Grammar of Embeddings: Introduces a novel approach to design embeddings from components. The compound embedding allows forming arbitrarily complex embeddings in accordance with a context-free grammar that defines a formal language for valid text featurization.
• GPU Native: Built with GPUs in mind. If it detects available hardware, the relevant models are automatically placed on the GPU.

TextWiser is developed by the Artificial Intelligence Center of Excellence at Fidelity Investments. Documentation is available at fidelity.github.io/textwiser. Here is the video of the paper presentation at AAAI 2021.

Quick Start

# Conceptually, TextWiser is composed of an Embedding, potentially with a pretrained model,
# that can be chained into zero or more Transformations
from textwiser import TextWiser, Embedding, Transformation, WordOptions, PoolOptions

# Data
documents = ["Some document", "More documents. Including multi-sentence documents."]

# Model: TFIDF `min_df` parameter gets passed to sklearn automatically
emb = TextWiser(Embedding.TfIdf(min_df=1))

# Model: TFIDF followed with an NMF + SVD
emb = TextWiser(Embedding.TfIdf(min_df=1), [Transformation.NMF(n_components=30), Transformation.SVD(n_components=10)])

# Model: Word2Vec with no pretraining that learns from the input data
emb = TextWiser(Embedding.Word(word_option=WordOptions.word2vec, pretrained=None), Transformation.Pool(pool_option=PoolOptions.min))

# Model: BERT with the pretrained bert-base-uncased embedding
emb = TextWiser(Embedding.Word(word_option=WordOptions.bert), Transformation.Pool(pool_option=PoolOptions.first))

# Features
vecs = emb.fit_transform(documents)

Available Embeddings

Embeddings	Notes
Bag of Words (BoW)	Supported by scikit-learn Defaults to training from scratch
Term Frequency Inverse Document Frequency (TfIdf)	Supported by scikit-learn Defaults to training from scratch
Document Embeddings (Doc2Vec)	Supported by gensim Defaults to training from scratch
Universal Sentence Encoder (USE)	Supported by tensorflow, see requirements Defaults to large v5
Compound Embedding	Supported by a context-free grammar
Word Embedding: Word2Vec	Supported by these pretrained embeddings Common pretrained options include crawl, glove, extvec, twitter, and en-news When the pretrained option is None, trains a new model from the given data Defaults to en, FastText embeddings trained on news
Word Embedding: Character	Initialized randomly and not pretrained Useful when trained for a downstream task Enable fine-tuning to get good embeddings
Word Embedding: BytePair	Supported by these pretrained embeddings Pretrained options can be specified with the string <lang>_<dim>_<vocab_size> Default options can be omitted like en, en_100, or en__10000 Defaults to en, which is equal to en_100_10000
Word Embedding: ELMo	Supported by these pretrained embeddings from TensorflowHub Defaults to original
Word Embedding: Flair	Supported by these pretrained embeddings Defaults to news-forward-fast
Word Embedding: BERT	Supported by these pretrained embeddings Defaults to bert-base-uncased
Word Embedding: OpenAI GPT	Supported by these pretrained embeddings Defaults to openai-gpt
Word Embedding: OpenAI GPT2	Supported by these pretrained embeddings Defaults to gpt2-medium
Word Embedding: TransformerXL	Supported by these pretrained embeddings Defaults to transfo-xl-wt103
Word Embedding: XLNet	Supported by these pretrained embeddings Defaults to xlnet-large-cased
Word Embedding: XLM	Supported by these pretrained embeddings Defaults to xlm-mlm-en-2048
Word Embedding: RoBERTa	Supported by these pretrained embeddings Defaults to roberta-base
Word Embedding: DistilBERT	Supported by these pretrained embeddings Defaults to distilbert-base-uncased
Word Embedding: CTRL	Supported by these pretrained embeddings Defaults to ctrl
Word Embedding: ALBERT	Supported by these pretrained embeddings Defaults to albert-base-v2
Word Embedding: T5	Supported by these pretrained embeddings Defaults to t5-base
Word Embedding: XLM-RoBERTa	Supported by these pretrained embeddings Defaults to xlm-roberta-base
Word Embedding: BART	Supported by these pretrained embeddings Defaults to facebook/bart-base
Word Embedding: ELECTRA	Supported by these pretrained embeddings Defaults to google/electra-base-generator
Word Embedding: DialoGPT	Supported by these pretrained embeddings Defaults to microsoft/DialoGPT-small
Word Embedding: Longformer	Supported by these pretrained embeddings Defaults to allenai/longformer-base-4096

Available Transformations

Transformations	Notes
Singular Value Decomposition (SVD)	Differentiable
Latent Dirichlet Allocation (LDA)	Not differentiable
Non-negative Matrix Factorization (NMF)	Not differentiable
Uniform Manifold Approximation and Projection (UMAP)	Not differentiable
Pooling Word Vectors	Applies to word embeddings only Reduces word-level vectors to document-level Pool options include max, min, mean, first, and last Defaults to max

Usage Examples

Examples can be found under the notebooks folder.

Installation

TextWiser requires Python 3.8+ and can be installed from PyPI using pip install textwiser, using pip install textwiser[full] to install from PyPI with all optional dependencies, or by building from source by following the instructions in our documentation.

Compound Embedding

A unique research contribution of TextWiser lies in its novel approach in creating embeddings from components, called the Compound Embedding.

This method allows forming arbitrarily complex embeddings, thanks to a context-free grammar that defines a formal language for valid text featurization. You can see the details in our documentation and in the usage example.

Fine-Tuning for Downstream Tasks

All Word2Vec and transformer-based embeddings and any embedding followed with an svd transformation are fine-tunable for downstream tasks. In other words, if you pass the resulting fine-tunable embedding to a PyTorch training method, the features will automatically be trained for your application. You can see the details in our documentation and in the usage example.

Tokenization

In general, text data should be whitespace-tokenized before being fed into TextWiser. Customized tokenization is also supported as described in more detail in our documentation

Support

Please submit bug reports, questions and feature requests as Issues.

Citation

If you use TextWiser in a publication, please cite it as:

  @article{textwiser2021,
    author={Kilitcioglu, Doruk and Kadioglu, Serdar},
    title={Representing the Unification of Text Featurization using a Context-Free Grammar},
    url={https://github.com/fidelity/textwiser},
    journal={Proceedings of the AAAI Conference on Artificial Intelligence},
    volume={35},
    number={17},
    year={2021},
    month={May},
    pages={15439-15445}
  }

License

TextWiser is licensed under the Apache License 2.0.