Skip to main content

How to represent sentences in a high-dimensional vector space, a.k.a., sentence embedding.

Project description

A Generic Sentence Embedding Library

In natural language processing, we need to vectorize or encode text data to let machine proocess it. In the past, we mostly use encoders such as one-hot, term frequency, and TF-IDF (normalized term frequency). There are many challenges with these techniques that you can read here. In the recent years, the deep learning advancements give us opportunity to encode sentences or words in more meaningful format. The word2vec library was one of the major advancement in this field. Or, the BERT language model provides us a powerful sentence encoders that can be used in many projects.

The sentence embedding or encoding is an important step of many NLP projects. Plus, we believe that a flexible sent2vec library is needed to build a prototype fast. That is why we have initiated this project. In the early releases, you will have access to the standard encoders. We will add more curated techniques in the later releases. Hope you can use this library in your exciting NLP projects.

Library

The library requires the following libraries:

  • transformers
  • pandas
  • numpy
  • torch

Install

It can be installed using pip:

pip install sent2vec

Usage

This is how to initialize the library and provide the data.

from sent2vec.vectorizer import Vectorizer

sentences = [
    "This is an awesome book to learn NLP.",
    "DistilBERT is an amazing NLP library.",
    "We can interchangeably use embedding, encoding, or vectorizing.",
]
vectorizer = Vectorizer(sentences)

If you want to use the pre-trained DistilBertModel, you should use the code below.

vectors = vectorizer.sent2vec_bert()

And, that's pretty much it!

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

sent2vec-0.0.1.tar.gz (3.3 kB view details)

Uploaded Source

Built Distribution

sent2vec-0.0.1-py3-none-any.whl (3.6 kB view details)

Uploaded Python 3

File details

Details for the file sent2vec-0.0.1.tar.gz.

File metadata

  • Download URL: sent2vec-0.0.1.tar.gz
  • Upload date:
  • Size: 3.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/49.2.0.post20200714 requests-toolbelt/0.9.1 tqdm/4.41.1 CPython/3.8.3

File hashes

Hashes for sent2vec-0.0.1.tar.gz
Algorithm Hash digest
SHA256 71b1db244d807b3e2094bae0343ed8324b7d8b5e842a3e80fd21e79ff6755cd1
MD5 f83a0f772b76296e6499049e9538d6be
BLAKE2b-256 0d00d9ac521a8bd8a7aec25e90bc7ddda35f22fe0716362cc348d6f611e69c4d

See more details on using hashes here.

File details

Details for the file sent2vec-0.0.1-py3-none-any.whl.

File metadata

  • Download URL: sent2vec-0.0.1-py3-none-any.whl
  • Upload date:
  • Size: 3.6 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/49.2.0.post20200714 requests-toolbelt/0.9.1 tqdm/4.41.1 CPython/3.8.3

File hashes

Hashes for sent2vec-0.0.1-py3-none-any.whl
Algorithm Hash digest
SHA256 8a781e91c4b2a289db2143ca5bc6cfca7c102bb9207253e98d1bbcfe5eb97b71
MD5 39823496f56ce3f1a5121ad24cc46f73
BLAKE2b-256 41f61e1a5ffc473aa849577d615b2e4bdac791f1504ef2e9c2cbbd93ceccef7d

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page