Bert Embeddings

BERT, published by Google, is new way to obtain pre-trained language model word representation. Many NLP tasks are benefit from BERT to get the SOTA.

The goal of this project is to obtain the token embedding from BERT's pre-trained model. In this way, instead of building and do fine-tuning for an end-to-end NLP model, you can build your model by just utilizing or token embedding.

This project is implemented with @MXNet. Special thanks to @gluon-nlp team.

Install

pip install bert-embedding
# If you want to run on GPU machine, please install `mxnet-cu92`.
pip install mxnet-cu92

Usage

from bert_embedding import BertEmbedding

bert_abstract = """We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers.
 Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations by jointly conditioning on both left and right context in all layers.
 As a result, the pre-trained BERT representations can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. 
BERT is conceptually simple and empirically powerful. 
It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE benchmark to 80.4% (7.6% absolute improvement), MultiNLI accuracy to 86.7 (5.6% absolute improvement) and the SQuAD v1.1 question answering Test F1 to 93.2 (1.5% absolute improvement), outperforming human performance by 2.0%."""
sentences = bert_abstract.split('\n')
bert_embedding = BertEmbedding()
result = bert_embedding(sentences)

If you want to use GPU, please import mxnet and set context

import mxnet as mx
from bert_embedding import BertEmbedding

...

ctx = mx.gpu(0)
bert = BertEmbedding(ctx=ctx)

This result is a list of a tuple containing (tokens, tokens embedding)

For example:

first_sentence = result[0]

first_sentence[0]
# ['we', 'introduce', 'a', 'new', 'language', 'representation', 'model', 'called', 'bert', ',', 'which', 'stands', 'for', 'bidirectional', 'encoder', 'representations', 'from', 'transformers']
len(first_sentence[0])
# 18


len(first_sentence[1])
# 18
first_token_in_first_sentence = first_sentence[1]
first_token_in_first_sentence[1]
# array([ 0.4805648 ,  0.18369392, -0.28554988, ..., -0.01961522,
#        1.0207764 , -0.67167974], dtype=float32)
first_token_in_first_sentence[1].shape
# (768,)

OOV

There are three ways to handle oov, avg (default), sum, and last. This can be specified in encoding.

...
bert_embedding = BertEmbedding()
bert_embedding(sentences, 'sum')
...

Available pre-trained BERT models

	book_corpus_wiki_en_uncased	book_corpus_wiki_en_cased	wiki_multilingual	wiki_multilingual_cased	wiki_cn
bert_12_768_12	✓	✓	✓	✓	✓
bert_24_1024_16	x	✓	x	x	x

Example of using the large pre-trained BERT model from Google

from bert_embedding import BertEmbedding

bert_embedding = BertEmbedding(model='bert_24_1024_16', dataset_name='book_corpus_wiki_en_cased')

Source: gluonnlp

Metadata

Release files for bert-embedding 1.0.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for bert-embedding 1.0.1
File	Size	Uploaded
bert_embedding-1.0.1.tar.gz	7.8 kB	Mar 28, 2019	Details

Built distribution (wheel)

Table of built distributions (wheels) for bert-embedding 1.0.1
File	Interpreter	ABI	Platform	Reset
bert_embedding-1.0.1-py3-none-any.whl 13.2 kB Mar 28, 2019	Python 3	none	any	Details

Total release size: 21.0 kB

Release files / bert_embedding-1.0.1.tar.gz

Download URL	bert_embedding-1.0.1.tar.gz
Size	7.8 kB
Tags	Source
SHA-256 checksum How to use checksums	`41accd9c43953148500c878562441f46e74163c3d773a58b0382159265ac6238`
BLAKE2b-256 checksum How to use checksums	`324913f76cef121677994bb1b0e8baa8b8bf88405eb1be554925fe8682b7b71e`
Upload date	Mar 28, 2019
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/1.13.0 pkginfo/1.5.0.1 requests/2.21.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.31.1 CPython/3.5.6`

Release files / bert_embedding-1.0.1-py3-none-any.whl

Download URL	bert_embedding-1.0.1-py3-none-any.whl
Size	13.2 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`20a2aa08a5d695c4dd221082922cf8c4387c167798d37573a6a969b69d465377`
BLAKE2b-256 checksum How to use checksums	`6285e0d56e29a055d8b3ba6da6e52afe404f209453057de95b90c01475c3ff75`
Upload date	Mar 28, 2019
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/1.13.0 pkginfo/1.5.0.1 requests/2.21.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.31.1 CPython/3.5.6`