Concatenate word and character embeddings in Keras
Project description
Introduction
Out-of-vocabulary words have bad effects on word embeddings. Sometimes both word and character features are used. The characters in a word are first mapped to character embeddings, then a bidirectional recurrent neural layer is used to encode the character embeddings to a single vector. The final feature of a word is the concatenation of word embedding and encoded character feature.
The repository contains some functions which could be used to generate the first few layers that encodes the features of words.
Install
pip install keras_word_char_embd
Demo
There is a sentiment analysis demo in the demo directory. Run the following commands, then your model should have about 70% accuracy:
cd demo
./get_data.sh
python sentiment_analysis.py
Functions
This section only introduces the basic usages of the functions. For more detailed information please refer to the demo and the doc comments describing the functions in the source code.
get_dicts_generator
The function returns a closure used to generate word and character dictionaries. The closure should be invoked for all the training sentences in order to record the frequencies of each word or character. After that, setting the parameter return_dict=True the dictionaries would be returned.
from keras_wc_embd import get_dicts_generator
sentences = [
['All', 'work', 'and', 'no', 'play'],
['makes', 'Jack', 'a', 'dull', 'boy', '.'],
]
dict_generator = get_dicts_generator(
word_min_freq=2,
char_min_freq=2,
word_ignore_case=False,
char_ignore_case=False,
)
for sentence in sentences:
dict_generator(sentence)
word_dict, char_dict, max_word_len = dict_generator(return_dict=True)
You can generate dictionaries on your own, but make sure index 0 and index for <UNK> are preserved.
get_embedding_layer
Generate the first few layers that encodes words in a sentence:
import keras
from keras_wc_embd import get_embedding_layer
inputs, embd_layer = get_embedding_layer(
word_dict_len=len(word_dict),
char_dict_len=len(char_dict),
max_word_len=max_word_len,
word_embd_dim=300,
char_embd_dim=50,
char_hidden_dim=150,
rnn='lstm',
)
model = keras.models.Model(inputs=inputs, outputs=embd_layer)
model.summary()
The output shape of embd_layer should be (None, None, 600), which represents the batch size, the length of sentence and the length of encoded word feature.
get_batch_input
The function is used to generate the batch inputs for the model.
from keras_wc_embd import get_batch_input
word_embd_input, char_embd_input = get_batch_input(
sentences,
max_word_len=max_word_len,
word_dict=word_dict,
char_dict=char_dict,
)
get_embedding_weights_from_file
A helper function that loads pre-trained embeddings for initializing the weights of the embedding layer. The format of the file should be similar to GloVe.
from keras_wc_embd import get_embedding_layer, get_embedding_weights_from_file
word_embd_weights = get_embedding_weights_from_file(word_dict, 'glove.6B.100d.txt', ignore_case=True)
inputs, embd_layer = get_embedding_layer(
word_dict_len=len(word_dict),
char_dict_len=len(char_dict),
max_word_len=max_word_len,
word_embd_dim=300,
char_embd_dim=50,
char_hidden_dim=150,
word_embd_weights=word_embd_weights,
rnn='lstm',
)
Citation
Several papers have done the same thing. Just choose the one you have seen.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for keras-word-char-embd-0.0.5.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | b801c877f4a6260f25502882f252a25cdeef993053f5d313896c5c7ac971da8d |
|
MD5 | 34f59be596d806262a97fa599816cb48 |
|
BLAKE2b-256 | 87c9bddda5a7abbf31f00fba133c1b6cd8e408ad12fe72848038e0748afe240d |
Hashes for keras_word_char_embd-0.0.5-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b1f119fb27ae2f4698b08438232c41d8a1b6c4359e1e4ffecd8cbd8ca60707fb |
|
MD5 | 27701d91d64d551e1d23a56fe1b698da |
|
BLAKE2b-256 | 3a3ee8d4f1e8c560ad3d4a6dfdd11a2169fc7d1ea09337530a138850d5b7317c |