Mbedder: A pytorch powered framework for seemlessly adding contextual text embeddings from pretrained models
Mbedder is a language framework for adding contextual embeddings of pretrained language models to deep learning models.Mbedder is powered by PyTorch and HuggingFace and requires as less as 1 line of code to add embeddings and works similar to how the Embedding Layer works in PyTorch.
List of supported architectures
The pretrained models for the mentioned architecures can be found here.
- Addition of embeddings with 1 line of code
- Embeddings can output Sentence as well as Token level embeddings
- Task specific combination strategies can be applied to hidden states and token embeddings
- Custom pre-trained hugging face transformer models can be used with Mbedder.
Requirements and Installation
- PyTorch version >= 1.6.0
- Python version >= 3.6
- Transformer >= 3.0.2
Mbedder can be using Pip as follows
pip install Mbedder
A basic example of using a Mbedder Bert embedding is shown below:
import torch from Mbedder import BertEmbedding class BertClassifier(torch.nn.Module): def __init__(self, num_classes): super(BertClassifier, self).__init__() self.embedding = BertEmbedding.from_pretrained('bert-base-uncased') self.fc = torch.nn.Linear(self.embedding.embedding_size, num_classes) def forward(self, input_ids, attention_mask): x = self.embedding(input_ids, attention_mask, output_token_embeddings=False) logits = self.fc(x) return logits
More advanced examples can be found in the examples folder.
Mbedder is MIT-licensed.
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.