Skip to main content

Mbedder: A pytorch powered framework for seemlessly adding contextual text embeddings from pretrained models

Project description


Mbedder is a language framework for adding contextual embeddings of pretrained language models to deep learning models.Mbedder is powered by PyTorch and HuggingFace and requires as less as 1 line of code to add embeddings and works similar to how the Embedding Layer works in PyTorch.

List of supported architectures

  • Bert
  • XLNet
  • Albert
  • TransfoXL
  • DistilBert
  • Roberta
  • XLM
  • XLMRoberta
  • GPT
  • GPT2
  • Flaubert

The pretrained models for the mentioned architecures can be found here.

Features

  • Addition of embeddings with 1 line of code
  • Embeddings can output Sentence as well as Token level embeddings
  • Task specific combination strategies can be applied to hidden states and token embeddings
  • Custom pre-trained hugging face transformer models can be used with Mbedder.

Requirements and Installation

  • PyTorch version >= 1.6.0
  • Python version >= 3.6
  • Transformer >= 3.0.2

Mbedder can be using Pip as follows

pip install Mbedder

Getting Started

A basic example of using a Mbedder Bert embedding is shown below:

import torch
from Mbedder import BertEmbedding

class BertClassifier(torch.nn.Module):
    def __init__(self, num_classes):
        super(BertClassifier, self).__init__()
        self.embedding = BertEmbedding.from_pretrained('bert-base-uncased')
        self.fc = torch.nn.Linear(self.embedding.embedding_size, num_classes)
    
    def forward(self, input_ids, attention_mask):
        x = self.embedding(input_ids, attention_mask, output_token_embeddings=False)
        logits = self.fc(x[0])
        return logits

More advanced examples can be found in the examples folder.

License

Mbedder is MIT-licensed.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for Mbedder, version 0.0.4
Filename, size File type Python version Upload date Hashes
Filename, size Mbedder-0.0.4.tar.gz (5.9 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page