Skip to main content

"WARP loss for Pytorch. WSABIE"

Project description

WARP-Pytorch

An implementation of WARP loss which uses matrixes and stays on the GPU in PyTorch.

An implementation of WARP loss which uses matrixes and stays on the GPU in PyTorch.

This means instead of using a for-loop to find the first offending negative sample that ranks above our positive, we compute all of them at once. Only later do we find which sample is the first offender, and compute the loss with respect to this sample.

The advantage is that it can use the speedups that comes with GPU-usage.

When is WARP loss advantageous?

If you're ranking items or making models for recommendations, it's often advantageous to let your loss function directly optimize for this case. WARP loss looks at 1 explicit positive up against the implicit negative items that a user never sampled, and allows us to adjust weights of the network accordingly.

Install

pip install warp_loss

How to use

The loss function requires scores for both positive examples, and negative examples to be supplied, such as in the example below.

from torch import nn
import torch

class OurModel(nn.Module):
    def __init__(self, num_labels, emb_dim=10):
        super(OurModel, self).__init__()
        self.emb = nn.Embedding(num_labels, emb_dim)
        self.user_embs = nn.Embedding(1, emb_dim)

    def forward(self, pos, neg):
        batch_size = neg.size(0)
        one_user_vector = self.user_embs(torch.zeros(1).long())
        repeated_user_vector = one_user_vector.repeat((batch_size, 1)).view(batch_size, -1, 1)
        pos_res = torch.bmm(self.emb(pos), repeated_user_vector).squeeze(2)
        neg_res = torch.bmm(self.emb(neg), repeated_user_vector).squeeze(2)

        return pos_res, neg_res

num_labels = 100
model = OurModel(num_labels)
pos_labels = torch.randint(high=num_labels, size=(3,1)) # our five labels
neg_labels = torch.randint(high=num_labels, size=(3,2)) # a few random negatives per positive

pos_res, neg_res = model(pos_labels, neg_labels)
print('Positive Labels:', pos_labels)
print('Negative Labels:', neg_labels)
print('Model positive scores:', pos_res)
print('Model negative scores:', neg_res)
loss = warp_loss(pos_res, neg_res, num_labels=num_labels, device=torch.device('cpu'))
print('Loss:', loss)
loss.backward()
Positive Labels: tensor([[65],
        [94],
        [21]])
Negative Labels: tensor([[ 8, 45],
        [37, 93],
        [88, 84]])
Model positive scores: tensor([[-3.7806],
        [-1.9974],
        [-4.1741]], grad_fn=<SqueezeBackward1>)
Model negative scores: tensor([[-1.5696, -4.4905],
        [-1.9300, -0.3826],
        [ 2.4564, -2.1741]], grad_fn=<SqueezeBackward1>)
Loss: tensor(54.7226, grad_fn=<SumBackward0>)
print('We can also see that the gradient is only active for 2x the number of positive labels:', (model.emb.weight.grad.sum(1) != 0).sum().item())
print('Meaning we correctly discard the gradients for all other than the offending negative label.')
We can also see that the gradient is only active for 2x the number of positive labels: 6
Meaning we correctly discard the gradients for all other than the offending negative label.

Assumptions

The loss function assumes you have already sampled your negatives randomly.

As an example this could be done in your dataloader:

  1. Assume we have a total dataset of 100 items
  2. Select a positive sample with index 8
  3. Your negatives should be a random selection from 0-100 excluding 8.

Ex input to loss function: model scores for pos: [8] neg: [88, 3, 99, 7]

Currently only tested on PyTorch v0.4

References

Project details


Release history Release notifications

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for warp-loss, version 0.0.1
Filename, size File type Python version Upload date Hashes
Filename, size warp_loss-0.0.1-py3-none-any.whl (6.0 kB) File type Wheel Python version py3 Upload date Hashes View hashes
Filename, size warp_loss-0.0.1.tar.gz (5.2 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page