BLOCK: Bilinear Superdiagonal Fusion for Visual Question Answering and Visual Relationship Detection
Project description
# block.bootstrap.pytorch
In Machine Learning, an important question is “How to embed two modalities in a same space”. For instance, in Visual Question Answering, one must embed the image and the question in a same bi-modal space which will be classified to provide the answer.
INSERT VQA_block.png
We introduce a novel module (BLOCK) to fuse two representations together. First, we experimentaly demonstrate that it is better than any available fusion. Secondly, we provide a therotical-grounded analysis around the notion of tensor complexity. For further details, please see:
INSERT BIB TEX
In this repo, we make available all the fusions from the state-of-the-art via a nice pip install interface. Also, we provide pretrained models and all the code needed to reproduce our experiments. The available fusions are the following:
Block
Mutan
Tucker
MCB
MLB
MFB
MFH
MLP
#### Summary
## Install
### Requirements
We don’t provide support for python2. We advise you to install python3 with Anaconda. Then, you can create an environment.
` conda create --name block python=3 source activate block `
We use Bootstrap.pytorch, a [high level framework](https://github.com/Cadene/bootstrap.pytorch) to focus on the model instead of boilerplate code.
` cd $HOME git clone https://github.com/Cadene/bootstrap.pytorch.git cd bootstrap.pytorch pip install -r requirements.txt git clone --recursive https://github.com/Cadene/block.bootstrap.pytorch.git block pip install -r block/requirements.txt `
### Data
### Pretrained models
## Documentation
### Quick examples
## Reproduce results
### Evaluate
## Authors
This repo was made by [Hedi Ben-Younes](https://twitter.com/labegne) (Sorbonne-Heuritech) and [Remi Cadene](http://remicadene.com) (Sorbonne) and their professors [Matthieu Cord](http://webia.lip6.fr/~cord) (Sorbonne) and [Nicolas Thome](http://webia.lip6.fr/~thomen) (CNAM).
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Hashes for block.bootstrap.pytorch-0.0.1-py3.7.egg
Algorithm | Hash digest | |
---|---|---|
SHA256 | f996d83b811c3f122082583b065e88a7718990cabe93a0797fe000c3aa359d8f |
|
MD5 | 79653af02d5e8d4b179f6af7f7ff2701 |
|
BLAKE2b-256 | b6d392773e89a52d0e6afb7c178adb4b55dbb299dfc23d121185ab1fdb35ca7a |