Transfusion in Pytorch
Project description
Transfusion - Pytorch (wip)
Pytorch implementation of Transfusion, "Predict the Next Token and Diffuse Images with One Multi-Modal Model", from MetaAI.
Once completed, will also extend this to flow matching, as well as audio, video, perhaps even policies.
Install
$ pip install transfusion-pytorch
Usage
One modality, say images
import torch
from transfusion_pytorch import Transfusion
model = Transfusion(
num_text_tokens = 256,
dim_latent = 192,
transformer = dict(
dim = 512,
depth = 8
)
)
text_ids = torch.randint(0, 256, (2, 1024))
modality_tokens = [[
torch.randn(6, 192),
torch.randn(4, 192)
], [
torch.randn(5, 192),
torch.randn(3, 192)
]]
modality_positions = [[
(2, 6),
(10, 4)
], [
(2, 5),
(10, 3)
]] # (offset, length)
loss, breakdown = model(
text_ids,
modality_tokens = modality_tokens,
modality_positions = modality_positions
)
loss.backward()
Multiple modalities
import torch
from transfusion_pytorch import Transfusion
model = Transfusion(
num_text_tokens = 256,
dim_latent = (384, 192),
transformer = dict(
dim = 512,
depth = 8
)
)
text_ids = torch.randint(0, 256, (2, 1024))
modality_tokens = [[
torch.randn(6, 384),
torch.randn(4, 192)
], [
torch.randn(5, 192),
torch.randn(3, 384)
]]
modality_positions = [[
(0, 2, 6),
(1, 10, 4)
], [
(1, 2, 5),
(0, 10, 3)
]] # (type, offset, length)
loss, breakdown = model(
text_ids,
modality_tokens = modality_tokens,
modality_positions = modality_positions
)
loss.backward()
Citations
@inproceedings{Zhou2024TransfusionPT,
title = {Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model},
author = {Chunting Zhou and Lili Yu and Arun Babu and Kushal Tirumala and Michihiro Yasunaga and Leonid Shamis and Jacob Kahn and Xuezhe Ma and Luke Zettlemoyer and Omer Levy},
year = {2024},
url = {https://api.semanticscholar.org/CorpusID:271909855}
}
@misc{Rubin2024,
author = {Ohad Rubin},
url = {https://medium.com/@ohadrubin/exploring-weight-decay-in-layer-normalization-challenges-and-a-reparameterization-solution-ad4d12c24950}
}
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
transfusion_pytorch-0.0.6.tar.gz
(346.3 kB
view hashes)
Built Distribution
Close
Hashes for transfusion_pytorch-0.0.6.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8e1e15e741c983abe8a9f50a3b77d9bd0d6b6e393cc123720873b8df6cee43c3 |
|
MD5 | f1a50aa1401eaf4c9c635541552edfec |
|
BLAKE2b-256 | df53e886b932bf27ea6d1b09c11d98ebee099a9ec7f8d9a98e77a2f67d503800 |
Close
Hashes for transfusion_pytorch-0.0.6-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6c81cb4e02927822468ce55f59e472b5729e780d14820d311383badc3687e23d |
|
MD5 | f12d6acabe5bd18d213ae2e907e9e6e7 |
|
BLAKE2b-256 | a2bc70c4b11c8264d7ab41a2af7e88d23348aa326440e73d0c24f428f6b9b703 |