Transfusion in Pytorch
Project description
Transfusion - Pytorch (wip)
Pytorch implementation of Transfusion, "Predict the Next Token and Diffuse Images with One Multi-Modal Model", from MetaAI.
Once completed, will also extend this to flow matching, as well as audio, video, perhaps even policies.
Install
$ pip install transfusion-pytorch
Usage
One modality, say images
from torch import randint, randn
from transfusion_pytorch import Transfusion
model = Transfusion(
num_text_tokens = 256,
dim_latent = 384,
transformer = dict(
dim = 512,
depth = 8
)
)
text_and_images = [
[randint(0, 256, (16,)), randn(4, 384), randint(0, 256, (8,)), randn(6, 384)],
[randint(0, 256, (16,)), randn(7, 384), randint(0, 256, (5,)), randn(2, 384), randint(0, 256, (9,))]
]
loss = model(text_and_images)
loss.backward()
Multiple different modalities
from torch import randint, randn
from transfusion_pytorch import Transfusion
model = Transfusion(
num_text_tokens = 256,
dim_latent = (384, 192), # specify multiple latent dimensions
transformer = dict(
dim = 512,
depth = 8
)
)
# then for the Tensors of type float, you can pass a tuple[int, Tensor] and specify the modality index in the first position
text_images_and_audio = [
[randint(0, 256, (16,)), (0, randn(4, 384)), randint(0, 256, (8,)), (1, randn(6, 192))],
[randint(0, 256, (16,)), randn(7, 384), randint(0, 256, (5,)), (1, randn(2, 192)), randint(0, 256, (9,))]
]
loss = model(text_images_and_audio)
loss.backward()
Citations
@inproceedings{Zhou2024TransfusionPT,
title = {Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model},
author = {Chunting Zhou and Lili Yu and Arun Babu and Kushal Tirumala and Michihiro Yasunaga and Leonid Shamis and Jacob Kahn and Xuezhe Ma and Luke Zettlemoyer and Omer Levy},
year = {2024},
url = {https://api.semanticscholar.org/CorpusID:271909855}
}
@misc{Rubin2024,
author = {Ohad Rubin},
url = {https://medium.com/@ohadrubin/exploring-weight-decay-in-layer-normalization-challenges-and-a-reparameterization-solution-ad4d12c24950}
}
@article{Nguyen2024MinPS,
title = {Min P Sampling: Balancing Creativity and Coherence at High Temperature},
author = {Minh Nguyen and Andrew Baker and Andreas Kirsch and Clement Neo},
journal = {ArXiv},
year = {2024},
volume = {abs/2407.01082},
url = {https://api.semanticscholar.org/CorpusID:270870613}
}
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
transfusion_pytorch-0.0.24.tar.gz
(348.7 kB
view hashes)
Built Distribution
Close
Hashes for transfusion_pytorch-0.0.24.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | cf0597c57c5457a93e9dfdb989f74e5d7ef7f3e61e738a44aee4dd7bc122cebe |
|
MD5 | 4c0458b232ef8bbf08a86412a2b080e0 |
|
BLAKE2b-256 | 422d8ff7783fd56121e2ce25bf8eb89b8a2b38ffb155b4d7ba3d7d83e340dc12 |
Close
Hashes for transfusion_pytorch-0.0.24-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5f303c4c70de3ca235d91d8ef3c52c347287064f4d0c2006a3df0b90c9065d01 |
|
MD5 | 388d037e1e0a644ea8aef9b662a949e5 |
|
BLAKE2b-256 | df4dc6c83b61118bdf135c8bca1a69850b736d9acdb0741c6dc1ec2e989927f4 |