Sima Implementation

Implementation of the model from the deepmind paper "Scaling Instructable Agents Across Many Simulated Worlds" PAPER LINK

Install

$ pip3 install -U sima-torch

Usage

import torch 
from sima_torch.transformer import SimaTransformer

# Example
x = torch.randint(0, 256, (1, 1024))

# Instantiate the model
model = SimaTransformer(
    dim=512,
    enc_depth=6,
    enc_heads=8,
    dec_depth=6,
    dec_heads=8,
    tie_token_emb=False,
    num_tokens=20000,
    num_memory_tokens=20,
    encoder_dim=512,
    decoder_dim=512,
    max_seq_len=1024,
)

out = model(x)
print(out.shape)  # torch.Size([1, 1024, 512])

License

MIT

Citation

@misc{simateam2024scaling,
      title={Scaling Instructable Agents Across Many Simulated Worlds}, 
      author={SIMA Team and Maria Abi Raad and Arun Ahuja and Catarina Barros and Frederic Besse and Andrew Bolt and Adrian Bolton and Bethanie Brownfield and Gavin Buttimore and Max Cant and Sarah Chakera and Stephanie C. Y. Chan and Jeff Clune and Adrian Collister and Vikki Copeman and Alex Cullum and Ishita Dasgupta and Dario de Cesare and Julia Di Trapani and Yani Donchev and Emma Dunleavy and Martin Engelcke and Ryan Faulkner and Frankie Garcia and Charles Gbadamosi and Zhitao Gong and Lucy Gonzales and Kshitij Gupta and Karol Gregor and Arne Olav Hallingstad and Tim Harley and Sam Haves and Felix Hill and Ed Hirst and Drew A. Hudson and Jony Hudson and Steph Hughes-Fitt and Danilo J. Rezende and Mimi Jasarevic and Laura Kampis and Rosemary Ke and Thomas Keck and Junkyung Kim and Oscar Knagg and Kavya Kopparapu and Andrew Lampinen and Shane Legg and Alexander Lerchner and Marjorie Limont and Yulan Liu and Maria Loks-Thompson and Joseph Marino and Kathryn Martin Cussons and Loic Matthey and Siobhan Mcloughlin and Piermaria Mendolicchio and Hamza Merzic and Anna Mitenkova and Alexandre Moufarek and Valeria Oliveira and Yanko Oliveira and Hannah Openshaw and Renke Pan and Aneesh Pappu and Alex Platonov and Ollie Purkiss and David Reichert and John Reid and Pierre Harvey Richemond and Tyson Roberts and Giles Ruscoe and Jaume Sanchez Elias and Tasha Sandars and Daniel P. Sawyer and Tim Scholtes and Guy Simmons and Daniel Slater and Hubert Soyer and Heiko Strathmann and Peter Stys and Allison C. Tam and Denis Teplyashin and Tayfun Terzi and Davide Vercelli and Bojan Vujatovic and Marcus Wainwright and Jane X. Wang and Zhengdong Wang and Daan Wierstra and Duncan Williams and Nathaniel Wong and Sarah York and Nick Young},
      year={2024},
      eprint={2404.10179},
      archivePrefix={arXiv},
      primaryClass={cs.RO}
}

Todo

• Implement Phenaki as a video encoder
• Create mouse and keyboard policy