seamstress-world-models 1.0.4

For investigating the manifold geometry of complex dynamical systems and using this information to learn improved world models.

Seamstress

It's this easy:

# Import pip installed modules
from seamstress import WorldModel

# Load weights from hugging face
model = WorldModel.from_pretrained("isaac-ronald-ward/seamstress-rotorcraft").eval()

# Pass in the context data, and the intended action sequence
# you want to imagine the outcome of
pred = model.imagine(
    # (context_len, |states_numeric|)
    past_states_numeric=...,  
    # (context_len, image_channels, image_height, image_width)
    past_states_image=..., 
    # (context_len, |actions|)
    past_actions=...,
    future_actions=...
)

# Inspect the imagined outputs
# (future_len, |states_numeric|)
pred["pred_future_states_numeric"]  
# (future_len, image_channels, image_height, image_width)
pred["pred_future_states_image"]    

Complete the instructions under installation before proceeding.

For the full inference walkthrough, including loading the shipped demo trajectory and writing predicted videos, see docs/README.inference.md.

Installation

Git LFS

Git Large File Storage (LFS) is used to manage large files in this repository. After cloning the repository, make sure to install Git LFS and pull the large files:

git lfs install
git lfs pull

Docker

Docker is used for reproducibility and ease of setup, and should be installed if you don't have it already.

We provide a development container style setup that bind-mounts the repo into the container, so edits on the host are reflected immediately inside the container, and vice versa.

To build and run the GPU container (requires NVIDIA drivers + nvidia-container-toolkit):

# The build step takes a while the first time round. Subsequent usage only requires the run command. For now, go grab a cuppa!
docker compose -f docker/compose.seamstress.yml build --no-cache seamstress-gpu
docker compose -f docker/compose.seamstress.yml run --rm -it seamstress-gpu bash

# Ensure that your GPUs are visible inside the container with:
nvidia-smi

This repository concerns the training of deep learning-based world models, so we recommend using a GPU. That said, we also provide a CPU container for development and testing:

# The build step takes a while the first time round. Subsequent usage only requires the run command. For now, go grab a cuppa!
docker compose -f docker/compose.seamstress.yml build --no-cache seamstress-cpu
docker compose -f docker/compose.seamstress.yml run --rm -it seamstress-cpu bash

Standard Docker commands apply:

# Leave the container
exit
# List all the containers and find the desired container id
docker ps
# Use the container id to start a new shell in the running container
docker exec -it ae8c43356adf bash
# Stop a running container with the container id
docker stop ae8c43356adf
# Stop all containers
docker stop $(docker ps -q)

Environment variables

Finally, copy .env.example to .env and fill in required fields (Weights & Biases, HuggingFace tokens, etc.). Note that nothing under the blocks:

# ----------------------------------------------------------------------
# Optional FiGS sidecar integration (GPU-only)
# ----------------------------------------------------------------------

# ----------------------------------------------------------------------
# The following can remain unchanged unless you are tinkering
# ----------------------------------------------------------------------

Need changing for standard usage, but feel free to edit these for tinkering and ablations.

Getting started

We provide the following specific guides to get started with different aspects of the project, recommending that you always start with 1.

1. Running the full workflow on a toy environment
2. Loading a pretrained Seamstress model and using it for inference
3. Using 'Flying in Gaussian Splats' for high fidelity quadrotor simulations
4. Training on real world data
5. Using your own data

Convienience commands

To delete all the logs and training artefacts in the repo use:

rm -rf ./logs/*
rm -rf ./wandb/*
rm -rf ./outputs/*

To delete all logs on Weights & Biases that do not include '*' in the run name use:

uv run -- python -m seamstress.utils.wandb_cleanup