perceiver-io 0.8.0

Perceiver IO

Perceiver, Perceiver IO and Perceiver AR

This repository is a PyTorch and PyTorch Lightning implementation of

Perceiver: General Perception with Iterative Attention (paper, video)
Perceiver IO: A General Architecture for Structured Inputs & Outputs (paper, blog post)
General-purpose, long-context autoregressive modeling with Perceiver AR (paper, blog post)

All model classes are written in plain PyTorch and can be wrapped into PyTorch Lightning modules for training at scale. The command line interface is implemented with the Lightning CLI. Pretrained weights can be imported for official models from the 🤗 Hub, training checkpoints from training examples are available for download too. Datasets used in the training examples are 🤗 datasets wrapped into PyTorch Lightning data modules. For NLP tasks, this library supports all 🤗 fast tokenizers and the 🤗 Perceiver UTF-8 bytes tokenizer.

Installation

Via pip

pip install perceiver-io[text,vision]

From sources

Installation from sources requires a Miniconda and a Poetry (1.2.0 or higher) installation.

Create and activate the perceiver-io conda environment:

conda env create -f environment.yml
conda activate perceiver-io

Install main and test dependencies, including all extras:

# Without dependencies required for examples
poetry install --all-extras

If you want to run the examples locally, additionally use --with examples:

poetry install --all-extras --with examples

Docker image

docker pull ghcr.io/krasserm/perceiver-io:latest

See Docker image for details.

Documentation

• Getting started
• Model construction
• Pretrained models
• Training examples
• Inference examples
• Building blocks

Articles

Articles referencing this repository:

• Training compute-optimal Perceiver AR language models
• A gentle introduction to Rotary Position Embedding

Other implementations

• Perceiver
• Perceiver IO
• Perceiver AR