Perceiver IO
Project description
Perceiver, Perceiver IO and Perceiver AR
This repository is a PyTorch and PyTorch Lightning implementation of
Perceiver: General Perception with Iterative Attention (paper, video) | |
Perceiver IO: A General Architecture for Structured Inputs & Outputs (paper, blog post) | |
General-purpose, long-context autoregressive modeling with Perceiver AR (paper, blog post) |
All model classes are written in plain PyTorch and can be wrapped into PyTorch Lightning modules for training at scale. The command line interface is implemented with the Lightning CLI. Pretrained weights can be imported for official models from the 🤗 Hub, training checkpoints from training examples are available for download too. Datasets used in the training examples are 🤗 datasets wrapped into PyTorch Lightning data modules. For NLP tasks, this library supports all 🤗 fast tokenizers and the 🤗 Perceiver UTF-8 bytes tokenizer.
Installation
Via pip
pip install perceiver-io[text,vision]
From sources
Installation from sources requires a Miniconda and a Poetry (1.2.0 or higher) installation.
Create and activate the perceiver-io
conda environment:
conda env create -f environment.yml
conda activate perceiver-io
Install main and test dependencies, including all extras:
# Without dependencies required for examples
poetry install --all-extras
If you want to run the examples locally, additionally use --with examples
:
poetry install --all-extras --with examples
Docker image
docker pull ghcr.io/krasserm/perceiver-io:latest
See Docker image for details.
Documentation
- Getting started
- Model construction
- Pretrained models
- Training examples
- Inference examples
- Building blocks
Articles
Articles referencing this repository:
- Training compute-optimal Perceiver AR language models
- A gentle introduction to Rotary Position Embedding
Other implementations
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for perceiver_io-0.8.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 31cb2c5b0f75d5b55faa266126f5040e9b591cf07a2406e03c7ca944fe510d7c |
|
MD5 | a69f920555ab77b92b1bdb10c1cddd9b |
|
BLAKE2b-256 | a7972e9ec83b70dd8dc37149b11f5b78a5ed61d180a2c07342653e6b9d30a561 |