velora 0.1.1

A Liquid Reinforcement Learning (RL) framework.

Found on:

• PyPi
• GitHub

Velora

Velora is a lightweight and extensible framework built on top of powerful libraries like Gymnasium and PyTorch, specializing in a unique approach to Deep Reinforcement Learning (RL) algorithms, a paradigm we call Liquid RL.

Instead of Fully-connected Networks, Velora combines Liquid Neural Networks (LNNs) with Neural Circuit Policies (NCPs), specifically Ordinary Neural Circuits (ONCs).

These two components have interesting benefits:

• LNNs are a powerful RNN architecture that learns system dynamics, not just data patterns.
• NCPs focus on sparsely connected neurons with distinct functions, mimicking biological behaviour.

From what we've seen, these networks are powerful, small-scale architectures that excel in model explainability, making them perfect for control tasks.

Velora offers Liquidfied PyTorch-based implementations of RL algorithms, designed to be intuitive, easy to use, and customizable.

In other frameworks, we've seen a trend of heavy abstraction in favour of minimal lines of code. Our approach aims to offer a best of both worlds, abstracting code away but making the details explainable on the backend, while giving you the freedom to customize as needed.

Installation

To get started, simply install it through pip using one of the options below.

GPU Enabled

For PyTorch with CUDA (recommended):

pip install torch torchvision velora --extra-index-url https://download.pytorch.org/whl/cu124

CPU Only

Or, for PyTorch with CPU only:

pip install torch torchvision velora

Example Usage

Here's a simple example that should work 'as is':

from functools import partial

from velora.models import LiquidDDPG
from velora.gym import wrap_gym_env
from velora.utils import set_device, set_seed

import gymnasium as gym
from gymnasium.wrappers import NormalizeObservation, NormalizeReward, ClipReward

# Setup reproducibility and PyTorch device
seed = 64
set_seed(seed)

device = set_device()

# Add extra wrappers to our environment
env = wrap_gym_env("InvertedPendulum-v5", [
    partial(NormalizeObservation, epsilon=1e-8),
    partial(NormalizeReward, gamma=0.99, epsilon=1e-8),
    partial(ClipReward, max_reward=10.0),
    # RecordEpisodeStatistics,  # Applied automatically!
    # partial(NumpyToTorch, device=device),  # Applied automatically!
])

# Or, use the standard gym API (recommended for this env)
env = gym.make("InvertedPendulum-v5")

# Set core variables
state_dim = env.observation_space.shape[0]  # in features
n_neurons = 20  # decision/hidden nodes
action_dim = env.action_space.shape[0]  # out features

buffer_size = 100_000
batch_size = 128

# Train a model
model = LiquidDDPG(
    state_dim, 
    n_neurons, 
    action_dim, 
    buffer_size=buffer_size,
    device=device,
)
model.train(env, batch_size, n_episodes=300)

Currently, the framework only supports Gymnasium environments and is planned to expand to PettingZoo for Multi-agent (MARL) tasks.

API Structure

The frameworks API is designed to be simple and intuitive. We've broken into two main categories: core and extras.

Core

The primary building blocks you'll use regularly.

from velora.models import [algorithm]
from velora.callbacks import [callback]

Extras

Utility methods that you may use occasionally.

from velora.gym import [method]
from velora.utils import [method]

Customization

Customization is at the heart of Velora but requires a deeper understanding of the API.

You can read more about it in the documentation tutorials.

Active Development

🚧 View the Roadmap 🚧

Velora is a tool that is continuously being developed. There's still a lot to do to make it a fully functioning framework, such as detailed API documentation, and more RL algorithms.

Our goal is to provide a quality open-source product that works 'out-of-the-box' that everyone can experiment with, and then gradually fix unexpected bugs and introduce more features on the road to a v1 release.