velora 0.2.0

A Liquid Reinforcement Learning (RL) framework.

Found on:

• PyPi
• GitHub

Velora

Velora is a lightweight and modular framework built on top of powerful libraries like Gymnasium and PyTorch. It is home to a new type of RL agent called NeuroFlow (NF) that specializes in Autonomous Cyber Defence through a novel Deep Reinforcement Learning (RL) approach we call Liquid RL.

Benefits

• Explainability: NF agents use Liquid Neural Networks (LNNs) and Neural Circuit Policies (NCPs) to model Cyber system dynamics, not just data patterns. Also, they use sparse NCP connections to mimic biological efficiency, enabling clear, interpretable strategies via a labeled Strategy Library.
• Adaptability: NF agents dynamically grow their networks using a fitness score, adding more neurons to a backbone only when new Cyber strategies emerge, keeping agents compact and robust.
• Planning: NF agents use a Strategy Library and learned environment model to plan strategic sequences for proactive Cyber defense.
• Always Learning: using EWC, NF agents refine existing strategies and learn new ones post-training, adapting to evolving Cyber threats like new attack patterns.
• Customizable: NF agents are PyTorch-based, designed to be intuitive, easy to use, and modular so you can easily build your own!

Installation

To get started, simply install it through pip using one of the options below.

GPU Enabled

For PyTorch with CUDA (recommended):

pip install torch torchvision velora --extra-index-url https://download.pytorch.org/whl/cu126

CPU Only

Or, for PyTorch with CPU only:

pip install torch torchvision velora

Example Usage

Here's a simple example that should work 'as is':

from velora.models import NeuroFlow, NeuroFlowCT
from velora.utils import set_device

# Setup PyTorch device
device = set_device()

# For continuous tasks
model = NeuroFlowCT(
    "InvertedPendulum-v5",
    20,  # actor neurons 
    128,  # critic neurons
    device=device,
    seed=64,  # remove for automatic generation
)

# For discrete tasks
model = NeuroFlow(
    "CartPole-v1",
    20,  # actor neurons 
    128,  # critic neurons
    device=device,
)

# Train the model using a batch size of 64
model.train(64, n_episodes=50, display_count=10)

Currently, the framework only supports Gymnasium environments and is planned to expand to PettingZoo for Multi-agent (MARL) tasks, with updated adaptations of CybORG environments.

API Structure

The frameworks API is designed to be simple and intuitive. We've broken into two main categories: core and extras.

Core

The primary building blocks you'll use regularly.

from velora.models import [algorithm]
from velora.callbacks import [callback]

Extras

Utility methods that you may use occasionally.

from velora.gym import [method]
from velora.utils import [method]

Active Development

🚧 View the Roadmap 🚧

Velora is a tool that is continuously being developed. There's still a lot to do to make it a great framework, such as detailed API documentation, and expanding our NeuroFlow agents.

Our goal is to provide a quality open-source product that works 'out-of-the-box' that everyone can experiment with, and then gradually fix unexpected bugs and introduce more features on the road to a v1 release.