sorrel 1.4.0

A flexible framework for developing reinforcement learning algorithms.

Sorrel

Sorrel is a general-purpose reinforcement learning engine that enables researchers, developers, and students to develop and deploy reinforcement learning algorithms on new and pre-existing environments.

Sorrel can be regarded as an “operating system” for the RL process, in the sense that it unifies and abstracts foundational components from environment simulation all the way to policy search, exploration, and function approximation.

Our hope is that Sorrel will foster new research ideas, applications, and tools for a unified RL approach. We believe Sorrel will accelerate the rate of progress of the RL research space, as well as allow us to experiment with our own solutions to both novel and long-standing RL problems.

To get and use Sorrel as a user, install it from pip:

pip install sorrel

You can also install additional optional dependencies. For example, to install sorrel with llm integration support, do:

pip install sorrel[llm]

[!NOTE] Sorrel is extremely experimental and subject to change!

Development

[!IMPORTANT] We recommend you follow these instructions in a fresh conda/virtual environment to keep packages isolated from other environments and/or Python versions. Python 3.12+ is required.

To create a virtual environment, navigate to your project directory in the terminal and run:

python -m venv ./venv

To activate the virtual environment, run

./venv/Scripts/activate  # windows

source ./venv/bin/activate  # mac

to deactivate the virtual environment, simply run deactivate in the terminal.

Getting started

Sorrel uses the poetry package manager to manage its dependencies. Start by running poetry --version in your terminal to make sure you have poetry installed.

[!IMPORTANT] If you do not have poetry, use pipx install poetry (not pip) to make sure you do not have poetry installed in the same environment that Sorrel is using. See the poetry documentation for more information and installation instructions.

With poetry available, to install Sorrel as a developer, run the following command:

poetry install --with dev,llm

in the folder containing the pyproject.toml file.

[!NOTE] If cannot run your script from terminal due to errors like module 'sorrel' not found, check if your virtual environment is activated. If it is, and you are still getting the same error, try running poetry run python your_script.py.

Workflow Tools

We use a number of tools to simplify the development workflow for contributors. Among these tools include code formatters (such as black) to report style errors and (try to) automatically format your code wherever possible, along with testing frameworks (such as pytest) to automatically test the code.

We have included a pre-commit configuration to automatically run all CI tasks whenever you attempt to commit to the Sorrel repository.

To set up pre-commit, start by confirming that it's installed by running:

pre-commit --version

If this fails, you'll need to install pre-commit by running pip install pre-commit. Then, run

pre-commit install

in the folder containing .pre-commit-config.yaml file.

Afterward, the Git hooks will be run automatically at every new commit.

You may also run these hooks manually with pre-commit run --all-files. If needed, you can skip the hooks (not recommended) with git commit --no-verify -m <commit message>.

[!NOTE] You may have to run pre-commit run --all-files manually a couple of times to make it pass when you commit, as each formatting tool will first format the code and fail the first time but should pass the second time.

Writing documentation

To contribute to the documentation, you may add, delete, or edit files in the .\docs\source folder. This project uses Sphinx to auto-build files in .rst, .md, or .ipynb format into html.

Note that since Sorrel is private (for now), we cannot host the documentation online yet. To view your documentation changes locally, navigate to the .\docs folder and run the following command:

Windows:

.\make html

Other systems:

make html

Then, view the updated documentations by opening .\docs\build\html\index.html with the browser of your choice.

[!NOTE] If the changes you made were not reflected across all the pages, you may have to run .\make clean (on Windows) or make clean (on other systems) before the above command to ensure the documentation is built from scratch. It is generally good practice to do so anyway but may take slightly longer.

Citing the project

Gelpí, R. A., Ju, Y., Jackson, E. C., Tang, Y., Verch, S., Voelcker, C., & Cunningham, W. A. (2025). Sorrel: A simple and flexible framework for multi-agent reinforcement learning. arXiv:2506.00228 [cs.MA]

Maintainers

Sorrel is currently maintained by Yibing Ju (aka @bing-j) and Rebekah Gelpi (aka @rgelpi).

[!NOTE] We do not do technical support, nor consulting and don't answer personal questions per email. If you have any questions, concerns, or suggestions, please post them on the GitHub issues page or the GitHub discussion page.

Acknowledgments

We would like to thank the other authors: Ethan Jackson, Yikai Tang, Shon Verch, Claas Voecker, and Wiliam Cunningham.