Sorrel

Sorrel is a general-purpose reinforcement learning engine that enables researchers, developers, and students to develop and deploy reinforcement learning algorithms on new and pre-existing environments.

Sorrel can be regarded as an “operating system” for the RL process, in the sense that it unifies and abstracts foundational components from environment simulation all the way to policy search, exploration, and function approximation.

Our hope is that Sorrel will foster new research ideas, applications, and tools for a unified RL approach. We believe Sorrel will accelerate the rate of progress of the RL research space, as well as allow us to experiment with our own solutions to both novel and long-standing RL problems.

To get and use Sorrel as a user, install it from pip:

pip install sorrel

You can also install additional optional dependencies. For example, to install sorrel with llm integration support, do:

pip install sorrel[llm]

Development

To create a virtual environment, navigate to your project directory in the terminal and run:

python -m venv ./venv

To activate the virtual environment, run

./venv/Scripts/activate  # windows

source ./venv/bin/activate  # mac

to deactivate the virtual environment, simply run deactivate in the terminal.

Getting started

Sorrel uses the poetry package manager to manage its dependencies. Start by running poetry --version in your terminal to make sure you have poetry installed.

With poetry available, to install Sorrel as a developer, run the following command:

poetry install --with dev,llm

in the folder containing the pyproject.toml file.

Workflow Tools

We use a number of tools to simplify the development workflow for contributors. Among these tools include code formatters (such as black) to report style errors and (try to) automatically format your code wherever possible, along with testing frameworks (such as pytest) to automatically test the code.

We have included a pre-commit configuration to automatically run all CI tasks whenever you attempt to commit to the Sorrel repository.

To set up pre-commit, start by confirming that it's installed by running:

pre-commit --version

If this fails, you'll need to install pre-commit by running pip install pre-commit. Then, run

pre-commit install

in the folder containing .pre-commit-config.yaml file.

Afterward, the Git hooks will be run automatically at every new commit.

You may also run these hooks manually with pre-commit run --all-files. If needed, you can skip the hooks (not recommended) with git commit --no-verify -m <commit message>.

Writing documentation

To contribute to the documentation, you may add, delete, or edit files in the .\docs\source folder. This project uses Sphinx to auto-build files in .rst, .md, or .ipynb format into html.

Note that since Sorrel is private (for now), we cannot host the documentation online yet. To view your documentation changes locally, navigate to the .\docs folder and run the following command:

Windows:

.\make html

Other systems:

make html

Then, view the updated documentations by opening .\docs\build\html\index.html with the browser of your choice.

Citing the project

Gelpí, R. A., Ju, Y., Jackson, E. C., Tang, Y., Verch, S., Voelcker, C., & Cunningham, W. A. (2025). Sorrel: A simple and flexible framework for multi-agent reinforcement learning. arXiv:2506.00228 [cs.MA]

Maintainers

Sorrel is currently maintained by Yibing Ju (aka @bing-j) and Rebekah Gelpi (aka @rgelpi).

Acknowledgments

We would like to thank the other authors: Ethan Jackson, Yikai Tang, Shon Verch, Claas Voecker, and Wiliam Cunningham.