PyCRM

A Python framework for formal task specification and efficient reinforcement learning with Reward Machines (RMs) and Counting Reward Machines (CRMs).

Documentation | Paper | Quick Start

Overview

PyCRM provides a unified framework for Reward Machines (RMs) and Counting Reward Machines (CRMs), offering a formal approach to reward specification in reinforcement learning. RMs handle regular tasks with finite-state automata, while CRMs extend this with counters for Turing-complete expressiveness, enabling efficient learning through structured reward functions and counterfactual experiences.

Features

• Unified RM/CRM Support: First-class support for both Reward Machines and Counting Reward Machines
• Reinforcement Learning Integration: Ready-to-use agents that leverage counterfactual experiences
• Cross-Product Environments: Framework for combining ground environments with RMs or CRMs
• Modular Design: Composable automata for complex task specifications
• Expressive Power: From regular languages (RMs) to Turing-complete specifications (CRMs)
• Example Environments: Complete worked examples for both RMs and CRMs

Quick Start

Installation

pip install pyrewardmachines

For detailed installation instructions and troubleshooting, see the Installation Guide.

Basic Usage

See the Quick Start Guide for complete examples of creating and using both Reward Machines and Counting Reward Machines, including:

• Setting up ground environments, labelling functions, and automata (RMs or CRMs)
• Creating cross-product environments
• Training agents with counterfactual experiences

For a comprehensive introduction to the framework, see the Introduction.

Key Components

The PyCRM framework consists of several key components:

• Ground Environment: The base environment (typically a Gymnasium environment)
• Labelling Function: Maps environment observations to symbolic events
• Automaton: Formal specification of the task (either a Reward Machine or Counting Reward Machine)
• Cross-Product Environment: Combines all components into a learning environment
• RL Agents: Algorithms that leverage counterfactual experiences for improved sample efficiency

For detailed explanations of these components, see the Core Concepts section in the documentation.

Applications

• Task-Oriented RL: Specify complex objectives with structured reward functions
• Robotics: Define temporally extended tasks with symbolic events
• Formal Verification: Guarantee task completion through CRM properties
• Curriculum Learning: Progressively build task complexity

For complete worked examples demonstrating these applications, see the Worked Examples section in the documentation.

Citation

If you use Counting Reward Machines in your research, please cite:

@article{bester2023counting,
  title={Counting Reward Automata: Sample Efficient Reinforcement Learning Through the Exploitation of Reward Function Structure},
  author={Bester, Tristan and Rosman, Benjamin and James, Steven and Tasse, Geraud Nangue},
  journal={arXiv preprint arXiv:2312.11364},
  year={2023}
}

Contributing

Contributions are welcome. To get started:

# Clone repository
git clone https://github.com/TristanBester/pycrm.git
cd pycrm

# Set up virtual environment
uv venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install development dependencies
uv pip install -e ".[dev]"

# Run tests
uv run pytest

# Run comprehensive testing across environments
uv pip install tox
uv run tox

License

This project is licensed under the MIT License - see the LICENSE file for details.