sme-rl 0.1.0

Synthetic monitoring environments for reinforcement learning

SME-RL: Synthetic Monitoring Environments for Reinforcement Learning

Figure 1: Evaluation performance during training (column 1) and final within-distribution and out-of-distribution performance for PPO, SAC and TD3 across different complexities of the optimal policy the agent seeks to mirror (C_pi*, columns 2-4). Performance is defined as the complement of the mean average error between action and optimal action. States within $\mathbb{R}^2(0,1)$ are within-distribution. States beyond the unit square are out-of-distribution. SD = State Dimension.

SME-RL provides highly configurable Synthetic Monitoring Environments for evaluating Deep Reinforcement Learning algorithms. Designed for rapid prototyping, sanity checking, and fundamental RL research, SME integrates natively with the Gymnasium API.

Installation

You can install the package directly from PyPI:

pip install sme-rl

Note: SME-RL supports Python 3.8+ and relies on PyTorch and Gymnasium.

Quick Start

SME-RL is registered as a standard Gymnasium environment. You can initialize it using gymnasium.make() and immediately use it with standard RL libraries like Stable-Baselines3.

import gymnasium
import sme_rl # Registers the environment
from stable_baselines3 import PPO

# 1. Initialize the environment
env = gymnasium.make(
    "SME-v0",
    n_state_channels=8,
    n_action_channels=4,
    episode_length=100
)

# 2. Train an agent
model = PPO("MlpPolicy", env, verbose=1)
model.learn(total_timesteps=100_000)

# 3. Evaluate your agent
env.eval(model)

Configuration Parameters

You can heavily customize the complexity and rules of the environment during instantiation:

Parameter	Symbol	Type	Default	Description
n_state_channels	N_s	int	Required	Dimensionality of the state space.
n_action_channels	N_a	int	Required	Dimensionality of the action space.
episode_length	—	int	100	Maximum steps per episode before truncation.
reward_every	k	int	1	Step interval for reward distribution.
min_reward	r_min	float	0.0	Minimum threshold for a step reward to be registered.
kill_threshold	D	float	0.0	Reward threshold below which the episode terminates early (survival difficulty).
policy_complexity	C_pi*	int	1	Depth of the optimal policy network.
seed	—	int	1	Global seed for procedural generation and reproducibility.

Within- and Out-of-Distribution Evaluation

SME-RL features a built-in evaluation framework to benchmark your trained agent's generalization capabilities. By using the .eval() method, you can probe the policy across both within-distribution (WD) states (typically bounded between 0 and 1) and wide out-of-distribution (OOD) states.

The evaluation computes the mean absolute error between your agent's choices and the ground-truth optimal policy network, mapping performance to a normalized reward score.

Usage Example

# After training your model (e.g., using Stable-Baselines3)
states, opt_actions, model_actions, rewards = env.eval(
    model=model,
    storepath="results/eval_run",   # Saves data as an .npz archive
    channel_min=-1.0,               # Lower bound for OOD exploration
    channel_max=2.0,                # Upper bound for OOD exploration
    probes=10000,                   # Number of evaluation sample points
    wd_weight=0.5                   # 50% within-distribution, 50% out-of-distribution
)

print(f"Evaluated {len(states)} states.")
print(f"Mean evaluation score: {rewards.mean():.4f}")

Evaluation Parameters

Parameter	Type	Default	Description
model	object	Required	The trained RL agent (expects a .predict() interface compatible with SB3).
storepath	str / Path	None	Filepath directory where evaluation metrics are written out as a compressed .npz file.
channel_min	float	0.0	Minimum bounds for sampling evaluation state channels. Limits below 0 test OOD behavior.
channel_max	float	1.0	Maximum bounds for sampling evaluation state channels. Limits above 1 test OOD behavior.
probes	int	10000	The total number of evaluation samples generated.
wd_weight	float	0.5	Ratio of within-distribution samples ([0, 1]) relative to wide distribution samples.

Returned Outputs

The evaluation function returns a tuple of four arrays:

Output	Description
states	The sampled synthetic verification states.
opt_a	Target action tensors generated by the ground-truth optimal policy structure.
a	Action predictions selected by your trained model.
raw_reward	Normalized scores scaled between 0.0 and 1.0 indicating how tightly your model mirrors optimal actions.

evaluation metrics are written out as a compressed .npz file if the storepath parameter in the eval() method is specified.

📖 Citation

If you use this environment in your research, please cite our paper:

@misc{pleiss2026smerl,
      title={Synthetic Monitoring Environments for Reinforcement Learning}, 
      author={Leonard Pleiss},
      year={2026},
      eprint={2603.06252},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}

License

This project is licensed under the MIT License.