NeatRL provides readable implementations of popular RL algorithms with a focus on simplicity and ease of use. Built with PyTorch and designed for research and experimentation.

Features

• Single-file implementations: Each algorithm lives entirely in one self-contained file -- no hunting across modules
• Experiment Tracking: Built-in support for Weights & Biases logging
• Gymnasium Compatible: Works with Gymnasium environments and adding many more!
• Atari Support: Full support for Atari games with automatic CNN architectures
• Parallel Training: Vectorized environments for faster data collection
• Easy to Extend: Modular design for adding new algorithms
• Video Recording: Automatic video capture and WandB integration

Installation

pip install neatrl

That is the whole default setup. It brings the terminal dashboard (rich, psutil), classic-control environments (CartPole, Pendulum, Acrobot, MountainCar) and everything the MLP aliases need — no extras to remember.

With uv, install the neatrl command as an isolated tool — no virtualenv to manage:

uv tool install neatrl

Image-based (-cnn) aliases

The -cnn aliases train on image observations, which only Atari and Box2D provide — gymnasium's classic control and MuJoCo envs are all vector observations, so there is no CNN equivalent for them. Add the family you want:

pip install "neatrl[atari]"    # ALE/Pong-v5, ALE/Breakout-v5 (default for -cnn)
pip install "neatrl[box2d]"    # CarRacing-v3, LunarLander-v3, BipedalWalker-v3
pip install "neatrl[mujoco]"   # HalfCheetah-v4, Ant-v5, …
pip install "neatrl[all]"      # all of the above in one go

If you run a -cnn alias without the matching extra, the CLI prints gymnasium's error unchanged and then tells you the exact command to run.

neatrl[cli] and neatrl[classic] still resolve, so older instructions keep working, but they are no longer needed.

Browse supported environments: MuJoCo · Box2D · Classic Control · Atari

CLI

Train any algorithm from the terminal:

neatrl train dqn CartPole-v1
neatrl train ppo LunarLander-v3
neatrl train sac Pendulum-v1
neatrl train td3 HalfCheetah-v4
neatrl train rnd CliffWalking-v1
neatrl train ddpg-cnn CarRacing-v3

The environment is optional — omit it to use the algorithm's default (neatrl train dqn trains on CartPole-v1, neatrl train sac on Pendulum-v1). --steps overrides the training length:

neatrl train dqn CartPole-v1 --steps 50000

Runs are local: nothing is uploaded to Weights & Biases unless you pass --wandb.

Pass --help to any command for the full list of the 21 algorithm aliases.

Quick Start

Train DQN on CartPole

from neatrl.dqn_mlp import train_dqn

model = train_dqn(env_id="CartPole-v1", total_timesteps=10000, seed=42)

Train tabular Q-learning on FrozenLake

from neatrl.q_learning import train_q_learning

q_table = train_q_learning(
    env_id="FrozenLake-v1",
    total_timesteps=50000,
    seed=42,
    use_wandb=False,
)

Works on simple discrete Gym envs only (FrozenLake, CliffWalking, Taxi, …). For continuous observations use DQN.

Train PPO on Classic Control

from neatrl.ppo_mlp import train_ppo

model = train_ppo(
    env_id="CartPole-v1",
    total_timesteps=50000,
    n_envs=4,  # Parallel environments
    GAE=0.95,  # Generalized Advantage Estimation lambda
    clip_value=0.2,  # PPO clipping parameter
    use_wandb=True,  # Track with WandB
    seed=42,
)

Train SAC on Continuous Control

from neatrl.sac_mlp import train_sac

model = train_sac(
    env_id="Pendulum-v1",
    total_timesteps=50000,
    alpha=0.2,  # Entropy regularization coefficient
    autotune_alpha=True,  # Automatically tune alpha
    use_wandb=True,  # Track with WandB
    seed=42,
)

Train SAC on Atari

from neatrl.sac_cnn import train_sac_cnn

model = train_sac_cnn(
    env_id="ALE/Breakout-v5",
    total_timesteps=100000,
    alpha=0.2,
    autotune_alpha=True,
    atari_wrapper=True,  # Automatic Atari preprocessing
    use_wandb=True,
    seed=42,
)

Documentation

Complete Documentation

The docs include:

• Detailed usage examples for every algorithm
• Hyperparameter tuning guides
• Environment compatibility notes
• Experiment tracking setup (WandB)

Supported Algorithms

Current Implementations

• DQN (Deep Q-Network) - Classic value-based RL algorithm

  • Support for discrete action spaces
  • Experience replay and target networks
  • Atari preprocessing and frame stacking

• Dueling DQN - Enhanced DQN with separate value and advantage streams

  • Improved learning stability
  • Better performance on complex environments

• Double DQN - Q-learning target with decoupled action selection/evaluation

  • Removes the upward bias of the max in the DQN target
  • Same API and hyperparameters as DQN (double_dqn_mlp, double_dqn_cnn)

• REINFORCE - Policy gradient method for discrete and continuous action spaces

  • Atari game support with automatic CNN architecture
  • Parallel environment training (n_envs support)
  • Continuous action space support
  • Episode-based Monte Carlo returns
  • Variance reduction through baseline subtraction

• DDPG (Deep Deterministic Policy Gradient) - Actor-critic method for continuous action spaces

  • Deterministic policy gradient for continuous control
  • Experience replay and target networks
  • Ornstein-Uhlenbeck noise for exploration
  • Support for exact continuous action spaces

• A2C (Advantage Actor-Critic) - Synchronous actor-critic algorithm

  • Advantage function for reduced variance
  • Support for both discrete and continuous action spaces
  • Parallel environment training with vectorized environments
  • Monte Carlo returns for value estimation

• TD3 (Twin Delayed DDPG) - Actor-critic method for continuous control

  • Twin Q-networks to reduce overestimation bias
  • Delayed policy updates for improved stability
  • Target policy smoothing with noise
  • Experience replay and target networks
  • CNN support for image-based environments

• SAC (Soft Actor-Critic) - Maximum entropy reinforcement learning

  • Stochastic Gaussian policies with entropy regularization
  • Twin Q-networks for stable learning
  • Automatic entropy tuning (alpha parameter)
  • Balances exploration and exploitation
  • CNN support for complex environments

• PPO (Proximal Policy Optimization) - Policy gradient method with GAE

  • Full PPO implementation with Generalized Advantage Estimation (GAE)
  • Support for both discrete and continuous action spaces
  • Atari game support with automatic CNN architecture
  • Clipped surrogate objective for stable policy updates
  • Value function clipping and entropy regularization
  • Vectorized environments for parallel training

• PPO-RND (Proximal Policy Optimization with Random Network Distillation) - Curiosity-driven exploration

  • Intrinsic motivation through novelty detection
  • Combined extrinsic and intrinsic rewards for better exploration
  • Support for both discrete and continuous action spaces
  • PPO with clipped surrogate objective
  • Vectorized environments for parallel training
  • Intrinsic reward normalization and advantage calculation

• More algorithms coming soon...

Contributing

Contributions are welcome. Please feel free to submit a Pull Request.

Development Setup

git clone https://github.com/YuvrajSingh-mist/NeatRL.git
cd NeatRL/NeatRL

# Use a non-hidden venv/ (avoids a macOS/uv flag issue with .venv/)
uv venv venv
uv pip install -e ".[cli,dev]" --python venv/bin/python
source venv/bin/activate

pytest tests/           # run the test suite
ruff check src/         # lint
neatrl train ppo lunar   # smoke test CLI

When to re-run uv pip install -e .: Only when you add/remove/rename a module file or change pyproject.toml dependencies. Editing existing .py files never requires a reinstall — the editable install picks up changes automatically.

For the complete changelog, see CHANGELOG.md.

Citation

If you use NeatRL in academic work, please cite it as below. The concept DOI always resolves to the newest archived version (see CITATION.cff for the machine-readable form).

@software{singh2026neatrl,
  title     = {NeatRL: Readable Single-File Reinforcement Learning Implementations in PyTorch},
  author    = {Singh, Yuvraj},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.23198154},
  url       = {https://github.com/YuvrajSingh-mist/NeatRL}
}

License

This project is licensed under the Apache License 2.0 for code, and CC BY 4.0 for documentation, diagrams, benchmark results and published artifacts. See NOTICE and CITATION.cff.

Made with ❤️ for the RL community