Horizons AI

Advanced Reinforcement Learning Environments

A comprehensive framework for building and managing synthetic environments designed specifically for training and evaluating long-horizon language agents.

🎯 Key Features

• 🔄 Snapshotting & Reproducibility - Full state capture and replay
• 🏗️ Statefulness First - Built-in state management across environments
• 🔌 Consistent APIs - Unified interface for all environment types
• 📊 Observability - Built-in tracing and monitoring
• 🌐 HTTP Access - RESTful API for remote training and evaluation
• 📚 Curriculum Learning - Configurable filtering and progression
• 🛠️ Agent Tools - Simple abstractions for agent-environment interaction

🚀 Quick Start

Installation

pip install horizons-ai

Basic Usage

import horizons

# Create environment
env = horizons.Environment("sokoban")

# Run agent
state = env.reset()
while not env.done:
    action = agent.act(state)
    state = env.step(action)

Running Evaluation Scripts

The framework includes ReAct agent evaluation scripts for testing language models on various environments. These scripts provide comprehensive metrics and shaped rewards for training.

Prerequisites

1. Start the synth service on port 8901:

   # In your service directory
   python -m uvicorn main:app --host 0.0.0.0 --port 8901
   

2. Ensure your model is available (OpenAI, Anthropic, etc.)

TicTacToe Evaluation

cd Environments
uvpm synth_env.examples.tictactoe.agent_demos.test_tictactoe_react_agent

Features:

• Tests strategic gameplay against random opponent
• Provides win/loss/draw statistics
• Validates coordinate parsing and legal moves
• Supports multiple models (gpt-4.1-mini, o3, etc.)

NetHack Evaluation

cd Environments
uvpm synth_env.examples.nethack.agent_demos.test_nethack_react_agent

Features:

• Comprehensive dungeon exploration evaluation
• 26+ shaped reward signals for training
• Balrog scoring system integration
• Progress bars for multi-trajectory runs
• Separates relevant vs. irrelevant metrics

Sokoban Evaluation

cd Environments  
uvpm synth_env.examples.sokoban.agent_demos.test_sokoban_react_agent

Features:

• Classic puzzle-solving evaluation
• Box-pushing logic validation
• Step efficiency analysis
• Multiple difficulty levels

Configuration

Edit the script configuration at the top of each file:

MODEL_NAME = "gpt-4.1-mini"  # or "o3", "claude-sonnet-4", etc.
NUM_INSTANCES = 5            # Number of test episodes
MAX_TURNS = 100             # Maximum steps per episode  
DIFFICULTY = "beginner"     # Environment-specific difficulty

All scripts provide detailed rubric results, progress metrics, and shaped rewards suitable for reinforcement learning applications.

Development Setup

# Clone repository
git clone https://github.com/your-org/synth-env.git
cd synth-env

# Install dependencies
uv sync

# Run tests
python dev/update_readme_metrics.py --fast

🎮 Supported Environments

Environment	Status	Description
Sokoban	✅ Stable	Classic puzzle game for planning
Hendryks Math	✅ Stable	Mathematical reasoning tasks
Crafter	✅ Stable	Minecraft-like survival environment
Verilog	🔄 Beta	Hardware description language tasks
Red Team	🚧 Development	Security testing scenarios
SWE-Bench	🚧 Development	Software engineering tasks

📖 Documentation

• API Reference - Complete API documentation
• Environment Guide - Detailed environment descriptions
• Contributing - Development setup and guidelines

🔧 Development

Health Check

# Check codebase health
python scripts/check_health.py

Testing

# Fast tests (~3 seconds)
python dev/update_readme_metrics.py --fast

# Full test suite
python dev/update_readme_metrics.py

Code Quality

# Format code
ruff format .

# Check linting
ruff check .

# Type checking
uvx ty check

Release

# Increment version and publish
python scripts/release.py

# Dry run
python scripts/release.py --dry-run

Pre-Merge Checklist

Before creating a PR, see dev/pr_checklist.md for the complete checklist.

🤝 Contributing

We welcome contributions! Please see our Contributing Guide for:

• Development setup
• Code style guidelines
• Testing requirements
• Pull request process

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

Special thanks to the research teams at DeepMind, Ragen AI, and other contributors to the environments included in this framework.

---

⚠️ Development Status: This project is under active development. While stable environments are production-ready, newer environments may have breaking changes.