OpenGym

127 challenges to test if your AI agent actually works — not just the model, but the infrastructure.

OpenGym is an open-source benchmark that evaluates AI agents across 7 capability dimensions: coding, memory persistence, tool discovery, multi-step planning, self-correction, safety boundaries, and multi-agent coordination. Unlike benchmarks that only test "can the model solve this?", OpenGym tests "does the agent system work reliably?"

git clone https://github.com/widingmarcus-cyber/opengym && cd opengym
pip install -e .
opengym run 101 --agent "python my_agent.py --task '{task}' --dir {workspace}"
opengym score all --summary

How It Works

Each challenge is a self-contained folder. Your agent reads the task, does the work, and the CLI scores it.

101-learn-and-recall/
├── README.md        ← Agent reads this
├── setup/           ← Agent edits these files
├── steps/           ← Multi-session task steps (if applicable)
├── tools/           ← Executable tools (if applicable)
├── tests/           ← Hidden verification (agent doesn't touch)
└── metadata.yaml

Two workflows:

# Manual: fetch, let your agent work, score
opengym fetch 001
# ... your agent solves it ...
opengym score 001

# Automated: opengym orchestrates your agent
opengym run 101 --agent "python my_agent.py --task '{task}' --dir {workspace}"
opengym run all --agent "..." --summary    # run the full gauntlet

7 Dimensions, 127 Challenges

Most benchmarks only test coding. OpenGym tests the infrastructure that makes agents reliable in production.

Coding — 100 challenges

The baseline. Read a task, write/fix code, pass tests. This is what every benchmark measures — OpenGym includes it but goes further.

Memory Persistence — 5 challenges

The key differentiator. Tests whether your agent's memory actually persists across sessions. The CLI kills the agent process between steps and clears context — only files the agent explicitly wrote survive. Context window tricks fail here.

Tool Discovery & Use — 5 challenges

Tests whether your agent can discover unfamiliar tools, handle failures, and manage rate limits.

Self-Correction & Resilience — 5 challenges

Tests whether your agent can recover from errors, trace root causes, and not get distracted by noise.

Safety & Boundaries — 5 challenges

Tests whether your agent resists prompt injection in data, refuses dangerous commands, and stays within scope.

Multi-Agent Coordination — 3 challenges

Tests whether agents can coordinate via shared files without overwriting each other.

Multi-Step Planning — 4 challenges

Tests decomposition, adaptation, and resource management.

Scoring

Every challenge scores 0-100 based on tests passed. Results are grouped by dimension so you see where your agent's infrastructure breaks down.

============================================================
  OpenGym Score: 68/100
  Passed: 87/127
============================================================

By Dimension:
  coding         [################....] 82/100
  memory         [########............] 40/100
  tool-use       [############........] 60/100
  resilience     [##########..........] 55/100
  safety         [##################..] 90/100
  multi-agent    [######..............] 30/100
  planning       [##########..........] 50/100

Diagnostics:
  - memory (40/100): Your agent cannot persist information across sessions.
    It needs a real memory system — not just context window.
  - multi-agent (30/100): Your agent cannot coordinate with other agents
    via shared resources.

CLI Reference

# List and filter
opengym list                              # List all 127 challenges
opengym list --dimension memory           # Filter by dimension
opengym list --category algorithm         # Filter by category
opengym list --difficulty hard            # Filter by difficulty
opengym list --json-output                # Machine-readable

# Fetch challenges
opengym fetch 001                         # Fetch one challenge
opengym fetch all                         # Fetch everything

# Score manually
opengym score 001                         # Score one challenge
opengym score all --summary               # Score all + diagnostics
opengym score all --json-output           # JSON output

# Run agent automatically (including multi-session orchestration)
opengym run 101 --agent "python my_agent.py --task '{task}' --dir {workspace}"
opengym run all --agent "..." --summary   # Full gauntlet

Test Your Agent

See docs/AGENT_GUIDE.md for copy-paste examples with Claude Code, OpenAI, LangChain, CrewAI, and custom agents.

Create Challenges

See docs/CHALLENGE_SPEC.md for the challenge format.

Tech Stack

Python 3.10+ / click / pytest / YAML / JSON

License

MIT