Navital Gym

Deterministic, auto-graded environments for AI agents doing tasks set in India: planning a trip around rail quotas and waitlists, GST invoicing, filling forms, answering customers in Hindi and other Indian languages. By Navital Labs. Navital Gym is public and in active development: results are added as runs finish (see docs/RESULTS.md).

• What: six sets of sandboxed tasks (trip planning, UPI, GST, forms, multi-document forms, customer support in Hindi and Hinglish), each graded by written rules, never by an AI judge.
• Why: to measure whether an agent gets Indian-reality tasks right, with results anyone can reproduce and verify.
• Try it: pip install navital-gym && navital-gym run --env trip --agent oracle (prints 1.00), then see the live leaderboard or docs/RESULTS.md.

Read this first. Every environment uses a simplified, documented version of Indian practice and synthetic data. These are tests for AI agents, not statements about current law, railway, bank or tax rules. Not affiliated with or endorsed by NPCI, RBI, GSTN / CBIC, IRCTC / Indian Railways, any bank or any government body; names are used only to describe the workflow. Not financial or tax advice. Each env README states its own rules and an "as of" date.

Why another gym

• Exact grading, no LLM judge. Each task is graded by deterministic rules that recompute everything from the sandbox world. Nothing the agent claims (cost, score, "solved") is trusted. Garbage input scores 0; the grader never crashes.
• Honest about impossible requests. Many tasks have no valid answer; the agent must say so and name why. Padding the claim with a wrong reason is marked wrong.
• Hidden fields stay hidden. The agent sees only the prompt, the tools and the policy text. Expected answers, feasibility and designer notes never reach it; a failing test enforces that for every task in every env.
• Standard library only. The core and every env run on plain Python 3.10+. Adapters for Inspect AI, verifiers and OpenEnv are optional extras.

The suite

Every env README carries its simplified rules and the sources of any real-world figure.

Quickstart

Install from PyPI, or straight from GitHub, or from a clone (for development):

pip install navital-gym
pip install git+https://github.com/Navitallabs/navital-gym.git
pip install -e .                                        # inside a clone

Then:

navital-gym list                                        # envs, task counts, baselines
navital-gym run --env trip --agent oracle               # a scripted reference agent: should print 1.00
navital-gym run --env trip --agent heuristic --k 3 --out runs/trip.json
python -m unittest discover -s tests                    # no dependencies needed

Run any model that speaks the OpenAI chat API with tool calling (Ollama, vLLM, LM Studio, hosted): --agent "openai:qwen2.5:7b@http://localhost:11434/v1" (the key is read from the provider's environment variable, OPENAI_API_KEY for openai: specs including a custom @base_url, GROQ_API_KEY, GEMINI_API_KEY, ANTHROPIC_API_KEY; never from the command line). Reports show pass@1, pass^k, per-rule failures, and separate tables per split (core / experimental) and kind (feasible / infeasible), so one mixed number is never the headline.

Python:

from navital_gym.envs.trip import load_env
env = load_env()
obs = env.reset("IN-TRIP-002")            # prompt + tool specs; nothing hidden
out = env.world.call("search_trains", {"from_city": "Delhi", "to_city": "Agra", "date": "2026-12-05"})

Adapters (optional extras)

Results

Pass@1 on deterministic stratified samples or whole splits (k=1, one run per cell; n = tasks, so one task is 1/n of the cell: read small-n cells as indicative). 'easy' columns are the easy tier (designed so a 7B-30B model can sometimes solve them); the others are the standard tasks. Full tables with format failures, strict (no-repair) pass@1, partial credit, spend, dates and commits: docs/RESULTS.md. Every row was re-graded by the same grading code (scripts/regrade.py); check it with git rev-parse v1.0.1:navital_gym (see docs/RESULTS.md).

Small and mid-size models: pass@1 (the typical user's models)

Small and mid-size models: partial credit (mean fraction of scored rules passed; shows progress where pass@1 is near zero)

Larger models, for reference (what strong models reach)

Most of the models people will test are small or mid-size, so they come first. Many score low: that is the point of a hard benchmark, but a cell near zero says little, so read partial credit next to it. Samples are small and the small and mid-size models above are not the strongest agents available; these results show that the environments separate models, not a leaderboard. Task difficulty labels are designer intent, not calibrated.

Build your own env

docs/ENV_AUTHORING.md (a new env is mostly data plus rules on the shared core) and docs/CORE-API.md (the frozen core interfaces).

Licences

Code: Apache-2.0 (LICENSE, copyright Navital Labs, see NOTICE). Task data: CC BY 4.0 (LICENSE-DATA), covering navital_gym/envs/*/tasks*.jsonl and navital_gym/envs/*/FLAG_CHANGES.md. Cite with CITATION.cff.