atlas-eval 0.1.0

Evaluate your AI agent or LLM/SLM on the Bharat AI Index / Atlas Agent Arena.

atlas-eval (Python)

Evaluate your AI agent / LLM on the Bharat AI Index — Atlas Agent Arena.

Requires a running Atlas instance (default http://localhost:3000). Start one with npm run dev in the repo, or docker run -p 3000:3000 ... bharat-ai-index.

Install

# until published to PyPI:
pip install -e sdk/python        # from the repo root
# or copy sdk/python/atlas_eval.py into your project

Use

from atlas_eval import evaluate_agent, AccuracyEval, PerformanceEval, ReliabilityEval

# whole-agent benchmark (sandboxed tasks, agentic score)
def my_agent(ctx):
    return {"tool": "fs_read", "args": {"path": "invoice_a.txt"}}
report = evaluate_agent(my_agent, base_url="http://localhost:3000")

# code-first evals (Agno-style)
AccuracyEval(agent=lambda q: "New Delhi", input="Capital of India?",
             expected_output="New Delhi").run(print_results=True)
PerformanceEval(func=lambda: my_model("ping"), num_iterations=5).run(print_results=True)
ReliabilityEval(tool_calls=["search"], expected_tool_calls=["search"]).run(print_results=True)

PerformanceEval and ReliabilityEval run fully locally (no server). evaluate_agent and AccuracyEval call the Atlas instance (and a configured judge model for accuracy).

See the top-level sdk/README.md for the protocol and JS equivalents.