zhanla 0.1.2.20

CLI from Zhanla for running and uploading AI component evaluations

zhanla

Command-line interface for validating, uploading, running, and evaluating Zhanla SDK components.

The CLI can be installed on its own for auth, listing web resources, dataset/component upload, and fully web-backed runs. Local component execution requires the matching language SDK.

Installation

pip install zhanla

Requires Python >=3.10.

Optional local execution dependencies:

pip install zhanla-sdk-py
npm install @zhanla/sdk-ts

Check the install:

zhanla --version
zhanla status

Authentication

Authenticate with an SDK API key:

zhanla login

The key format is:

bm_kid_XXXX.bm_sec_XXXX

Credentials are stored at .zhanla/credentials.json in the current repo root. The saved login is repo-local, not global machine state.

By default the CLI targets the production control plane. For local development, set ZHANLA_API_URL to a loopback host to repoint the CLI without re-logging-in, e.g. ZHANLA_API_URL=http://localhost:3001 zhanla serve --repo acme/app. If ZHANLA_API_URL is unset, a loopback NEXT_PUBLIC_APP_URL from .env.local is also honored. Non-loopback hosts are ignored because the CLI sends a bearer token. Only localhost/127.0.0.1/::1 are accepted.

Remove saved credentials:

zhanla logout

Authenticated commands exchange the saved API key for a fresh short-lived token. If you are not logged in, they exit with:

Not logged in. Run `zhanla login` first.

Environment

The CLI loads env files from the current directory or nearest ancestor containing one of:

1. .env.local
2. .env
3. .env.example

Existing shell variables are not overwritten. Provider API keys such as ANTHROPIC_API_KEY, OPENAI_API_KEY, GOOGLE_API_KEY, and OPENROUTER_API_KEY should live there for local component execution.

The CLI always connects to the hosted Zhanla app at https://benchmark-black.vercel.app. Base URL env overrides are ignored for CLI backend calls.

Command overview

zhanla validate COMPONENT.py:component-key
zhanla upload COMPONENT.py:component-key
zhanla upload --dataset DATASET.json --name "Dataset name"
zhanla run COMPONENT.py:component-key --dataset DATASET.json --eval EVAL.py:eval-key
zhanla list datasets
zhanla list autoraters
zhanla list components

Advanced local-runner/refinement commands:

zhanla serve --repo ORG/REPO
zhanla apply-patch --run RUN_ID

Terminal output

The CLI prints progress and a compact completion summary for run, upload, daemon, patch, and output-file flows. Run summaries show how many rows completed, how many were scored, failure counts, upload status, result links, and output files when applicable.

Errors redact common token/key shapes and secret-like environment values before printing. Command failures include next steps such as the validation, login, list, or retry command to run next. Empty list results also include next steps to upload resources or relax filters.

zhanla validate

Validate component definitions offline, without auth or network:

zhanla validate components.py
zhanla validate components.ts:priority-tool

Targets resolve by SDK component key. If a file contains multiple runnable components or evals, include the :key suffix.

zhanla upload

Upload a component or eval definition:

zhanla upload components.py:priority-tool
zhanla upload evals.ts:priority-eval --yes

Upload a standalone dataset:

zhanla upload --dataset tickets.json
zhanla upload --dataset tickets.csv --name "Support Tickets" --yes

Component/eval upload syncs the full discoverable closure. Dataset upload creates or updates rows idempotently.

zhanla run

zhanla run supports three common execution modes:

1. Local component + local eval.
2. Local component + web autorater.
3. Fully web-backed component + dataset + autorater.

Local component + local eval

zhanla run components.py:priority-tool \
  --dataset tickets.json \
  --eval evals.py:priority-eval \
  --dry-run \
  --yes

For TypeScript:

zhanla run components.ts:priority-tool \
  --dataset tickets.json \
  --eval evals.ts:priority-eval \
  --dry-run \
  --yes

--dry-run keeps both component and eval local and does not require login.

Without --dry-run, the component still runs locally, definitions and rows/results are synced, then the eval runs as a managed evaluate-only run. That path requires zhanla login.

Local component + web autorater

zhanla run components.py:priority-tool \
  --dataset tickets.json \
  --web-eval support-quality \
  --yes

You can also use a web dataset:

zhanla run components.py:priority-tool \
  --web-dataset support-tickets \
  --web-eval support-quality \
  --yes

This flow runs the local component, uploads or reuses rows and outputs, starts the managed autorater, and polls until completion.

Fully web-backed run

zhanla run \
  --web-config support-agent \
  --web-dataset support-tickets \
  --web-eval support-quality \
  --yes

The server generates component responses using the web component definition and org provider keys, then runs the managed autorater.

Run flags

Flag	Short	Description
--eval <spec>	-e	Local eval target in file.py:key or file.ts:key form
--dataset <path>	-d	Local .json or .csv dataset
--web-eval <key-or-id>		Managed autorater key or ID
--web-dataset <key-or-id>		Web dataset key or ID
--web-config <key-or-id>		Web component key or ID for fully web-backed runs
--dry-run		Execute locally and skip sync/upload
--yes	-y	Skip confirmation prompts
--max-rows <N>		Run only the first N dataset rows
--fail-on-row-error		Exit non-zero when any row has component or eval execution errors
--output <path>	-o	Write machine-readable per-item result JSON

Removed or unsupported zhanla run flags include --model-endpoint, --parallel, and --verbose.

Valid combinations

• Provide exactly one component source: local target or --web-config.
• Provide exactly one dataset source: --dataset or --web-dataset.
• Do not combine --eval and --web-eval.
• --web-config only supports --web-config + --web-dataset + --web-eval.
• --dry-run cannot combine with --web-config or --web-eval.
• --max-rows must be positive.
• Local component and local eval files must use the same language.
• Skill is prompt-only configuration and cannot be run directly.

Target syntax and discovery

Use file.py[:component-key] for Python and file.ts[:component-key] for TypeScript. The suffix is the SDK component key, not the Python variable name, TypeScript export name, or display name.

Python discovery imports the file and scans module-level zhanla component instances.

TypeScript discovery uses the SDK helper binary internally. Components must be exported, but normal users should run the main zhanla CLI.

If a target accidentally uses a display name or variable/export name, the CLI rejects it and suggests matching keys when possible.

Dataset formats

JSON

JSON datasets must be a top-level array whose first item contains _schema:

[
  {
    "_schema": {
      "message": "string",
      "expected_output": "object"
    }
  },
  {
    "_config": {
      "name": "support tickets",
      "description": "Priority classification cases"
    }
  },
  {
    "message": "Reset my password",
    "expected_output": { "priority": "normal" }
  }
]

Leading _schema and _config rows are metadata and are not executed.

Object-shaped JSON datasets with schema and rows are not accepted.

CSV

CSV datasets use the header row as field names:

message,expected_output
Reset my password,normal

CSV values are loaded as strings.

Empty datasets

If the resolved dataset has no rows, the CLI exits with:

Dataset is empty.

Eval input contract

Local evals and CLI-managed eval-only runs receive canonical text kwargs:

• model_input: serialized dataset row, or serialized row input when present.
• model_response: serialized component output.
• expected_output: serialized row expected_output, row output, or the full row.

Python CodeEval functions receive keyword arguments:

def score(model_response, expected_output=None, model_input=None, **_):
    response = zhanla.parse_json_response(model_response)
    expected = zhanla.parse_json_response(expected_output or "{}")
    return {"score": 1.0 if response == expected else 0.0}

TypeScript CodeEval functions receive one kwargs object:

fn: ({ model_response, expected_output }) => {
  const response = model_response ? JSON.parse(model_response) : {};
  const expected = expected_output ? JSON.parse(expected_output) : {};
  return { score: response.priority === expected.priority ? 1.0 : 0.0 };
}

model_response_format / modelResponseFormat is synced as eval metadata for platform compatibility. Current local and CLI-managed eval-only execution still passes strings, so parse structured data inside the eval body.

Local execution semantics

• Tool: runs local code and normalizes non-object output to {"result": value}.
• CodeEval: runs local scoring code and normalizes non-object output to {"score": value}.
• Skill: raises an error when executed directly.
• Agent: requires model plus client or runner.
• LLMProcessor: requires model plus client or runner.
• LLMEval: requires model plus client or runner.
• Checklist: runs child evals and computes a weighted score.
• EvalTree: routes through branches and aggregates leaf scores.
• Orchestration: executes the DAG and returns the final executed step output.

The CLI validates the first component output against output_schema when present.

List commands

zhanla list datasets
zhanla list autoraters
zhanla list components

Useful filters:

zhanla list datasets --component-type tool --name support
zhanla list autoraters --component-type agent --component-id component-uuid
zhanla list components --component-type orchestration --name support
zhanla list components --all --verbose

Supported component filters are agent, skill, tool, and orchestration.

Output JSON

--output FILE.json writes:

{
  "mean_score": 0.9,
  "run_id": "run-id",
  "items": [
    {
      "item_id": "0",
      "model_input": "...",
      "model_response": "...",
      "expected_output": "...",
      "score": 1.0,
      "reason": "..."
    }
  ]
}

For managed runs, output is written after remote results are available.

File locations

Path	Purpose
.zhanla/credentials.json	Repo-local SDK credentials and org metadata
.zhanla/version_cache.json	Repo-local update-notifier cache

Running tests

python -m pytest packages/cli/tests -v

Broaden to SDK tests when discovery, execution, manifests, eval contracts, or helper integration change.