kbws-forge-cli

Scaffolding and local development CLI for the kbws-forge-runtime agent framework. Generate a production-shaped FastAPI agent service and inspect its agent runs in a local browser UI.

Install

pip install kbws-forge-cli

Requires Python ≥ 3.13 and uv.

Usage

Run forge init and answer the prompts (Vite-style interactive picker):

forge init
✔ Project name: … my-agent
✔ Select a template: › service-agent

Or pass everything explicitly for non-interactive/scripted use:

forge init my-agent                       # name given, template picked interactively
forge init my-agent -t base-agent         # fully non-interactive

Then start developing:

cd my-agent
uv sync                                   # installs kbws-forge-runtime from PyPI
uv run uvicorn app.main:app --reload      # dev server
uv run pytest                             # tests (fake models, no cost)

Trace UI

With the generated service running, start the local Trace UI in another terminal:

forge trace --api-url http://127.0.0.1:8000/api/v1

The command serves a loopback-only browser UI at http://127.0.0.1:8765, modeled on the Google ADK / LangSmith trace consoles: a session list, a per-session Turns panel (conversation context that stays visible while you debug), and an execution tree with a Chrome-devtools-style time ruler and waterfall bars. It connects automatically on load (API URL comes from the CLI's --api-url; connection settings — including an optional API key — are remembered in the page's localStorage) and ships light/dark themes.

The service records every run (from any client — curl, Yaak, scripts) into a server-side trace store exposed via GET /api/v1/traces and GET /api/v1/traces/{run_id} (generated by the service-agent template, so any forge init project supports the panel out of the box). Traces produced outside the UI appear automatically (the list auto-refreshes every few seconds). The runtime persists runs to logs/traces.json (bounded, newest-first), so they survive restarts.

Evaluating agents (CLI)

forge eval drives the service's eval API and produces CI-ready output:

forge eval run <suite> [--mode live|replay] [--cases a,b] [--repetitions N]
                      [--fail-under 0.9] [--report json junit] [-o out/]
forge eval compare --baseline <eval_run_id> [--candidate <id> | --live]
forge eval dataset export <suite> -o cases.jsonl
forge eval dataset import cases.jsonl --name golden --agent test_agent

• run executes a suite on the service (live calls the model, replay re-grades recorded runs with zero external calls), prints a per-case score table, writes JSON/JUnit4 reports, and --fail-under gates the exit code for CI.
• compare diffs two eval runs per case (regressions list, score deltas) and exits 1 when a case regressed or the candidate falls below --fail-under.
• dataset exports a suite's cases to JSONL (graders as names) and imports a JSONL file back into a code-first evals/ module.
• --api-key also reads the FORGE_API_KEY env var.

Browser e2e (opt-in)

The Trace UI has a Playwright regression suite (e2e/) that runs the full connect → run → inspect → reload-persistence flow against a hermetic Node stub of the agent API (deterministic SSE events, no real model or network):

FORGE_E2E=1 uv run pytest tests/test_e2e_trace_ui.py -v

It starts the real forge trace server on an ephemeral port and drives a real Chrome/Chromium. Prerequisites on first opt-in run: Node.js + npm (the harness auto-installs @playwright/test into e2e/) and a Chrome/Chromium binary (or npx playwright install chromium, then drop channel: "chrome" in e2e/playwright.config.mjs). Without FORGE_E2E=1 the test is skipped, so the default test suite stays dependency-free.

Templates

New templates placed in the CLI's templates/ directory appear in the interactive picker automatically.

Generated project

forge init my-service produces:

my-service/
├── agents/                      # business units: one directory per agent
│   ├── <module_name>/           #   main agent
│   │   ├── agent.py             #   exports `agent`; middleware example
│   │   ├── prompts.py           #   composable Prompt components (code-first)
│   │   └── tools.py             #   this agent's tools
│   └── extract/                 #   structured-output example (output_schema)
│       ├── agent.py             #   pydantic schema -> Agent(output_schema=...)
│       └── prompts.py
├── app/                         # technical layering
│   ├── main.py                  #   create_app() + lifespan (load_agents)
│   ├── core/                    #   config / errors / response / security / logging
│   ├── api/v1/                  #   agents / sessions / chat / chat_stream / health
│   ├── schemas/                 #   request & response models
│   ├── services/                #   chat orchestration
│   └── providers/               #   LLM factory
├── tests/                       # unit / api / integration (real-provider gated)
├── scripts/run.sh
└── .env.example                 # multi-env config template

Endpoints: GET /api/v1/health · GET /api/v1/agents · POST /api/v1/sessions · POST /api/v1/chat (supports variables, returns parsed for structured output) · POST /api/v1/chat_stream (SSE), all behind X-API-Key / Bearer auth except health.

Development

uv sync
uv run pytest packages/forge-cli/tests

License

MIT License