runport 0.1.0

Benchmark any model, prompt or AI workflow.
Know exactly what it cost and what it got right.

Runport is an open-source, local-first benchmarking platform. Point it at an LLM API, a RAG pipeline, an agent, an MCP server or a Dify / n8n workflow, run your own dataset under hard spend caps, and get scores, latency, tokens and dollars you can defend. Nothing is invented, and nothing leaves your machine unless you approve it.

Quickstart · Why Runport · Integrations · Use it your way · Docs · Contributing

_{Results explorer in the local web dashboard. A local HTTP agent run on the bundled synthetic dataset: $0 spend, scores labelled Heuristic, missing tokens shown as None.}

Why Runport

Most benchmark tools give you a leaderboard. Runport gives you numbers you can put in front of a skeptical reviewer.

• Nothing is invented. Missing latency, tokens or cost stay None. They are never defaulted to zero.
• Agreement is not correctness. Accuracy comes only from human-approved ground truth. Heuristic and judge scores are stored and labelled as such.
• Spend is governed, not hoped for. Hard caps are checked before every live request. A request whose cost cannot be resolved is booked UNRESOLVED and blocks the next live run until you reconcile it.
• Nothing runs by surprise. A plan is fingerprinted with SHA-256. Live execution needs an approval bound to that fingerprint, and a changed plan invalidates it.
• Local by default. Results, the spend ledger and approvals live in SQLite on your machine. Exported traces are scrubbed of secrets.
• One engine, every interface. The CLI, terminal UI, web dashboard, Python SDK and MCP server all go through the same guarded runner.

Quickstart

Requirements: Python 3.11+ and uv. Node.js is only needed to rebuild the dashboard; a prebuilt bundle ships with the package.

git clone https://github.com/Sudharsanselvaraj/Runport.git
cd Runport
uv sync --extra dev --extra report

uv run runport init                 # create a local workspace: config, budget policy, database

Run your first benchmark offline. It replays recorded outputs, needs no API key and books $0.00:

uv run runport benchmark --replay --dataset examples/replay-demo/fixtures.jsonl --split ALL
uv run runport budget status
uv run runport web                  # dashboard at http://127.0.0.1:8000

Prefer the terminal? uv run runport ui opens the terminal dashboard and uv run runport opens the interactive launcher.

╭───────────────────────────────────────────────────────────────╮
│ Runport Offline Replay Benchmark                              │
│ Target Model:  google/gemini-3.5-flash-lite                   │
│ Dataset:       fixtures.jsonl                                 │
│ Mode:          REPLAY RUN (Provenance: REPLAYED, $0.00 spend) │
╰───────────────────────────────────────────────────────────────╯
 Recorded Outputs     3 / 3        Fixtures: fixtures.jsonl · Split: ALL
 Mean Latency         151.9 ms     From recorded latencies
 Mean Accuracy        —            Word overlap with the expected answer (labelled cases only)
 Run Spend (USD)      $0.0000      Provenance: REPLAYED ($0.00 spend)

 Case ID   Query Snippet                          Latency   Accuracy   Provenance   Cost
 rep-001   What is the capital of France?         120 ms    —          REPLAYED     $0.0000
 rep-002   Convert 100 EUR to USD                 145 ms    —          REPLAYED     $0.0000
 rep-003   Explain quantum entanglement brief...  190 ms    —          REPLAYED     $0.0000

How it works

The runner applies its guards in a fixed order: authorization → privacy gate → budget reservation → transport → validation → persistence → spend booking. Replays never touch the network or the live ledger.

1. Plan. A dataset (hashed, immutable) and your targets become a fingerprinted execution plan with a worst-case cost projection.
2. Approve. A human approves the exact plan. Any change invalidates the approval.
3. Run. Cases execute under hard caps, with bounded retries and resumable checkpoints.
4. Score. Scoring is separate from inference. Every score carries its kind and its evaluator.
5. Report. Latency, tokens and cost, each with its provenance, in the CLI, the TUI, the web dashboard or a sanitized export.

More detail: Architecture · Universal execution architecture.

Product tour

Plan before you spend. A SHA-256 fingerprint over targets, dataset and caps, a preflight verdict and a worst-case cost projection.	Approve explicitly. Live execution needs a human approval bound to that fingerprint.
Account for every dollar. An append-only SQLite ledger books each request as ACTUAL, ESTIMATED, UNRESOLVED or UNAVAILABLE.	Work from the terminal. The same engine drives the CLI, the Rich terminal UI and the web dashboard, in dark and light themes.

What you can benchmark

Integrations

The connector registry in src/runport/connectors/registry.py is the source of truth for what is supported.

Configuration for each connector: Connector catalogue · Connector reference.

Use it your way

Documentation

Status

Runport is alpha. No version has been tagged or published yet, and interfaces may change.

Available today: the guarded execution path behind every interface, fingerprinted plans and persisted approvals, the atomic budget ledger, score provenance, the stable connectors above, an MCP client connector and a local stdio MCP server, an in-process Python SDK, the CLI, the terminal UI and the web dashboard.

Not yet built (designs and plans live in docs/):

• canonical trace and span storage, and agent trajectory evaluation
• per-step instrumentation for LangGraph and other workflow frameworks
• OpenTelemetry / OpenInference ingestion and Langfuse export
• a TypeScript SDK and a mature remote Python SDK
• authenticated remote MCP and production multi-user hosting

See the roadmap audit for what is verified and what is open.

Contributing and community

Contributions are welcome. Tests run fully offline with sockets blocked: uv run pytest. Start with the contributing guide, then:

• Report a bug
• Request a feature or a connector
• Code of conduct · Security policy

License

Runport is licensed under the Apache License 2.0. See NOTICE for attributions.

_{Runport: benchmark results you can defend, spend you can account for, on your own machine.
If it saves you time or money, a star helps others find it.}