Clousight Bench · 指北测评

云计算指北 / Clousight 出品的云产品可复现测评框架 — agent runtimes today; big data clusters, databases, compute and messaging via the same abstraction.

Clousight Bench is the measuring stick of Clousight: open methods anyone can reproduce; evidence-graded results, never a blended vanity score.

0.2.0 Developer Preview. The local reference baselines are runnable, and every cloud adapter runs end-to-end in mode: mock with no account. The Aliyun AgentRun adapter is experimental — its in-tree runtime provider (with its ECI probe carrier, reaper and Terraform) has run a full 27-task live campaign (cn-hangzhou, 2026-08-15: 25 completed + 2 honestly unsupported). It is not yet promoted to wired (reserved for a repeatedly-validated path); the other clouds are skeletons.

Repository status. This repository is public and Apache-2.0 licensed. main is protected: every change lands through a pull request that passes ruff, pytest and the no-cloud smoke on Python 3.10–3.13 plus a separate installed-wheel smoke. No approving review is required, force pushes and branch deletion are blocked, and the rules bind administrators too. Commercial plugins are developed in a separate private repository and are not required to run anything in this one.

Run csbench list --verbose to inspect task metadata and adapter readiness.

Adapter status is part of the public contract: reference and wired can run; experimental can run with preview caveats; skeleton is discoverable for contributors but is rejected before preflight. A skeleton cloud has two runnable paths without editing its adapter: mode: mock exercises the whole harness against the in-process simulated runtime with no account; and in real mode it becomes runnable once a runtime provider is registered for that cloud through the clousight_bench.runtime_providers entry point. The open core already ships and registers that provider in-tree for aliyun-agentrun (experimental — validated by a full 27-task live campaign) and aws-agentcore; additional clouds wire the same way — an open-core or third-party pack registering a provider, not patching the adapter, is what wires them.

The reproducibility contract (read this first)

Every number this framework produces is classified before you trust it:

• Controlled-variable measurement (evidence layer C) — the tested variable is controlled by our runner and mock services. Precisely reproducible: run the same code against your own account and challenge our numbers.
• Environment observation (evidence layer B) — cold starts, network-sensitive latency. The method is reproducible; the numbers depend on your network / hardware / region.
• Documentation reading (evidence layer A) — vendor-stated limits we did not measure.
• Marketing material (evidence layer D) — never used as load-bearing evidence.

Every result record is schema 0.2 and is attributable on three independent axes, so you can tell whether two numbers are even comparable:

Each measurement carries its own value, unit and evidence layer, and each finding carries a stable code, a severity and its evidence. A run ends in exactly one status: completed, failed, invalid or unsupported — there is no boolean ok, because "the platform does not support this" and "the run crashed" are different results. We publish per-dimension results, never a single blended score — blended agent-benchmark rankings have near-zero cross-benchmark agreement.

Results written by an older version use schema 1.0. Convert them with:

csbench migrate-results old-results/ --output new-results/

The migrator never writes in place, never fabricates a fingerprint (unknown ones are the literal string unknown), and produces byte-identical output when run twice.

One run is not a measurement. Repeat a benchmark and get a distribution:

csbench run --domain agent-runtime --task T1.3 --platform local-sim \
  --repeat 5 --warmup 1

The warmup run is discarded; the five measured runs are reduced to mean, stdev, p95 and cv (numeric) or a value distribution (labels), and only runs that share a benchmark and environment fingerprint are ever pooled. csbench report flags any cell whose numbers are not actually comparable.

Why another benchmark framework

Existing benchmarks pin the runtime and swap the model to report accuracy. Nobody independently benchmarks the platform runtime engineering — session hosting, tool-failure recovery, trace completeness, cost attribution — of managed cloud products. Clousight Bench does, and the abstraction generalizes: workloads differ wildly across cloud products, but the pipeline is identical:

provision -> setup -> execute -> collect -> teardown -> score -> report

The core only orchestrates that lifecycle. Everything product-specific is a plugin:

Domains register via the clousight_bench.domains entry point — third-party packs install like any Python package and appear in csbench list.

Quick start (no cloud account needed)

git clone https://github.com/clousight/clousight-bench.git && cd clousight-bench
python3 -m venv .venv && .venv/bin/pip install -e ".[dev]"

# what is installed?
.venv/bin/csbench list

# T1.3 tool-failure recovery against the local simulated runtime:
# a deterministic fault hits the 3rd tool call; watch two runtime policies react
.venv/bin/csbench run --domain agent-runtime --task T1.3 --platform local-sim
.venv/bin/csbench run --domain agent-runtime --task T1.3 --platform local-sim \
    --config configs/local-sim.fail-fast.yaml

# J1.1 wordcount through the packaged local-process workload:
.venv/bin/csbench run --domain bigdata-emr --task J1.1 --platform local-process

# aggregate everything under results/ into a comparison report
.venv/bin/csbench report

可选时序存储（Parquet + DuckDB）：pip install clousight-bench[store]

查询与分析（跨云/跨产品统一）——把结果展平成 records / measurements / findings / series 四张视图跑 SQL，或导出给 notebook/BI（详见 docs/querying.md）：

csbench query "SELECT platform, avg(value_num) FROM measurements WHERE name='cold_start_ms' GROUP BY platform"
csbench export measurements --out m.parquet

成本按 list（厂商原始价）→ discount（折扣）→ net（真实应付） 三维呈现：公开价 feed 走 CLOUSIGHT_PRICING_DATA，私有折扣走 CLOUSIGHT_PRICING_DISCOUNTS（详见 docs/querying.md）。

测评集分发（内置 / 公开远程下载校验 / 私有授权）见 examples/asset-manifests/（assets: 三层模板 + 公开数据集样例）。

Expected: the default (auto-retry) run ends recovery_mode=auto-retry, final_state=completed; the fail-fast run ends recovery_mode=fail-fast, final_state=aborted. The mock tool universe is pinned and fault injection is counter-based (the Nth call fails, no randomness), so the run is replayable by construction.

Benchmarking a real platform

Credentials are never stored in configs. Clousight Bench reuses the cloud's own default credential chain — the same env vars / CLI profile / role you already use for aws, aliyun, etc. — so you don't mint a benchmark-only secret.

# 1. scaffold a private config + .env.example (auto-gitignored, no secrets)
csbench init aws                 # or: aliyun / huawei / volcengine
#    -> agent-runtime-aws.local.yaml  +  .env.example

# 2. provide credentials via ANY of:
#      export AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=...   (or copy .env.example -> .env)
#      set target.profile in the config                          (a named CLI profile)
#      an attached role / SSO / instance metadata                (nothing to set)

# 3. expose the pinned mock tool universe where the cloud runtime can reach it
#    (localhost is NOT reachable from a cloud runtime — use a tunnel / cloud function)
python -m clousight_bench.domains.agent_runtime.mock_tools --port 8770
#    then set mock_base_url in the config to that public URL

# 4. preflight — checks provider, SDK, credential chain, and mock reachability
csbench doctor --config agent-runtime-aws.local.yaml

# 5. run
csbench run --domain agent-runtime --task T1.3 --platform aliyun-agentrun --config your.local.yaml

Adapters surface the runtime's own retry / session / trace behavior and must never touch tasks or scoring. You pay your own cloud bill; you get numbers for your own account, network and region. That is the point.

Status

• Core: lifecycle orchestrator, unified RunSpec/ResultRecord schema, entry-point plugin registry, cross-language workload protocol, Markdown + self-contained HTML/ECharts comparison report, DuckDB-backed csbench query, cost budget + live-run gate + resource reaper (csbench sweep)
• Onboarding: csbench init (scaffold private config + .env.example, auto-gitignored) and csbench doctor (preflight credentials + connectivity); credentials reuse the cloud's default chain (env / profile / role), never stored in configs
• agent-runtime: fault-injectable mock tool server, local-sim adapter, 27 tasks end-to-end on local-sim, spanning provisioning (T0.1/T0.2), runtime behaviour (T1.1 cold/warm start · T1.2 state persistence · T1.3 fault recovery · T1.4 sustained load & tail · T1.5 warm retention · T1.6 soak · T1.7 rate limiting · T1.8 timeout/cancellation · T1.9 TTFT · T1.10 retry storm · T1.11 concurrent state writes · T1.12 head-of-line blocking · T1.13 startup-convergence curve · T1.14 idle-timeout config honor), tools (T2.1 registration paths), observability (T4.1 trace completeness · T4.2 OTel export · T4.3 metrics/log signals · T4.4 span propagation · T4.5 export latency), cost (T5.1 attribution · T5.2 elasticity · T5.3 idle/scale-to-zero · T5.4 concurrency ceiling) and isolation (T6.1). Latency-class "data-plane" tasks run through the run_data_plane_probe adapter seam, so a wired cloud can sink the load generation into an in-region probe instead of the operator's machine.
• agent-runtime first real-cloud run: aliyun-agentrun (experimental) ran a full 27-task live campaign (cn-hangzhou, 2026-08-15 — 25 completed + 2 honestly unsupported) via its in-tree runtime provider + ECI probe carrier + reaper + Terraform. Not yet promoted to wired (reserved for a repeatedly-validated path).
• agent-runtime: wire the remaining clouds — huawei-agentarts / volcengine-agentkit / aws-agentcore remain skeletons, wired the same way via the clousight_bench.runtime_providers entry point.
• bigdata-emr: J1.1 wordcount smoke via the cross-language workload protocol and local-process reference adapter
• bigdata-emr: wire the aws-emr Terraform-backed adapter (skeleton in-tree)
• database / compute / messaging domain packs

Contributing

Sign your commits (git commit -s, DCO). Adding a platform = one adapter file + one example config; adding a dimension = one task file with its scoring and declared evidence layer; adding a product category = one DomainPack. PRs that change task or scoring logic for a shipped dimension require a version bump and a changelog entry — published numbers must stay attributable.

This checkout has no origin remote — commit/push/PR/merge go through scripts/gitsync.sh (requires the clousight-dev gh account and forces commit identity to that account's noreply email; push refuses main — land via a feature-branch PR with squash merge; run cp .gitsync.env.example .gitsync.env once to set the target repo).

License

Apache-2.0