Skip to main content

AI Forest Agent Performance Test (AFPerf)

A local-first Python package and visual configuration studio for MCP / REST API / Agent / LLM gateway benchmarking and assumption-based Kubernetes capacity forecasting. Engineering prototype v0.7.0.

Safety: The default project runs mock tests only. Live traffic requires approved endpoints and --ack-live. Do not publish the unauthenticated Streamlit dashboard to the internet. Keep .env secret.

Installation

After the first approved PyPI release, install the published distribution with:

python -m pip install afperf
afperf init my-benchmarks
cd my-benchmarks
afperf start

Note: pip install afperf requires a successful, publicly available PyPI release. Before publishing, use the local wheel. The single-command install includes Streamlit.

Local source installation (Windows Git Bash, macOS, Linux)

cd agent-performance-test
python -m pip install -e '.[dashboard,test]'
afperf --version

Both afperf and aiforest-perf run the same CLI. agent-performance-test remains available as a compatibility entrypoint. Unlike the old apt name, afperf does not shadow the Linux package manager.

Install an offline-built wheel

python -m pip install ./dist/afperf-0.7.0-py3-none-any.whl

The wheel contains application code, dashboard and JSON templates, not vendored Python dependencies. Runtime dependencies still need to be installed from an approved package index or internal wheelhouse. The ZIP includes the wheel separately.

Quick start

afperf init my-benchmarks
cd my-benchmarks
afperf start                  # local browser dashboard http://127.0.0.1:8501

The dashboard lets users design, validate and download benchmark and capacity JSON. It explains each chart, axis and metric; 1c, 2c and 4c mean one, two or four concurrent HTTP test workers, not CPU cores or Kubernetes replicas.

Benchmark and forecast

afperf performance benchmark.json --out results/benchmark
afperf forecast forecast.json --out results/forecast

afperf benchmark benchmark.json aliases afperf performance. afperf forecast benchmark.json also works when a sibling forecast.json exists: it explicitly prints that it is using the capacity profile. Benchmark results do not automatically determine sustainable per-pod TPS. Enter measured or assumed RPS-per-pod in forecast.json.

Approved live testing only

Edit benchmark.json to switch mode to live; use only approved read-only operations with bounded load:

afperf performance benchmark.json --ack-live --out results/approved-live

Live traffic is rejected unless acknowledged. The current library additionally enforces conservative caps on rate, concurrency and test length. Synthetic payloads are preferable to confidential user data.

Docker scaffolding

afperf docker
sh docker/start.sh build
sh docker/start.sh run
# Or: sh docker/start.sh compose
# Shutdown: sh docker/start.sh down

afperf docker creates docker/Dockerfile, docker/src/, docker/config/, docker/docker-compose.yml, .dockerignore and docker/start.sh. It does not build or run containers automatically. Source is bundled locally, so Docker does not depend on this package being published on PyPI. No .env or tokens are copied into the generated image. The Dockerfile still downloads Python dependencies when it builds. Use host.docker.internal rather than 127.0.0.1 for endpoints on the host under Docker Desktop.

Default Docker dashboard binds only to 127.0.0.1:8501; it does not expose remote/public access. Authentication and access controls are not yet suitable for a multi-tenant enterprise SaaS.

Project layout

my-benchmarks/
  benchmark.json           # versioned MCP/API/agent/LLM tests
  forecast.json            # capacity assumptions and service profiles
  .env.example             # safe variable names only
  .env                     # optional local secrets: create yourself
  examples/                # AI Forest starter profiles
  results/                 # reports, raw CSV, charts and K8s templates
  docker/                  # generated offline source and Docker scaffold

Why it exists

Performance benchmarking and Kubernetes capacity planning are frequently disconnected, especially for workflows composed of tools, memory, retrieval and model calls. AFPerf aims to bring representative synthetic tests, p20/p50/p95/p99 latency, errors, token usage and versioned capacity assumptions into one local-friendly workflow.

The white paper explains the approach, limitations, competitive positioning and roadmap. The original AFPerf code is published under Apache License 2.0, with AI Forest attribution in NOTICE. Third-party library licenses apply independently.

Verification

The Python CLI, project initializer, mock benchmark, forecast, offline Docker generator, and existing unit tests can be tested without a network connection. The Streamlit UI and actual Docker image execution must be verified on a developer workstation / CI runner with those tools installed.

CLI naming (v0.6.1)

The v0.6.0 apt executable was removed because Linux distributions use apt for package management. Use afperf (recommended) or aiforest-perf. The Python distribution is renamed to afperf starting with v0.7.0. The Python import path remains agent_perf. This package has not been published to PyPI in this release. A preliminary name search is not trademark clearance.

Building and publishing the official package

See RELEASING.md. bash scripts/release.sh --version 0.7.0 --build-only builds locally; --publish performs authenticated release actions after safety checks and legal license approval. Never paste credentials into a command.

License

AFPerf is licensed under the Apache License, Version 2.0 (Apache-2.0). See LICENSE, NOTICE and LICENSING.md. AI Forest names and marks remain separate from the software license.

Metadata

Release files for afperf 0.7.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for afperf 0.7.0
File Size Uploaded Provenance Actions
afperf-0.7.0.tar.gz 80.7 kB Without provenance Details for afperf-0.7.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for afperf 0.7.0
File Interpreter ABI Platform Actions
afperf-0.7.0-py3-none-any.whl Python 3 none any Details for afperf-0.7.0-py3-none-any.whl

Total release size: 152.5 kB

Release files / afperf-0.7.0.tar.gz

Download URL afperf-0.7.0.tar.gz
Size 80.7 kB
Tags Source
SHA-256 checksum
How to use checksums
ec1bfd68d52e32ea4665709f298f53b4e3ab4515c85f68e28486e26bc4dadc00
BLAKE2b-256 checksum
How to use checksums
f91fdc271318074c2bdae2c8268d814e130f3da81bf80f1fa16f8cc8132673bd
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.11.17

Release files / afperf-0.7.0-py3-none-any.whl

Download URL afperf-0.7.0-py3-none-any.whl
Size 71.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
60b66465a182c2f3bb3be04aa38106ed4897bf7fd9e51f5c140ba599f23691a8
BLAKE2b-256 checksum
How to use checksums
bd36c5bab4d33f1306f1ee7cbafeadc4f791019f893d0da4e20f3f65b477bc7f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.11.17

Security for afperf 0.7.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.7.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page