AgentFT
Agent Flow Test - Pytest for AI agents
AgentFT (Agent Flow Test) is a pytest-style evaluation framework for AI agents. It provides composable primitives for tasks, scenarios, agents, and judges, with production-ready infrastructure for concurrency, retries, observability, and regression control.
What's New
- Secure declarative config mode:
--config-json --strict-config - Distributed local orchestration:
aft orchestratefor sharded multi-process runs + merge - Retry policy by error type with richer error taxonomy
- Task-level vs judge-level aggregation with configurable task pass rules
- Ranking support:
aft rank(Elo) - Artifact migration tooling:
aft migrate - Columnar export API (DuckDB/Parquet)
- Environment/episode scenario toolkit and additional preset benchmark suites
- Structured external event sinks for run lifecycle telemetry
- Interactive report drilldown with filtering
Features
- Pytest-like simplicity: Minimal boilerplate, clear abstractions
- Production-ready execution: Async runner, retries, rate limiting, timeouts, fail-fast modes
- Flexible parallelism: Independent controls for agents, tasks, and judges
- Deterministic reproducibility: Seeded shuffle + deterministic sharding
- Strong artifact model: JSONL or SQLite backends with schema versioning and checkpoints
- Regression workflows: Compare, gate, rejudge, analyze, rank, merge
- Extensibility: Plugin registry for agents, scenarios, and judges
- Scenario breadth: List/CSV/JSONL/HuggingFace + rollout-based episode scenarios
- Governance support: API stability policy and governance/compliance guidance
Quick Start
Install from PyPI:
pip install agentft
Run an example:
git clone https://github.com/Geddydukes/agentflowtest
cd agentflowtest
aft run --config examples/config_example.py
Or use in code:
from agentft import RunConfig, run
from agentft.presets import build_math_basic_scenario, ExactMatchJudge
class MyAgent:
name = "my_agent"
version = "0.1.0"
provider_key = None
async def setup(self):
pass
async def reset(self):
pass
async def teardown(self):
pass
async def run_task(self, task, context=None):
return {"response": "42"}
config = RunConfig(
name="quick_test",
agents=[MyAgent()],
scenarios=[build_math_basic_scenario()],
judges=[ExactMatchJudge()],
)
results = run(config)
print(f"Passed: {sum(r.passed for r in results)}/{len(results)}")
Core CLI Commands
aft run
aft run --config path/to/config.py
Strict non-executable mode:
aft run --config-json path/to/config.json --strict-config
Useful overrides:
aft run --config examples/config_example.py \
--max-tasks-parallel 8 \
--max-agents-parallel 2 \
--max-judges-parallel 4 \
--artifact-backend sqlite \
--shard-count 4 --shard-index 1
aft summary
aft summary --run-dir runs/my_run/
Task-level rollup mode:
aft summary --run-dir runs/my_run/ --aggregation-level task --task-pass-rule majority
aft compare
aft compare --run-a runs/base/ --run-b runs/candidate/
aft gate
aft gate --run-a runs/base/ --run-b runs/candidate/ --max-regressions 0
aft analyze
aft analyze --run-dir runs/my_run/ --output-json runs/my_run/analytics.json
aft rejudge
aft rejudge --run-dir runs/my_run/ --config examples/config_example.py
aft merge
aft merge --run-dirs runs/shard0/ runs/shard1/ --output-dir runs/merged/
aft orchestrate
aft orchestrate --config examples/config_example.py --shards 4 --output-dir runs/merged_orchestrated/
aft rank
aft rank --run-dir runs/my_run/
aft migrate
aft migrate --run-dir runs/legacy_run/ --target-schema 1.1.0
Run Artifacts
Each run creates runs/<run_id>/ containing:
results.jsonlorartifacts.dbtraces.jsonlrun_metadata.jsonrun_checkpoint.jsonagent_outputs.jsonl(JSONL backend cache, optional)cached_outputstable inartifacts.db(SQLite backend cache, optional)report.html
Smoke Test Visualization (2026-02-11)
Recent 5-run local validation matrix:
| Run | Mode | Path | Result | Notes |
|---|---|---|---|---|
| R1 | Baseline | runs/smoke_r1_baseline-5a5f7d3c |
4/4 (100%) | Control run |
| R2 | Parallel + Shuffle | runs/smoke_r2_parallel-0e22f470 |
4/4 (100%) | max_agents_parallel=2, max_tasks_parallel=4, seed=123 |
| R3 | SQLite + Cache | runs/smoke_r3_sqlite_cache-5890a078 |
4/4 (100%) | SQLite artifacts and cached outputs |
| R4 | Resume/Checkpoint | runs/smoke_r4_resume-e3bb2645 |
1/1 -> 4/4 (100%) | Early stop then resumed to completion |
| R5 | Orchestrated Shards | runs/smoke_r5_orchestrated |
4/4 (100%) | 2 shards + merged output |
Validation checks:
summaryon R1: pass rate 100%compareR1 vs R2: delta 0.00%, regressions 0gateR1 vs R2: passedanalyzeon R3: pass rate 100%, macro scenario rate 100%rejudgeon cached outputs: 4/4 passed
Pass-rate bars:
- R1:
██████████100% - R2:
██████████100% - R3:
██████████100% - R4:
██████████100% (after resume) - R5:
██████████100%
Extended Functionality Guide
A comprehensive guide for all updated functionality is available at:
docs/UPDATED_FUNCTIONALITY.md
Testing
AgentFT includes 90+ automated tests across runner behavior, storage backends, reporting, CLI, presets, plugins, and resilience checks.
pip install -e ".[dev]"
PYTHONPATH=src pytest -q
Optional columnar export dependency:
pip install -e ".[columnar]"
Project Status
AgentFT is in active development.
Current version: 0.1.0
Links
- PyPI: https://pypi.org/project/agentft/
- GitHub: https://github.com/Geddydukes/agentflowtest
- API Stability:
docs/API_STABILITY.md - Governance:
docs/GOVERNANCE.md - Docs Map:
docs/DOCS_UNIFICATION.md
Metadata
Release files for agentft 0.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| agentft-0.2.0.tar.gz | 67.0 kB | Without provenance | Details for agentft-0.2.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| agentft-0.2.0-py3-none-any.whl | Python 3 | none | any | Details for agentft-0.2.0-py3-none-any.whl |
Total release size: 118.9 kB
Release files / agentft-0.2.0.tar.gz
| Download URL | agentft-0.2.0.tar.gz |
|---|---|
| Size | 67.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
7b46a26291ee12757f5a3f4662e0aa253ae85a05c348eb753165a02b029c5c97
|
|
BLAKE2b-256 checksum How to use checksums |
252d71ce68f07aa29af9422a5869cb0cfe710c9e89547b0289c8e1e863e42491
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.11.14
|
Release files / agentft-0.2.0-py3-none-any.whl
| Download URL | agentft-0.2.0-py3-none-any.whl |
|---|---|
| Size | 51.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
72fe0c5191afbe7b84fd7fc239892f762b5d8d70d95fd1f916a0e2c43227335a
|
|
BLAKE2b-256 checksum How to use checksums |
06c8e04bc75c3e2cccf9a11c3da69547ef000a7e1a3b4d13f816f0ab05aeda45
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.11.14
|
Security for agentft 0.2.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.