agent-tco
agent-tco compares model assignments for agentic workflows on two objectives: Autonomous Completion Rate (ACR) and Expected Total Cost of Ownership (E[TCO]) per incoming task. It supports exact sequential and structured-block evaluation, approximate profile-based execution-graph simulation, and measured validation of shortlisted configurations.
Cost terminology
The package name and public API use TCO as an umbrella term for expected trajectory cost. The computed boundary is determined by the supplied model and cost inputs; the library does not automatically add every acquisition, integration, infrastructure, governance, maintenance, or retirement cost that may belong to a full lifecycle TCO study.
The companion paper
uses the narrower term expected operating cost (EOC) for inference, retry,
human-review, and terminal manual-replacement cost per incoming task. With the
paper's supplied profiles, the library value exposed as
expected_tco is exactly this EOC quantity. The names agent-tco, agent_tco,
expected_tco, and recompute_tco() are retained for package and API
compatibility.
Outcome model
Each StepConfig defines mutually exclusive per-attempt probabilities,
conditional on execution reaching that touchpoint with its required predecessors
in a passing state.
qis autonomous completion on one attempt.hitl_probis human-assisted step completion, which advances to downstream work while removing the trajectory from ACR.partial_probis human-assisted partial completion.- The residual
1 - acr - hitl_prob - partial_probis retryable non-completion.
The three supplied probabilities must sum to at most one. A non-completion receives one retry. Two consecutive non-completions invoke the configured manual fallback.
How workflow ACR and E[TCO] compose
For a required series chain with autonomous-passage probabilities a_k, workflow
ACR is prod(a_k) under the evaluator's conditional-independence assumption.
The structured evaluator recursively applies the policy of each block instead of
blindly multiplying arbitrary subgraphs: eager AND has the series reliability
formula, independent OR has 1 - prod(1 - a_k), ordered fallback invokes later
branches only after earlier failures, and bounded retry repeats its child.
Costs add along each executed trajectory and E[TCO] averages those trajectory
costs. Consequently, a stop-on-failure chain with invocation costs c_k has
c_1 + a_1*c_2 + a_1*a_2*c_3 + ... before review and manual-fallback terms; it
does not normally pay for unreached nodes. Eager parallel blocks do pay for every
invoked branch. This reach-weighted accounting is why topology affects cost as
well as reliability.
Features
- Monte Carlo simulation with Wilson confidence intervals for scenario probabilities and ACR
- Exact finite-state evaluation for categorical sequential chains
- Full Cartesian enumeration and sensitivity-screened configuration sets
- Pareto-frontier analysis on ACR and E[TCO]
- Hoeffding intervals for expected cost using a true per-run cost bound
- Categorical
StepOutcomeexecutor hook for measured model calls - JSON persistence and review/fallback repricing without re-running behavior
- Exact compositional evaluation for nested series, eager AND, ordered-fallback OR, eager OR, and bounded-retry blocks
- Exhaustive and deterministic budgeted guided search for structured-topology assignments
- Discrete-event Monte Carlo for finite execution graphs with shared descendants, joins, routes, bounded cycles, first-success races, cancellation billing, and common-cause failures
- Budgeted DAG assignment screening with uncertainty intervals
- Balanced synchronous or bounded-concurrency asynchronous experiments through a user-supplied real-agent executor
- Per-run measured evidence, measured Pareto extraction, JSON export, and frontier plotting
- Reproducible stylized computational tests of the five conditional hypotheses
Installation
pip install agent-tco
Python 3.12 or later is required.
Minimal example
from agent_tco import (
CostParams,
StepConfig,
WorkflowConfig,
evaluate_exact,
pareto_frontier,
)
costs = CostParams(
labor_rate=60.0,
review_time_hours=0.25,
manual_process_cost=4.50,
)
lightweight = WorkflowConfig("CC", [
StepConfig(1, "C", q=0.78, hitl_prob=0.08,
partial_prob=0.02, inference_cost=0.003),
StepConfig(2, "C", q=0.74, hitl_prob=0.10,
partial_prob=0.03, inference_cost=0.003),
])
mixed = WorkflowConfig("BA", [
StepConfig(1, "B", q=0.88, hitl_prob=0.04,
partial_prob=0.01, inference_cost=0.08),
StepConfig(2, "A", q=0.94, hitl_prob=0.02,
partial_prob=0.005, inference_cost=0.50),
])
results = [evaluate_exact(config, costs) for config in (lightweight, mixed)]
frontier = pareto_frontier(results)
for result in frontier:
print(result.config.config_id, result.acr, result.expected_tco)
E[TCO]/ACR is available as executive_metric. It is a descriptive cost-effectiveness ratio. Compare E[TCO] directly with manual task cost when both alternatives deliver comparable task value.
Repricing stored results
save_results() stores the original review and manual-fallback cost parameters.
After loading results, recompute_tco() can change those two cost components
without repeating workflow evaluation:
from agent_tco import CostParams, load_results, pareto_frontier, recompute_tco
stored = load_results("results.json")
updated_costs = CostParams(
labor_rate=75.0,
review_time_hours=0.20,
manual_process_cost=12.0,
)
repriced = recompute_tco(stored, updated_costs)
frontier = pareto_frontier(repriced)
Inference prices are part of each StepConfig and are not changed by
recompute_tco(). Re-evaluate configurations after changing model behavior,
transition probabilities, retry policy, topology, or inference cost.
Screening before measured validation
Profiles should be calibrated from historical traces, controlled evaluations, or
pilot runs. Use exact composition or DAG simulation to reduce the assignment
space, then run the shortlist against representative inputs through the actual
agent stack. ProductionExperimentRunner and
AsyncProductionExperimentRunner randomize a balanced configuration/case
schedule and aggregate observed autonomy, total cost, and latency. The supplied
executor remains responsible for authentication, rate limiting, tracing,
scoring, retries performed by the real agent, and provider billing telemetry.
The measured Pareto frontier—not the simulated frontier—is the appropriate
evidence for a deployment decision.
run_stylized_hypothesis_tests() exercises model-implied comparative statics for
convex capability prices, finite reliability ceilings, dependent redundancy,
labor rates, and importance/review alignment. Its constructed results are
implementation checks and illustrations, not external empirical validation.
Reproducing the companion paper
The exact constructed profiles used by the paper are included in
agent_tco.paper. In this section, values printed from expected_tco are the
paper's EOC values under its stated operating-cost boundary.
From the repository root, install the package environment and run:
uv sync
uv run python - <<'PY'
from agent_tco import (
evaluate_paper_chain,
evaluate_paper_structured_exhaustive,
evaluate_paper_structured_guided,
pareto_frontier,
)
chain_results = evaluate_paper_chain()
chain_frontier = pareto_frontier(chain_results)
chain_best = min(chain_frontier, key=lambda result: result.executive_metric)
print(
"chain", len(chain_results), len(chain_frontier),
chain_best.config.config_id,
f"ACR={chain_best.acr:.3f}",
f"EOC=${chain_best.expected_tco:.2f}",
f"EOC/ACR=${chain_best.executive_metric:.2f}",
)
structured = evaluate_paper_structured_exhaustive()
structured_best = min(
structured.frontier,
key=lambda result: result.executive_metric,
)
guided = evaluate_paper_structured_guided()
guided_recovers_frontier = {
result.config_id for result in guided.frontier
} == {
result.config_id for result in structured.frontier
}
print(
"structured", len(structured.results), len(structured.frontier),
structured_best.config_id,
f"ACR={structured_best.acr:.3f}",
f"EOC=${structured_best.expected_tco:.2f}",
f"EOC/ACR=${structured_best.executive_metric:.2f}",
)
print("guided", guided.evaluated, len(guided.frontier), guided_recovers_frontier)
PY
Expected output:
chain 243 7 BABBB ACR=0.893 EOC=$2.76 EOC/ACR=$3.09
structured 6561 8 BAAAABAA ACR=0.725 EOC=$4.85 EOC/ACR=$6.70
guided 500 8 True
The first line reproduces the five-step chain's complete seven-point frontier and its minimum-ratio assignment. The second line reproduces the structured workflow's complete eight-point frontier and minimum-ratio assignment. The third line confirms retrospectively that the 500-evaluation guided search contains all eight frontier assignments for this constructed instance.
The shorter Python calls below are useful when inspecting each result set interactively. The five-step example and its full frontier can be reproduced without copying constants from the manuscript:
from agent_tco import evaluate_paper_chain, pareto_frontier
chain_results = evaluate_paper_chain()
chain_frontier = pareto_frontier(chain_results)
print(len(chain_results), [r.config.config_id for r in chain_frontier])
The eight-touchpoint block-tree example exposes both reported searches:
from agent_tco import (
evaluate_paper_structured_exhaustive,
evaluate_paper_structured_guided,
)
exact = evaluate_paper_structured_exhaustive() # all 6,561 assignments
guided = evaluate_paper_structured_guided() # deterministic budget of 500
print(len(exact.frontier), guided.evaluated)
These profiles are constructed evaluator checks, not calibrated production estimates. Both reported frontiers use deterministic exact evaluation; Monte Carlo is not required to reproduce them.
Scope
The sequential API executes chains with one retry after a non-completion. The structured API evaluates nested series, eager AND, ordered-fallback OR, eager OR, and bounded-retry blocks exactly under conditionally independent task outcomes. The DAG API approximately evaluates finite graphs using declared joins, routes, visit/attempt limits, duration profiles, cancellation policy, and common-cause domains. Every cycle must terminate through finite visit/event/time guards; unbounded hidden-state behavior is not modeled. Guided and DAG searches are heuristic and report a frontier only over evaluated assignments. Only exhaustive evaluation certifies its declared finite candidate space under evaluator assumptions.
License
MIT License
Metadata
Release files for agent-tco 0.0.9.3
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| agent_tco-0.0.9.3.tar.gz | 41.3 kB | Without provenance | Details for agent_tco-0.0.9.3.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| agent_tco-0.0.9.3-py3-none-any.whl | Python 3 | none | any | Details for agent_tco-0.0.9.3-py3-none-any.whl |
Total release size: 92.8 kB
Release files / agent_tco-0.0.9.3.tar.gz
| Download URL | agent_tco-0.0.9.3.tar.gz |
|---|---|
| Size | 41.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
f4fcb2786c5cdb09375b4aef1addb152744f5f3c80ea874cc782d7cfb9c71ba9
|
|
BLAKE2b-256 checksum How to use checksums |
8938a785aa9156f3bb1f6653f9f292526e0343646442252c5d571a089a5e73f1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.8.11
|
Release files / agent_tco-0.0.9.3-py3-none-any.whl
| Download URL | agent_tco-0.0.9.3-py3-none-any.whl |
|---|---|
| Size | 51.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
78e272b5661eddc44fd1954472b04be753eb3455e7b1c743586ba2966b5aa1f1
|
|
BLAKE2b-256 checksum How to use checksums |
60238dff51a2679a89ab036c5d18a819e354042ec16cf2697625fd5f0ff9ccce
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.8.11
|
Security for agent-tco 0.0.9.3
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.