maf-cachebench
Experimental. Releases before 1.0 may change or remove APIs. Importing this package emits
MafCachebenchExperimentalWarning.
A benchmark for Microsoft Agent Framework compaction strategies. It measures what each strategy costs in provider prompt-cache hits and what it destroys while doing it: the framework's own strategies and the ones in maf-compaction, twenty in all, on the same conversation, priced at the provider's cached and uncached rates.
This is an independent package. It is not affiliated with or endorsed by Microsoft.
What it measures
Provider prompt caches match on an exact prefix, and every compaction strategy rewrites history inside that prefix, so compaction breaks the cache by construction. The question is what that costs and what it saves. The benchmark plants facts in a conversation's tool results, runs the conversation under each strategy, and then asks for the facts back from the compacted context. Each strategy gets a cost, a cache hit rate, and a count of facts it kept, and the strategies are ranked on cost behind an accuracy bar.
Two harnesses answer two questions:
cachebenchreplays a scripted transcript. Every provider and strategy replays the same scripted structure, with a fixed-width per-cell salt isolating its cache prefix. These controlled workloads support comparisons across providers. It answers how a prompt caches.cachebench_livedrives a real agent, with real replies and real tool calls, so compaction acts on the history an agent would accumulate. Its numbers compare strategies within one model. It answers what a strategy costs an agent.
Install
pip install maf-cachebench[openai] # Azure OpenAI, OpenRouter and the Azure Responses route
pip install maf-cachebench[foundry] # Microsoft Foundry project endpoints
pip install maf-cachebench[mistral]
pip install maf-cachebench[ollama]
pip install maf-cachebench[tiktoken] # then pass --tokenizer tiktoken for exact counts
Five commands land on the path: cachebench, cachebench_live, cachebench_advise, cachebench_recall and cachebench_summary. Each prints its own --help.
Pointing it at a provider
A provider is selected as provider or provider:model, so two models on one provider can sit in one run. Credentials come from the environment:
| Provider | Variables |
|---|---|
azure |
AZURE_OPENAI_ENDPOINT, AZURE_OPENAI_API_KEY, AZURE_OPENAI_CHAT_COMPLETION_MODEL |
azure-responses |
AZURE_OPENAI_ENDPOINT, AZURE_OPENAI_RESPONSES_MODEL, and a working DefaultAzureCredential (Entra authentication) |
foundry |
FOUNDRY_PROJECT_ENDPOINT, FOUNDRY_MODEL, and a working DefaultAzureCredential |
openrouter |
OPENROUTER_API_KEY, OPENROUTER_MODEL; pin OPENROUTER_PROVIDER_ORDER, or you measure the router |
mistral |
MISTRAL_API_KEY, MISTRAL_CHAT_MODEL |
ollama |
OLLAMA_MODEL, optionally OLLAMA_HOST and OLLAMA_API_KEY; this provider caches and never reports it |
Prices are passed as --price-input, --price-cached and --price-output per million tokens, except on OpenRouter where they are discovered.
A first run
# see the plan and the prompt sizes without spending anything
cachebench_live foundry:gpt-5.6-luna --dry-run --strategies none,anchored,tool_summary_anchored
# a live cell: a 120K window, a conversation sized to 1.5 times it, five seeds, records kept
cachebench_live foundry:gpt-5.6-luna --agent harness --context-window 120000 --fill 1.5 --repeats 5 \
--strategies none,tool_summary_anchored,tool_and_user_summary_anchored --summarizer-provider foundry:gpt-5.6-luna \
--price-input 0.20 --price-cached 0.02 --price-output 1.20 --results-jsonl runs/cell.jsonl
# render the table again from the records, calling nothing
cachebench_live --from-jsonl runs/cell.jsonl
--results-jsonl is not optional in practice: a cell runs for hours, and the file is appended one record per seed the moment it is scored, so a stopped run keeps what it paid for. --from-jsonl rebuilds the table and the verdict from the records with no provider configured, and files that measured the same cell merge into one table.
Compaction controls and session experiments
cachebench_live supports maf-compaction>=0.8.0,<0.9, including the SessionPersistence API used by record-bearing roundtrip and restart experiments. Persistence support is checked before any model calls.
cachebench_live foundry:gpt-5.6-luna --dry-run \
--strategies none,tool_and_user_summary_anchored --summarizer-provider foundry:gpt-5.6-luna \
--preserve-prompt-cache --session-mode restart --history-mode replay --stream \
--max-summary-input-tokens 12000 --summary-input-policy chunk \
--summary-options-json '{"max_tokens": 1024}' --summary-timeout-seconds 60
| Area | Controls |
|---|---|
| Recall | --preserve-prompt-cache, --record-instruction, --no-supports-tool-choice, --forced-call-timeout-seconds, --min-groups-before-repeat |
| Summary input | --max-summary-input-tokens, --summary-input-policy skip|chunk, --chunk-cache-size (zero disables the cache) |
| Summary requests | --summary-options-json, --retry-without-summary-options, --summary-timeout-seconds |
| Summary backoff | --max-consecutive-summary-failures and --summary-backoff-seconds, set together |
| Summary prompts | --user-summary-prompt, --user-fold-prompt, --record-merge-prompt |
| Composed overflow | --overflow-policy shed|raise, --estimate-margin |
| Request lifecycle | --session-mode memory|roundtrip|restart, --history-mode incremental|replay, --stream |
roundtrip serializes and reloads the whole framework session between requests. restart also reconstructs the agent and compaction objects. Record-bearing strategies use the native persistence adapter with detached JSON snapshots and an exclusive local lease, saving only completed requests; other strategies restore their available decision state. These experiments use an in-memory store and rebuild objects in the same process: they do not measure disk latency, process crashes or distributed locking. replay resubmits the original conversation, including previously summarized messages, so replayed summaries and restored recall identities are exercised without duplicating stored history. --stream consumes the complete response within the persistence lifetime.
Programmatic callers pass CompactionOptions(...) through StrategyOptions(compaction=...). Existing record, user-summary, fold, coverage and chain controls remain available. Each option and the installed CacheBench, compaction and framework versions become part of the recorded cell identity. Schema 23 records also contain compaction_diagnostics, including forced-call failures, ignored pins, summary replay/skips and final recall-disabled state. Older records remain readable with unknown new settings and diagnostics; they are grouped separately.
Each completed provider attempt is metered, including an attempt rejected and retried by recall middleware. Independent probes restore the complete supported strategy, middleware and gate state, including pin disablement and summary backoff. Costs from older schemas can undercount internal retries and should not be pooled with new runs.
Reading the table
Rows that keep at least --min-correctness of the control's accuracy come first, cheapest first; the rest follow below a line. The columns to read, in order:
seed$is what the conversation cost, the strategy's own summariser calls included and the probes left out. The ranking and the verdict are on it.seed$+-is the spread between the cheapest and dearest seed. A gap smaller than this is not a result, and the verdict line saysNOT SUPPORTEDwhen that happens.seed hit%is the share of the conversation's input the provider served from its cache, probes excluded. This is what compaction did to the cache.factsandlostare the planted facts that survived into the compacted context, and the ones compaction removed.acc1andacc2are the model's accuracy on the scoped and the combined closing questions, asked from the same restored snapshot.dqmarks a row that sent a prompt larger than the window it stood in for. A cell that disqualifies at all leaves the ranking.flagsis where a row says it is not measuring what its name claims. Read it before the money columns.
The full option reference, the flags legend and the measurement design are in the benchmark documentation, and the findings in the compaction overview.
What it depends on
agent-framework-core, pinned to one minor because the benchmark reads the framework's private _compaction helpers; maf-compaction for the strategies it was built to measure; and httpx for the advisor's price lookup. The provider clients are extras.
Licence
MIT. See the LICENSE.
All commands default to the dependency-free estimator. Install the tiktoken extra and pass --tokenizer tiktoken for BPE counts. Explicit price rates must be finite and non-negative.
Metadata
Release files for maf-cachebench 0.5.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| maf_cachebench-0.5.0.tar.gz | 211.8 kB | With provenance | Details for maf_cachebench-0.5.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| maf_cachebench-0.5.0-py3-none-any.whl | Python 3 | none | any | Details for maf_cachebench-0.5.0-py3-none-any.whl |
Total release size: 437.7 kB
Release files / maf_cachebench-0.5.0.tar.gz
| Download URL | maf_cachebench-0.5.0.tar.gz |
|---|---|
| Size | 211.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
056ae27e8a3ab4314c8e53fadc3e9f6fe7c3db5dd2bca71de1d941a84e9a5e93
|
|
BLAKE2b-256 checksum How to use checksums |
752be35a0b6a4d120662fe534c58051e88f7e5a23df1fdd1d14faaed7d642f85
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 28, 2026.
Transparency logRelease files / maf_cachebench-0.5.0-py3-none-any.whl
| Download URL | maf_cachebench-0.5.0-py3-none-any.whl |
|---|---|
| Size | 225.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
86251d5266e14f671a9e25b514d5a66638273d45602b7adf7df51dcbf5747b6a
|
|
BLAKE2b-256 checksum How to use checksums |
c80208c0a26ca485dde401e8bb17f72e8dfb1a426642a150a2bc1c9c2f48640f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 28, 2026.
Transparency logSecurity for maf-cachebench 0.5.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for maf_cachebench-0.5.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 28, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.