Skip to main content

maf-cachebench

PyPI Python License

Experimental. Releases before 1.0 may change or remove APIs. Importing this package emits MafCachebenchExperimentalWarning.

A benchmark for Microsoft Agent Framework compaction strategies. It measures what each strategy costs in provider prompt-cache hits and what it destroys while doing it: the framework's own strategies and the ones in maf-compaction, twenty in all, on the same conversation, priced at the provider's cached and uncached rates.

This is an independent package. It is not affiliated with or endorsed by Microsoft.

What it measures

Provider prompt caches match on an exact prefix, and every compaction strategy rewrites history inside that prefix, so compaction breaks the cache by construction. The question is what that costs and what it saves. The benchmark plants facts in a conversation's tool results, runs the conversation under each strategy, and then asks for the facts back from the compacted context. Each strategy gets a cost, a cache hit rate, and a count of facts it kept, and the strategies are ranked on cost behind an accuracy bar.

Two harnesses answer two questions:

  • cachebench replays a scripted transcript. Every provider and strategy replays the same scripted structure, with a fixed-width per-cell salt isolating its cache prefix. These controlled workloads support comparisons across providers. It answers how a prompt caches.
  • cachebench_live drives a real agent, with real replies and real tool calls, so compaction acts on the history an agent would accumulate. Its numbers compare strategies within one model. It answers what a strategy costs an agent.

Install

pip install maf-cachebench[openai]        # Azure OpenAI, OpenRouter and the Azure Responses route
pip install maf-cachebench[foundry]       # Microsoft Foundry project endpoints
pip install maf-cachebench[mistral]
pip install maf-cachebench[ollama]
pip install maf-cachebench[tiktoken]      # then pass --tokenizer tiktoken for exact counts

Five commands land on the path: cachebench, cachebench_live, cachebench_advise, cachebench_recall and cachebench_summary. Each prints its own --help.

Pointing it at a provider

A provider is selected as provider or provider:model, so two models on one provider can sit in one run. Credentials come from the environment:

Provider Variables
azure AZURE_OPENAI_ENDPOINT, AZURE_OPENAI_API_KEY, AZURE_OPENAI_CHAT_COMPLETION_MODEL
azure-responses AZURE_OPENAI_ENDPOINT, AZURE_OPENAI_RESPONSES_MODEL, and a working DefaultAzureCredential (Entra authentication)
foundry FOUNDRY_PROJECT_ENDPOINT, FOUNDRY_MODEL, and a working DefaultAzureCredential
openrouter OPENROUTER_API_KEY, OPENROUTER_MODEL; pin OPENROUTER_PROVIDER_ORDER, or you measure the router
mistral MISTRAL_API_KEY, MISTRAL_CHAT_MODEL
ollama OLLAMA_MODEL, optionally OLLAMA_HOST and OLLAMA_API_KEY; this provider caches and never reports it

Prices are passed as --price-input, --price-cached and --price-output per million tokens, except on OpenRouter where they are discovered.

A first run

# see the plan and the prompt sizes without spending anything
cachebench_live foundry:gpt-5.6-luna --dry-run --strategies none,anchored,tool_summary_anchored

# a live cell: a 120K window, a conversation sized to 1.5 times it, five seeds, records kept
cachebench_live foundry:gpt-5.6-luna --agent harness --context-window 120000 --fill 1.5 --repeats 5 \
  --strategies none,tool_summary_anchored,tool_and_user_summary_anchored --summarizer-provider foundry:gpt-5.6-luna \
  --price-input 0.20 --price-cached 0.02 --price-output 1.20 --results-jsonl runs/cell.jsonl

# render the table again from the records, calling nothing
cachebench_live --from-jsonl runs/cell.jsonl

--results-jsonl is not optional in practice: a cell runs for hours, and the file is appended one record per seed the moment it is scored, so a stopped run keeps what it paid for. --from-jsonl rebuilds the table and the verdict from the records with no provider configured, and files that measured the same cell merge into one table.

Compaction controls and session experiments

cachebench_live supports maf-compaction>=0.12.0,<0.13, including the SessionPersistence API used by record-bearing roundtrip and restart experiments and the per-part media selector. Persistence support is checked before any model calls.

cachebench_live foundry:gpt-5.6-luna --dry-run \
  --strategies none,tool_and_user_summary_anchored --summarizer-provider foundry:gpt-5.6-luna \
  --preserve-prompt-cache --session-mode restart --history-mode replay --stream \
  --max-summary-input-tokens 12000 --summary-input-policy chunk \
  --summary-options-json '{"max_tokens": 1024}' --summary-timeout-seconds 60
Area Controls
Recall --preserve-prompt-cache, --record-instruction, --no-supports-tool-choice, --forced-call-timeout-seconds, --min-groups-before-repeat
Summary input --max-summary-input-tokens, --summary-input-policy skip|chunk, --chunk-cache-size (zero disables the cache)
Summary requests --summary-options-json, --retry-without-summary-options, --summary-timeout-seconds
Summary backoff --max-consecutive-summary-failures and --summary-backoff-seconds, set together
Summary prompts --user-summary-prompt, --user-fold-prompt, --record-merge-prompt
Composed overflow --overflow-policy shed|raise, --estimate-margin
Request lifecycle --session-mode memory|roundtrip|restart, --history-mode incremental|replay, --stream
Simulated storage --persistence-scenario none|fenced|stale|branch|migrate|invalid_migration|after_save|save_failure|after_save_failure
Synthetic host hooks --media-profile none|preserve|describe|decline|select_pdf, --media-tokens, --result-rewrite-profile none|preserve|invalid

roundtrip serializes and reloads the whole framework session between requests. restart also reconstructs the agent and compaction objects. Record-bearing strategies use the native persistence adapter with detached JSON snapshots and an exclusive local lease, saving only completed requests; other strategies restore their available decision state. These experiments use an in-memory store and rebuild objects in the same process: they do not measure disk latency, process crashes or distributed locking. replay resubmits the original conversation, including previously summarized messages, so replayed summaries and restored recall identities are exercised without duplicating stored history. --stream consumes the complete response within the persistence lifetime.

Storage scenarios require roundtrip or restart and apply to recall-bearing strategies. Other rows, including the required none control, keep their ordinary local restoration and report storage_evidence=not_applicable_no_recall_stack. Applicable rows report events, commit outcome and notification outcome in compaction_diagnostics.persistence_requests, including when a request fails. branch saves each request under a new ID and leaves its source snapshot intact; live probes branch independently from the seed snapshot. Fault scenarios stop the affected row after the injected failure. A post-save failure can leave a committed snapshot, so request success and commit success are recorded separately.

The host-hook profiles are synthetic fixture policies. describe supplies a fixed description without fetching media or calling a model; decline returns no description; select_pdf describes PDFs and preserves other media turns while allowing later turns to compact. preserve installs only the media estimator. --media-tokens is a host estimate per part, never billed provider usage. The result preserve profile authorizes shortening a single text item while retaining its metadata and non-text items; it declines multiple text items or raw provider representations. invalid deliberately drops media to exercise validation. These policies are for fixtures and controlled experiments; their fixed descriptions do not interpret arbitrary user media. Defaults install none of these hooks.

Record merge loss

Run uv run cachebench_merge_loss for a matched comparison of count-only coverage and coverage that credits distinct values present in the retained messages. Add --fixture-judge to include a scripted judge. Both commands print JSON and run entirely offline.

Six synthetic cases use the same source records and next task. Each row reports the raw and surviving value counts, later messages mentioning each missing value, acceptance, source facts preserved, and whether the projected prompt identifies the correct deployment target. The task reader requires an unambiguous line with the deployment role. Swapping the active and retired identifiers therefore fails even at full token coverage. A false acceptance loses the correct next target and a false refusal rejects a candidate that would have preserved it. The budget counts estimated tokens in the projected fixture text at one decision per case.

The count policy uses compaction's distinctive tokens and default coverage share. The context policy credits equal tokens anywhere in the final retained messages, so a mention with a different role can earn credit. Both context and judge policies require the candidate to meet the configured raw floor. The judge receives one batch of missing values with their source lines, later mention counts, candidate, retained messages, task and versioned rubric. A confident material verdict vetoes acceptance and an all-minor verdict allows it. Uncertain or unavailable judgments use the count decision. Full-coverage candidates have no missing values to judge.

For a live experiment, await maf_cachebench.merge_loss.evaluate(judge=adapter, model_id=deployment, rubric=criteria, rubric_version=version) with a host-owned async callable from JudgeRequest to JudgeAnswer. The adapter supplies one typed Verdict per missing value and optional USD cost. The report records the model identity, per-value probabilities, request count and elapsed time. Unknown cost is JSON null. The request size limit applies to the UTF-8 JSON of the request fields. The timeout cancels a cooperative adapter and caller cancellation propagates. Each evaluation scores fresh fixture projections. Runtime compaction and session replay use their existing policy.

With the default 80% threshold, 60% floor and 45-token text budget, the six fixtures produce the following counts. Crediting surviving tokens accepts the repeated audits but also credits a target mentioned in the wrong role. The scripted judge uses the deployment label to catch that case. Every policy accepts the role swap.

Policy False accepts False refusals Projections over budget
Count 2 3 4
Count with surviving context 3 2 2
Scripted judge 1 1 3

Preserving roles at full value coverage

uv run cachebench_role_preservation compares the missing-value policy with a literal check on structured label–value pairs. Six pairs of source records cover deployment targets and approved amounts. Each pair has three shorter candidates containing both original values. A faithful merge reorders fields, a paraphrase expresses their roles in prose, and a swap assigns each value to the other role. The reference answer comes from the fixture. Each row reports the answer the candidate supplies and the answer left by the policy's decision.

The report separates two development sources from four held-out sources with different wording and values. The missing-value rule makes zero requests at full coverage. The deterministic check compares the complete sets of colon-delimited pairs, normalizing label case and surrounding whitespace. An unrecognized prose format causes a refusal. To measure a third policy, await maf_cachebench.role_preservation.evaluate_roles(judge=adapter, model_id=name). The host-owned async adapter receives the task, source records, candidate and fixed rubric, and returns a CandidateVerdict with a label and its probability. Each candidate is judged independently. The expected answer, split and variant stay with the scorer.

The held-out gate requires zero false accepts, zero false refusals and zero fallbacks. An uncertain response, confidence below 0.8, timeout, malformed response or oversized request falls back to the count decision. The request limit is 4,096 UTF-8 JSON bytes and the timeout is five seconds. Each split reports requests per candidate, median request latency, abstentions and all fallbacks. The opt-in Foundry runner uses the same scoring path and saves native choices and usage alongside the report.

Offline capability scenarios

cachebench_capabilities supplies the content, model responses and storage faults needed to exercise these paths without credentials or external I/O. Persistence cases use the same agent builder and session runner as cachebench_live; media and result cases use the shared strategy builders directly. The command writes one JSON object per scenario and exits nonzero if any contract check fails. Expected injected failures pass only when their observed commit, cleanup and ordering outcomes match the contract.

uv run cachebench_capabilities --scenario all --output capabilities.jsonl
uv run cachebench_capabilities --scenario all --stream --output capabilities-stream.jsonl
uv run cachebench_capabilities --scenario branch --session-mode roundtrip --history-mode incremental
uv run cachebench_capabilities --scenario media_select_pdf --media-tokens 800
uv run cachebench_capabilities --scenario result_invalid

Output files must not already exist. Each row identifies itself as cachebench_capability, schema 1, with offline_simulated evidence, fixture version, package/Python versions, resolved controls, observations and individual checks. These records are separate from cost/cache benchmark records and must not be averaged with them. Live records use schema 25; the reader still accepts schemas 2–24 and retains older control mappings without filling absent settings with current defaults.

Scenario Observation
fenced, stale Non-None token forwarding and atomic simulated stale-token rejection
branch Separate source/destination IDs, unchanged source, retained history and restorable compaction state
migrate, invalid_migration Migration before validation, including rejection before model work
after_save, save_failure, after_save_failure Post-save notification inside the lease, failure before commit, and failure after commit
request_failure, interrupted_stream Failed model response or early consumer exit closes the stream and leaves storage unchanged
media_preserve, media_describe, media_decline, media_select_pdf Protected/selected media, description input, retained source parts and estimated tokens before/after compaction
result_preserve, result_rewrite, result_invalid Default protection, authorized shortening with retained labels/media, and rejected invalid rewriting

--stream applies to persistence cases. all always exercises interrupted_stream with streaming enabled; selecting that case alone requires --stream. Media/result cases apply the strategy directly and do not measure provider streaming. The persistence model fixtures report no billable usage. No result here qualifies real storage durability, distributed lease enforcement, provider cache behavior, model quality or live multimodal handling. Live provider measurements still require cachebench_live and suitable provider/workload qualification.

Programmatic callers pass CompactionOptions(...) through StrategyOptions(compaction=...). Existing record, user-summary, fold, coverage and chain controls remain available. Each option and the installed CacheBench, compaction and framework versions become part of the recorded cell identity. Schema 23 records also contain compaction_diagnostics, including forced-call failures, ignored pins, summary replay/skips and final recall-disabled state. Older records remain readable with unknown new settings and diagnostics; they are grouped separately.

Each completed provider attempt is metered, including an attempt rejected and retried by recall middleware. Independent probes restore the complete supported strategy, middleware and gate state, including pin disablement and summary backoff. Costs from older schemas can undercount internal retries and should not be pooled with new runs.

Record reduction and background merges

The composed row supports --record-budget-share (default 0.5, or none to disable the share), --chain-seconds (default 30, or none to disable the admission deadline), --record-merge-split-depth (default 2) and --rewrite-gain-fraction (default 0.1). A rewrite must save that fraction of its source records and satisfy the existing value-coverage check. Zero gain permits any smaller answer. Zero split depth disables smaller retries after a lossy or truncated batch.

Pass --background-record-merges to cachebench_live to prepare record merges during tools on tool_and_user_summary_anchored. Both agent kinds support it with memory, round-trip or restart sessions, original-history replay and streaming. Other strategy rows keep their existing execution path. The first model call, user summaries, harder rewrites and budget enforcement remain foreground work. Only completed answers accepted by a foreground pass enter persisted history; run completion, failure, cancellation and stream closure discard outstanding work. The synthetic tools require no approval, so the harness approval wrapper is disabled.

Use --tool-delay-seconds 1 to give each synthetic lookup one second of asynchronous work during which a background merge can complete. The default is zero; immediate tool fixtures may leave little opportunity for overlap. The delay applies equally to every live comparison row and is recorded with the controls.

Schema 25 records these controls in the cell identity and adds chain_timeouts (CHAINTIMEOUT), records_reshaped (RECSHAPE) and calls_resumed (CALLRESUME) to diagnostics. Diagnostics also state whether background merging applied to the row and count summarizer_requests_cancelled. A cancelled request without returned usage makes pricing incomplete; it is not counted as free. Tests use deterministic local models to verify lifecycle and state behavior. Live provider cost, quality and latency comparisons require a separate benchmark run.

For schema 26, add --record-merge-diagnostics to an ordinary composed run to retain pressure, eligibility, batch admission and foreground/background outcomes. The diagnostics store counts and bounded samples without record text or fingerprints. Use --qualify-record-merges for the separate live merge workload. It requires a checkpoint merge that preserves its planted source values and appears in a subsequent model call. Background qualification requires foreground application of the answer prepared during the checkpoint lookup.

Reading the table

Rows that keep at least --min-correctness of the control's accuracy come first, cheapest first; the rest follow below a line. The columns to read, in order:

  • seed$ is what the conversation cost, the strategy's own summariser calls included and the probes left out. The ranking and the verdict are on it.
  • seed$+- is the spread between the cheapest and dearest seed. A gap smaller than this is not a result, and the verdict line says NOT SUPPORTED when that happens.
  • seed hit% is the share of the conversation's input the provider served from its cache, probes excluded. This is what compaction did to the cache.
  • facts and lost are the planted facts that survived into the compacted context, and the ones compaction removed.
  • acc1 and acc2 are the model's accuracy on the scoped and the combined closing questions, asked from the same restored snapshot.
  • dq marks a row that sent a prompt larger than the window it stood in for. A cell that disqualifies at all leaves the ranking.
  • flags is where a row says it is not measuring what its name claims. Read it before the money columns.

The full option reference, the flags legend and the measurement design are in the benchmark documentation, and the findings in the compaction overview.

What it depends on

agent-framework-core, pinned to one minor because the benchmark reads the framework's private _compaction helpers; maf-compaction for the strategies it was built to measure; and httpx for the advisor's price lookup. The provider clients are extras.

Licence

MIT. See the LICENSE.

All commands default to the dependency-free estimator. Install the tiktoken extra and pass --tokenizer tiktoken for BPE counts. Explicit price rates must be finite and non-negative.

Metadata

Release files for maf-cachebench 0.7.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for maf-cachebench 0.7.0
File Size Uploaded Provenance Actions
maf_cachebench-0.7.0.tar.gz 247.4 kB With provenance Details for maf_cachebench-0.7.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for maf-cachebench 0.7.0
File Interpreter ABI Platform Actions
maf_cachebench-0.7.0-py3-none-any.whl Python 3 none any Details for maf_cachebench-0.7.0-py3-none-any.whl

Total release size: 505.3 kB

Release files / maf_cachebench-0.7.0.tar.gz

Download URL maf_cachebench-0.7.0.tar.gz
Size 247.4 kB
Tags Source
SHA-256 checksum
How to use checksums
dcb808b61a222e3ebdd20230ad6a6c2a3afd1d39c00249fdcefb200561b81e55
BLAKE2b-256 checksum
How to use checksums
2372a9003438663b069180371552fff93ba92ac78de70d2fffb133f89ae519cf
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.

Transparency log

Release files / maf_cachebench-0.7.0-py3-none-any.whl

Download URL maf_cachebench-0.7.0-py3-none-any.whl
Size 257.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
33d140e4f1c6f30c8c87ac8ebdd7c3b09d98aeacb6510c0edabed7daa2c3720a
BLAKE2b-256 checksum
How to use checksums
8c45fca48682bfa6b43e16bbf3f6a35521680aae2f722bf4b13fd32b1e8e8d36
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.

Transparency log

Security for maf-cachebench 0.7.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for maf_cachebench-0.7.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page