compactbench 0.1.0

Open benchmark for AI conversation compaction methods

CompactBench

Open benchmark for AI conversation compaction methods.

CompactBench measures whether language models still behave correctly after long conversation history is replaced with a compacted representation. It runs adversarial, deterministic, multi-cycle benchmarks and publishes ranked results on a public leaderboard.

• Deterministic generation — same template + seed + version always yields the same case
• Hidden ranked set — public practice cases for development, hidden templates for ranked scoring
• Multi-cycle drift — methods are evaluated across repeated compact → continue → compact loops
• State-fidelity scoring — correctness of retained decisions, constraints, and entities, not output style
• Versioned everywhere — benchmark suite, template, scorer, model, and method versions are recorded with every result

Install

pip install compactbench

Or with uv (recommended for development):

uv pip install compactbench

Quickstart

Run a built-in compactor against the starter suite using a local Ollama model:

compactbench run \
  --method built-in:hybrid-ledger \
  --suite starter \
  --provider ollama \
  --model llama3.2

Generate a single case deterministically for inspection:

compactbench generate --template buried_constraint_v1 --seed 42

Score an existing results file:

compactbench score --results results.jsonl

Writing your own compactor

Implement the Compactor interface and register it.

from compactbench.compactors import Compactor
from compactbench.contracts import CompactionArtifact, Transcript

class MyCompactor(Compactor):
    name = "my-method"
    version = "0.1.0"

    def compact(self, transcript: Transcript, config: dict) -> CompactionArtifact:
        ...

Then run:

compactbench run --method path/to/my_compactor.py:MyCompactor --suite elite_practice

See docs/writing-a-compactor.md for full details.

Leaderboard

The public leaderboard is at https://compactbench.github.io/compactbench/leaderboard.

Submissions are evaluated against hidden ranked benchmark cases by a maintainer-operated runner. To submit:

1. Write and test your compactor locally against elite_practice.
2. Open a PR to submissions/ with your method source and config.
3. A maintainer runs it against the hidden set and merges if it qualifies.

See docs/submitting.md for the full submission protocol.

Project status

v0.1.0 launch-ready. All ten workorders from the implementation roadmap have landed:

• Core: DSL parser, case generation, scoring engine, mock + real providers (Groq / Google AI Studio / Ollama)
• Methods: four built-in compactors (naive-summary, structured-state, hierarchical-summary, hybrid-ledger)
• Runtime: end-to-end compactbench run with drift cycles, JSONL event log, --resume
• Leaderboard: PR-based submission workflow on GitHub-hosted runners, static site fed by a qualification + ranking core
• Content: 15 public Elite practice templates + 15 hidden ranked templates across three launch families
• Release: PyPI trusted-publishing workflow wired up; tag v0.1.0 to ship

See CHANGELOG.md for the full breakdown. Post-launch work (hidden-set content expansion, additional template families, shadow evaluation automation, custom domain) is tracked via GitHub issues.

Contributing

Bug reports, template proposals, and new compactors are welcome. See CONTRIBUTING.md.

Please also read our Code of Conduct.

License

Apache License 2.0 — see LICENSE.