acumen 0.0.1.dev0

Build, benchmark, and optimize agentic skills for your Python package.

Agentic skills, tool instructions writen in plain text, allow agents to use tools more succesfuly and efficient. However most python packages do not ship skills with them because developers have no easy way to build and benchmark skills for their tools. Acumen closes this gap. Point it at a Python package and a few evaluation tasks, and it drafts a skill, benchmarks it and improves it across a train/test split so the gains are generalizable.

Many good tools are unusable by coding agents because their maintainers have no way to write a skill for them — or, having written one, no way to tell whether it helps. acumen closes that loop: point it at a Python package and a few tasks, and it drafts a skill, benchmarks it against a no-skill baseline, and improves it across a train/test split so the gains are real generalization, not memorized answers.

• acumen draft — write skills/v1 from the package's own source.
• acumen bench — score a skill against a no-skill baseline, in a scrubbed sandbox where the skill is the only difference between arms.
• acumen improve — refine the skill from its train results, then benchmark again.
• acumen report — aggregate every run into one self-contained report.html: success rate per version, train vs. test. Bars are coloured by model, with a grey bar pooling all of them; pass --palette claude-opus-5=#3b7ea1 (repeatable) to recolour any of them.

You decide when to stop. Every version is benchmarked on both splits, and only train results reach the improver — so a widening train/test gap is a visible sign a skill is overfitting rather than genuinely helping.

Quickstart

# 1. Scaffold a starter config.yaml and tasks.yaml
acumen init

# 2. Fill in config.yaml (repo). Write tasks.yaml by hand, or generate it:
acumen tasks                     # mine the package for real analyses -> tasks.yaml

# 3. Then run the loop:
acumen bench --no-skill          # the baseline arm
acumen draft                     # generate skills/v1 from the package source, or write by hand
acumen bench --skill v1          # benchmark the skill against the baseline
acumen improve                   # generate skills/v2 from v1's train results, or write by hand
acumen bench --skill v2
acumen report                    # aggregate every run into report.html

# 4. Once a version proves out, ship it into the package itself:
acumen ship --skill v2           # add a <dist>-install-skills console script (PR, or local edit)

acumen ship packages the chosen skill version into the target: the package gains a <dist>-install-skills command that installs the skill into the skills directory of whichever agent the user names — --agent {claude,codex,agents,claude-science}, or an explicit --dest — so the package's own users get the guidance with one command, wherever they run their agent. The same bundle installs verbatim into every framework.

acumen tasks, acumen draft, and acumen improve each accept --feedback "…" to steer the agent with context it can't infer — which functionality to skip when generating tasks, what a skill should emphasise or fix. The guidance is added to the prompt without overriding the train/test isolation, and for draft/improve it is recorded in the version's meta.json and shown in the report. (Don't paste held-out test answers into improve feedback — that would defeat the split.)

draft, improve, tasks, and ship each drive a long autonomous agent. Every run writes a live logs/acumen-<command>-<datetime>.jsonl (one event per step, flushed as it goes — so you can watch progress by reading the file) and a rendered .html transcript. Add --stream to mirror the conversation to the terminal, or --log-dir to change where the logs land.

Getting started

Please refer to the documentation, in particular, the API documentation.

Installation

You need to have Python 3.12 or newer installed on your system. If you don't have Python installed, we recommend installing uv.

And to install the acumen skill that ships with the package into your agent's skills directory, run acumen-install-skills --agent {claude,codex,agents,claude-science} (or --dest <dir> to choose the directory yourself):

acumen-install-skills --agent claude

Release notes

See the changelog.

Contact

For questions and help requests, you can reach out in the scverse discourse. If you found a bug, please use the issue tracker.

Citation

t.b.a