aspera-grader-kit
A small, readable pattern for grading the files that knowledge-work agents produce: Word memos, PowerPoint decks and Excel models.
Most agent evals grade the chat transcript. The client never sees the transcript. They open the file. A memo with its headings faked in bold, a deck whose chart is a pasted picture, or a model with hard-coded numbers inside its formulas all look correct. They are still wrong, and a grader that only reads text will reward them.
This kit grades the file itself:
output file ──► inspect ──► facts ──► checks ──► results ──► rubric ──► score + reasons
(docx/pptx/xlsx) (one thing each, (weights in plain JSON,
says why) next to the task)
Quick start
pip install aspera-grader-kit
To run the worked example and the tests, clone the repo and install it in editable mode:
pip install -e ".[test]"
python examples/board-memo/make_samples.py
python -m aspera_grader_kit examples/board-memo examples/board-memo/samples/board_memo_weak.docx
board-memo board_memo_weak.docx -> 26.4 / 100
Task requirements 16.4 / 60.0
Craft 10.0 / 40.0
[FAIL] R2 Required sections exist as real headings (0 of 2 items met)
- missed: Summary (no heading called Summary)
[FAIL] R4 Metrics table has every metric with its value and change (2 of 4 items met)
- missed: Operating expenses (no row)
- missed: EBITDA (no row)
[FAIL] C2 No faked headings (bold Normal text standing in for a heading) (fake headings: ['Summary', 'Risks'])
...
The strong sample scores 100. Every point lost comes with the check that took it and the evidence, so a disagreement with the grader is a conversation about one line, not about a number.
What's inside
| Part | What it does |
|---|---|
inspect/docx.py |
Body blocks in order (style, numbering, run formatting), tables (header rows, shading, margins), tracked changes, comments, content controls, fields, headers/footers, footnotes, hyperlinks |
inspect/pptx.py |
Slides in presentation order, titles, text and numbers, chart count and whether each keeps its workbook, tables, pictures, off-canvas shapes, body vs footer font floors, source lines, notes |
inspect/xlsx.py |
Formulas vs typed numbers, hard-coded constants inside formulas, error cells, charts and their ranges, validation, names, external links, and a LibreOffice recalculation to catch stale cached values |
checks.py |
@check decorator, Result.items() for per-item checks, runner that turns a crashing check into an error result instead of a crashed grade |
rubric.py |
Scores from a JSON rubric: categories, per-item counting, empty categories hand their points on, gates (zero, cap) |
examples/board-memo |
A complete task: brief, input CSV, answer key, rubric, nine checks and a strong/weak pair of outputs |
Design rules
- Grade the artifact, not the transcript. Inspect the file the way the client will open it.
- One check, one variable. A check that tests three things can't tell you which one failed.
- Every check explains itself. Evidence is part of the result, not a log line.
- Count requirements item by item. A task with ten required values should weigh them ten times, not once.
- The answer key never reaches the agent. Keep
key.jsonout of anything the agent can read. - Gates for the unforgivable. A missing or unopenable file scores zero, however good the reasoning was.
- Weights live in data. The rubric is JSON in the task folder, versioned with it, readable by a reviewer.
Writing your own task
Copy examples/board-memo, then edit:
task.md: the brief the agent sees.key.json: what a correct answer contains.rubric.json: categories, points, gates.checks.py: checks registered with@check("ID", "category", "description", registry="<task id>").
Where this comes from
This is the open, generic core of the harness we use at Motif Platforms to grade AI agents on real business deliverables. Our production tasks, answer keys, weights and judge prompts stay private. The reasoning behind the pattern is in the series Teaching machines to do real work:
- Reward hacking explained: how one bad grader teaches AI a thousand bad lessons
- How to design training exercises for AI agents
- Ten ways a grader lies in AI post-training
- The correct-looking file: why AI agents fail and what experts offer
Benchmark results from the full harness: motifplatforms/mosaic-benchmarks.
Need graders or exercises built for your own agents? dsaran@motifplatforms.com
License
MIT
Metadata
Release files for aspera-grader-kit 0.1.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| aspera_grader_kit-0.1.1.tar.gz | 21.7 kB | With provenance | Details for aspera_grader_kit-0.1.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| aspera_grader_kit-0.1.1-py3-none-any.whl | Python 3 | none | any | Details for aspera_grader_kit-0.1.1-py3-none-any.whl |
Total release size: 41.8 kB
Release files / aspera_grader_kit-0.1.1.tar.gz
| Download URL | aspera_grader_kit-0.1.1.tar.gz |
|---|---|
| Size | 21.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
4a2709f0a73f823d8f2d1ce11c89cc7f238dc20f93ba0a7f2e9890b307741438
|
|
BLAKE2b-256 checksum How to use checksums |
bc95c397ab1b32e14001338edd9ef3fa810f5646d85187296ab8fc1ac87d6cfb
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 14, 2026.
Transparency logRelease files / aspera_grader_kit-0.1.1-py3-none-any.whl
| Download URL | aspera_grader_kit-0.1.1-py3-none-any.whl |
|---|---|
| Size | 20.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
66e11001f0de5bfc98135e9fdbacf4008686371c5b901ce965772c6f3b9c75ec
|
|
BLAKE2b-256 checksum How to use checksums |
52cf12e45c7c79d19f0bd2b30a4e502dc18d62355acc6f5013cfc11350d19900
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 14, 2026.
Transparency logSecurity for aspera-grader-kit 0.1.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for aspera_grader_kit-0.1.1.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 14, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.