Skip to main content

aspera-grader-kit

tests License: MIT PyPI Python 3.10+

A small, readable pattern for grading the files that knowledge-work agents produce: Word memos, PowerPoint decks and Excel models.

Most agent evals grade the chat transcript. The client never sees the transcript. They open the file. A memo with its headings faked in bold, a deck whose chart is a pasted picture, or a model with hard-coded numbers inside its formulas all look correct. They are still wrong, and a grader that only reads text will reward them.

This kit grades the file itself:

output file ──► inspect ──► facts ──► checks ──► results ──► rubric ──► score + reasons
               (docx/pptx/xlsx)        (one thing each,       (weights in plain JSON,
                                        says why)              next to the task)

Quick start

pip install aspera-grader-kit

To run the worked example and the tests, clone the repo and install it in editable mode:

pip install -e ".[test]"
python examples/board-memo/make_samples.py
python -m aspera_grader_kit examples/board-memo examples/board-memo/samples/board_memo_weak.docx
board-memo  board_memo_weak.docx  ->  26.4 / 100
  Task requirements     16.4 /  60.0
  Craft                 10.0 /  40.0
  [FAIL] R2   Required sections exist as real headings  (0 of 2 items met)
           - missed: Summary (no heading called Summary)
  [FAIL] R4   Metrics table has every metric with its value and change  (2 of 4 items met)
           - missed: Operating expenses (no row)
           - missed: EBITDA (no row)
  [FAIL] C2   No faked headings (bold Normal text standing in for a heading)  (fake headings: ['Summary', 'Risks'])
  ...

The strong sample scores 100. Every point lost comes with the check that took it and the evidence, so a disagreement with the grader is a conversation about one line, not about a number.

What's inside

Part What it does
inspect/docx.py Body blocks in order (style, numbering, run formatting), tables (header rows, shading, margins), tracked changes, comments, content controls, fields, headers/footers, footnotes, hyperlinks
inspect/pptx.py Slides in presentation order, titles, text and numbers, chart count and whether each keeps its workbook, tables, pictures, off-canvas shapes, body vs footer font floors, source lines, notes
inspect/xlsx.py Formulas vs typed numbers, hard-coded constants inside formulas, error cells, charts and their ranges, validation, names, external links, and a LibreOffice recalculation to catch stale cached values
checks.py @check decorator, Result.items() for per-item checks, runner that turns a crashing check into an error result instead of a crashed grade
rubric.py Scores from a JSON rubric: categories, per-item counting, empty categories hand their points on, gates (zero, cap)
examples/board-memo A complete task: brief, input CSV, answer key, rubric, nine checks and a strong/weak pair of outputs

Design rules

  1. Grade the artifact, not the transcript. Inspect the file the way the client will open it.
  2. One check, one variable. A check that tests three things can't tell you which one failed.
  3. Every check explains itself. Evidence is part of the result, not a log line.
  4. Count requirements item by item. A task with ten required values should weigh them ten times, not once.
  5. The answer key never reaches the agent. Keep key.json out of anything the agent can read.
  6. Gates for the unforgivable. A missing or unopenable file scores zero, however good the reasoning was.
  7. Weights live in data. The rubric is JSON in the task folder, versioned with it, readable by a reviewer.

Writing your own task

Copy examples/board-memo, then edit:

  • task.md: the brief the agent sees.
  • key.json: what a correct answer contains.
  • rubric.json: categories, points, gates.
  • checks.py: checks registered with @check("ID", "category", "description", registry="<task id>").

Where this comes from

This is the open, generic core of the harness we use at Motif Platforms to grade AI agents on real business deliverables. Our production tasks, answer keys, weights and judge prompts stay private. The reasoning behind the pattern is in the series Teaching machines to do real work:

Benchmark results from the full harness: motifplatforms/mosaic-benchmarks.

Need graders or exercises built for your own agents? dsaran@motifplatforms.com

License

MIT

Metadata

Release files for aspera-grader-kit 0.1.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for aspera-grader-kit 0.1.1
File Size Uploaded Provenance Actions
aspera_grader_kit-0.1.1.tar.gz 21.7 kB With provenance Details for aspera_grader_kit-0.1.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for aspera-grader-kit 0.1.1
File Interpreter ABI Platform Actions
aspera_grader_kit-0.1.1-py3-none-any.whl Python 3 none any Details for aspera_grader_kit-0.1.1-py3-none-any.whl

Total release size: 41.8 kB

Release files / aspera_grader_kit-0.1.1.tar.gz

Download URL aspera_grader_kit-0.1.1.tar.gz
Size 21.7 kB
Tags Source
SHA-256 checksum
How to use checksums
4a2709f0a73f823d8f2d1ce11c89cc7f238dc20f93ba0a7f2e9890b307741438
BLAKE2b-256 checksum
How to use checksums
bc95c397ab1b32e14001338edd9ef3fa810f5646d85187296ab8fc1ac87d6cfb
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 14, 2026.

Transparency log

Release files / aspera_grader_kit-0.1.1-py3-none-any.whl

Download URL aspera_grader_kit-0.1.1-py3-none-any.whl
Size 20.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
66e11001f0de5bfc98135e9fdbacf4008686371c5b901ce965772c6f3b9c75ec
BLAKE2b-256 checksum
How to use checksums
52cf12e45c7c79d19f0bd2b30a4e502dc18d62355acc6f5013cfc11350d19900
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 14, 2026.

Transparency log

Security for aspera-grader-kit 0.1.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for aspera_grader_kit-0.1.1.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 14, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.1 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page