Skip to main content

aspera-grader-kit

tests License: MIT Python 3.10+

A small, readable pattern for grading the files that knowledge-work agents produce: Word memos, PowerPoint decks and Excel models.

Most agent evals grade the chat transcript. The client never sees the transcript. They open the file. A memo with its headings faked in bold, a deck whose chart is a pasted picture, or a model with hard-coded numbers inside its formulas all look correct. They are still wrong, and a grader that only reads text will reward them.

This kit grades the file itself:

output file ──► inspect ──► facts ──► checks ──► results ──► rubric ──► score + reasons
               (docx/pptx/xlsx)        (one thing each,       (weights in plain JSON,
                                        says why)              next to the task)

Quick start

pip install -e ".[test]"
python examples/board-memo/make_samples.py
python -m aspera_grader_kit examples/board-memo examples/board-memo/samples/board_memo_weak.docx
board-memo  board_memo_weak.docx  ->  26.4 / 100
  Task requirements     16.4 /  60.0
  Craft                 10.0 /  40.0
  [FAIL] R2   Required sections exist as real headings  (0 of 2 items met)
           - missed: Summary (no heading called Summary)
  [FAIL] R4   Metrics table has every metric with its value and change  (2 of 4 items met)
           - missed: Operating expenses (no row)
           - missed: EBITDA (no row)
  [FAIL] C2   No faked headings (bold Normal text standing in for a heading)  (fake headings: ['Summary', 'Risks'])
  ...

The strong sample scores 100. Every point lost comes with the check that took it and the evidence, so a disagreement with the grader is a conversation about one line, not about a number.

What's inside

Part What it does
inspect/docx.py Body blocks in order (style, numbering, run formatting), tables (header rows, shading, margins), tracked changes, comments, content controls, fields, headers/footers, footnotes, hyperlinks
inspect/pptx.py Slides in presentation order, titles, text and numbers, chart count and whether each keeps its workbook, tables, pictures, off-canvas shapes, body vs footer font floors, source lines, notes
inspect/xlsx.py Formulas vs typed numbers, hard-coded constants inside formulas, error cells, charts and their ranges, validation, names, external links, and a LibreOffice recalculation to catch stale cached values
checks.py @check decorator, Result.items() for per-item checks, runner that turns a crashing check into an error result instead of a crashed grade
rubric.py Scores from a JSON rubric: categories, per-item counting, empty categories hand their points on, gates (zero, cap)
examples/board-memo A complete task: brief, input CSV, answer key, rubric, nine checks and a strong/weak pair of outputs

Design rules

  1. Grade the artifact, not the transcript. Inspect the file the way the client will open it.
  2. One check, one variable. A check that tests three things can't tell you which one failed.
  3. Every check explains itself. Evidence is part of the result, not a log line.
  4. Count requirements item by item. A task with ten required values should weigh them ten times, not once.
  5. The answer key never reaches the agent. Keep key.json out of anything the agent can read.
  6. Gates for the unforgivable. A missing or unopenable file scores zero, however good the reasoning was.
  7. Weights live in data. The rubric is JSON in the task folder, versioned with it, readable by a reviewer.

Writing your own task

Copy examples/board-memo, then edit:

  • task.md: the brief the agent sees.
  • key.json: what a correct answer contains.
  • rubric.json: categories, points, gates.
  • checks.py: checks registered with @check("ID", "category", "description", registry="<task id>").

Where this comes from

This is the open, generic core of the harness we use at Motif Platforms to grade AI agents on real business deliverables. Our production tasks, answer keys, weights and judge prompts stay private. The reasoning behind the pattern is in the series Teaching machines to do real work:

Benchmark results from the full harness: motifplatforms/mosaic-benchmarks.

Need graders or exercises built for your own agents? dsaran@motifplatforms.com

License

MIT

Metadata

Release files for aspera-grader-kit 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for aspera-grader-kit 0.1.0
File Size Uploaded Provenance Actions
aspera_grader_kit-0.1.0.tar.gz 21.6 kB With provenance Details for aspera_grader_kit-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for aspera-grader-kit 0.1.0
File Interpreter ABI Platform Actions
aspera_grader_kit-0.1.0-py3-none-any.whl Python 3 none any Details for aspera_grader_kit-0.1.0-py3-none-any.whl

Total release size: 41.6 kB

Release files / aspera_grader_kit-0.1.0.tar.gz

Download URL aspera_grader_kit-0.1.0.tar.gz
Size 21.6 kB
Tags Source
SHA-256 checksum
How to use checksums
1502be2e72697ced8ada347ff6076b4a2ac51d87accfde6b45224f83335ad54c
BLAKE2b-256 checksum
How to use checksums
82a9a51bdf3568a202af0f2ca84148c3d7ba5bf0d5e5449302a171322131782e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 14, 2026.

Transparency log

Release files / aspera_grader_kit-0.1.0-py3-none-any.whl

Download URL aspera_grader_kit-0.1.0-py3-none-any.whl
Size 20.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
427543aa5a02e6bb02089cbd6a3a9e5f8ac7fbba48d8a82853422bd4c3297d6e
BLAKE2b-256 checksum
How to use checksums
ad5bd4e297a301984193bfb7204740cd4dbe2c8540c3f0dc97e7f7209cf6dcf7
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 14, 2026.

Transparency log

Security for aspera-grader-kit 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for aspera_grader_kit-0.1.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 14, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page