assessment-lens

Part of the lens family.

Signals-based assessment. Analysers generate signals about student submissions; this lens maps those signals to a rubric as observations, not grades. A lecturer reads the observations, weighs them, and assigns the mark. The AI narrates and cites; it never scores. A human stays in the loop.

assessment-lens is a lens (an assessment-aware product), not an -analyser. Analysers are assessment-agnostic signal generators; lenses sit above them. It consumes analysers (via bundle-analyser); it never generates signals. See ADR-0001.

Why "observations, not grades"

LLMs are inconsistent at the precise act of marking. So the lens keeps them off it. The deterministic signals (from analysers) are the anchor; an Observation maps a signal or two to a rubric Criterion with cited evidence, a short narration bound to that evidence, and a coverage (present / partial / absent) — which is "is the evidence there?", derived from thresholds, not a mark. There is deliberately no score/mark/weight field anywhere in the model.

Pipeline

Specification ──draft-rubric (LLM, reviewed)──▶ Rubric (criteria + mapping) + deliverables  [YAML]
Submissions root ──discover──▶ one Submission per subfolder
each Submission folder ──bundle-analyser──▶ Signals
Signals + Criteria + Deliverables ──alignment-check──▶ Observations
                                                         └─▶ cohort triage sheet + per-student reports
lecturer reads observations → assigns marks → writes feedback

assess is folder-dumb: one rubric + one folder of submission subfolders. Group/individual splits and mark-combining are handled outside (pre/post) or by running assess per folder.

Three commands

• assess — structured rubric + a submissions folder → observations (cohort sheet + per-student reports). Add --llm for evidence-bound narration on each observation (opt-in; needs the [llm] extra + a provider — ANTHROPIC_API_KEY by default, or a local Ollama via ASSESSMENT_LENS_PROVIDER=ollama).
• draft-rubric — free-form specification → a proposed structured rubric for the lecturer to review/edit before use. LLM-assisted; reads text/markdown directly and .pdf/.docx/.pptx via the [documents] extra.
• serve — opt-in HTTP API ([serve] extra) so a desktop shell / UI can drive the lens: same /health + /manifest contract as the analysers, role: lens. Binds to 127.0.0.1; set ASSESSMENT_LENS_AUTH_TOKEN to require a bearer token on the assessment routes.

Install

# from PyPI
pip install assessment-lens

# from source (family layout)
uv venv && source .venv/bin/activate
uv pip install -e ".[dev]"

# pull the whole analyser stack into the same env (for `assess` to run for real):
uv pip install -e ".[dev,analysers]"

# opt-in LLM features (narration + draft-rubric):
uv pip install -e ".[llm]"     # + export ANTHROPIC_API_KEY=...

assess shells out to the bundle-analyser CLI (it must be on PATH). The [analysers] extra installs it; or point assessment-lens at an environment where it is already installed.

Quick start

# 1. author a structured rubric (or start from the example)
cp examples/data-viz-rubric.yaml my-rubric.yaml

# 2. lay out submissions: one subfolder per student/group
#    submissions/alice/{report.pdf,demo.mp4}  submissions/bob/...

# 3. assess (add --llm for narrated observations)
assessment-lens assess my-rubric.yaml submissions/ -o out/
#    -> out/cohort-sheet.csv         (row per submission × criterion, sortable)
#    -> out/reports/<id>.md          (per-student observation sheet, no marks)

# or draft a rubric from a free-form spec, then review/edit it
assessment-lens draft-rubric assignment-brief.md -o my-rubric.yaml

Submissions are analysed sequentially (one bundle-analyser run each), so budget accordingly for large cohorts — e.g. 300 submissions × 30 s ≈ 2.5 h. A submission that fails to analyse is recorded as an error row in the cohort sheet and never aborts the run; re-run just that one with --only <id> once fixed.

Rubric schema (the central contract)

assignment: "Data-Viz Project"
component: individual                 # plain label; assess ignores it
expected_deliverables:
  - id: report
    description: "Written report (~2000 words)"
    accepts: [document]               # content kinds that satisfy it
  - id: demo
    description: "≤5-min recorded demo"
    accepts: [video]
rubric:
  - id: critical-thinking
    description: "Evidence of critical engagement / analysis"
    signals_of_interest: [conversation.critical_thinking, reflection.depth]  # OPTIONAL mapping

signals_of_interest is the signal→criterion mapping, and it's optional. Pinned → deterministic coverage. Blank → alignment-check selects signals at runtime and shows its choice (near-term).

Status

v0.5. Working today:

• ✅ Rubric load/validate; deliverable reconciliation; submission discovery
• ✅ assess orchestration + evidence-bound observations + threshold coverage; per-submission fault isolation (one failure never aborts the cohort)
• ✅ cohort sheet (CSV, spreadsheet-injection-safe) + per-student reports (Markdown)
• ✅ bundle-analyser integration — verified against the real CLI output schema
• ✅ LLM narration (assess --llm) — narrate-and-cite, bound to evidence; degrades to empty notes when the [llm] extra/key is missing
• ✅ multi-provider LLM — Anthropic by default; local Ollama or any OpenAI-compatible endpoint via ASSESSMENT_LENS_PROVIDER
• ✅ draft-rubric — proposes a rubric from a free-form spec (always review before use)
• ✅ cohort-relative distinctiveness ([distinctiveness] extra) — three comparison spaces; neutral, never a verdict
• ✅ serve ([serve] extra) — HTTP face for the desktop shell, optional bearer-token auth via ASSESSMENT_LENS_AUTH_TOKEN
• 📋 Runtime signal selection when signals_of_interest is blank — deferred

Development

ruff format . && ruff check . && pytest -v

License

MIT — see LICENSE.