neurodock-evals

The versioned eval corpora and the air-gapped harness that runs ND prompts against them.

The corpus is the strategic asset that makes the translation layer honest. We prove that ND-aware prompts help neurodivergent users in real situations, and we catch regressions when prompts change. The harness gates prompt PRs in CI.

This package is v0.0.2 — the scaffold, the harness, and 6-10 hand-authored seed examples. The seeds are synthesised by to demonstrate the format — they are NOT real corporate messages. Real contributed corpora arrive over Phase 2 (target ~300 examples by month 6, per ).

What's here

packages/evals/
├── src/neurodock_evals/        # Harness, anonymiser, deduper, scorer
├── corpora/                    # Versioned YAML eval examples by slice
├── schemas/                    # JSON Schemas for examples + annotations
└── tests/                      # Tests for the harness itself

Quick start

Run the harness against the seed corpora:

uv run python -m neurodock_evals.harness --corpus translation/incoming \
    --tool translate_incoming

Run all four translation slices:

uv run python -m neurodock_evals.harness --ci

Anonymise a contribution before opening a PR:

uv run python -m neurodock_evals.anonymise path/to/example.yaml

Air-gapped by design

The harness never calls an LLM. It exercises each tool's deterministic baseline (the heuristic layer the translation server returns even before any LLM refinement) and scores the baseline against the human-rated expected block. Any LLM-side eval is a separate concern that the maintainer reviews under a different policy.

Privacy

• The harness never logs example contents to stdout or to anywhere outside .eval-reports/.
• Reports contain example IDs and scores only — never verbatim text.
• The contribution pipeline (anonymise.py) is a safety net, NOT a substitute for contributor judgement. See CONTRIBUTING.md.
• All corpora are licensed AGPL-3.0-or-later.

Glossary

Status

• v0.0.2: scaffold + harness + synthesised seed examples across four tool slices
• v0.0.3 (current): R6 per-neurotype eval slices — an optional neurotypes tag on examples (canonical profile enum), per-neurotype scoring + reporting, and a translation/neurotype seed slice (14 synthesised examples across seven neurotypes). Cross-cutting, additive, back-compatible.
• v0.1.0 (planned): first contributed corpus + HuggingFace publication pipeline under the neurodock org

See CHANGELOG.md for detail.

Per-neurotype slices (R6)

Examples MAY carry an optional neurotypes array (the canonical profile enum: adhd, asd, audhd, ocd, dyslexia, dyspraxia, tourette, other). The harness aggregates a score per neurotype — alongside the per-tool slices — so a prompt change can be measured against, e.g., the dyslexia slice. The tag is additive and optional: an untagged example scores exactly as before and is absent from every per-neurotype aggregation. Thresholds stay permissive; R6 ships the per-type measurement, not a hard per-type gate. See corpora/translation/neurotype/README.md.