truereward-harwell

The recorded-state reader for truereward, over the Harwell engine.

It grades a change to a codebase that has no unit test suite. A case records what a job leaves behind when the code is right: the data sets, the spool files and the tables. Grading runs the same job against the changed codebase and compares what it left with that approved baseline.

• A case is data. A directory with case.json, a start/ snapshot and an expected/ snapshot, validated against the JSON Schema shipped in the package. Every field a case reads is resolved to bytes when the case is recorded, so grading never opens the candidate's source to decide what is compared.
• grade_case(case, codebase) runs the case twice in a contained run, compares the first end state with the baseline under the case's policy, and answers with a truereward.CaseResult. The two runs are compared with each other under the author's own mask, which answers the determinism check.
• record.record(...) records a case: run the job, keep what it left as the approved baseline, freeze every field map.
• selftest.run(...) asks whether a case set would catch a bug: the reference change must score 1.0, the unchanged codebase 0.0, and every seeded defect less than 1.0.

A task's test file:

from functools import partial
from pathlib import Path

import pytest
from truereward_harwell import cases, grade_case

CASES = Path(__file__).parent / "cases"


@pytest.mark.parametrize("case", cases(CASES), ids=lambda c: c.id)
def test_case(case, grade):
    grade(case, partial(grade_case, codebase="/grade"))

Run it with pytest -p truereward_harwell, which brings the truereward plugin with it.

Licensed under Apache-2.0.