truereward

Turn the cases of a graded run into one reward, or into the reason there is none.

A grader looks at some kind of evidence: a recorded end state, a test report, a list of criteria, a measured figure. Whatever it looked at, it answers with one record per case, CaseResult. This package is everything after that record, and it is the same for every kind of evidence.

• The fold. fold(rows) applies gated-binary/1: the reward is 1.0 when every pass-to-pass case holds and every fail-to-pass case passes, and 0.0 otherwise. fix_rate and guards_hold are reported beside it.

• Three outcomes. A case passed, failed, or could not be graded. The third carries a code from a closed list, each with who failed: the run, or the grader. An ungradable run has no reward. It is never scored 0.

• The document. grade_document builds grade.json; write publishes it with ctrf.json, reward.json and reward.txt into a verifier's logs directory, each file whole or not at all. An ungradable run publishes no reward file.

• The pytest plugin. One test per case, folded once at the end of the session, in one process or under pytest-xdist:

  @pytest.mark.parametrize("case", cases(CASES), ids=lambda c: c.id)
  def test_case(case, grade):
      grade(case, grade_case)
  

  Run it with pytest -p truereward.plugin. Install the pytest extra for it: pip install "truereward[pytest]". Folding and writing need the standard library alone.

A reader for one kind of evidence is a separate distribution. It may declare refusal codes of its own with truereward.codes.declare.

Licensed under Apache-2.0.