decision-scope

What a claim declares about its own boundary, and whether that declaration holds up.

uv tool install decision-scope

decision-scope check manifests/          # before a run: do its declarations contradict?
decision-scope score audits/             # after: what does the record settle?

An evaluation result is produced under a particular model version, task distribution, operating environment, and measurement procedure. It is then used elsewhere — in a deployment decision, a safety report, a regulatory filing. That use assumes the conditions under which the result was obtained still describe the setting where it is applied.

This separates three operations over those conditions. A declaration states the condition. A check compares it against the artifacts that produced the evidence. A verification measures whether it holds where the result is used.

Before a run

A run is launched with a boundary, a configuration, and a task. All three are values the operator holds at launch, so a contradiction between them is decidable before execution.

decision-scope check examples/manifests/

  DL-v1           containment    FAIL  the task requires reaching 10.20.5.7, which the
                                       declaration places out of scope via 10.20.0.0/16;
                                       task and declaration cannot both be satisfied
  DL-v2           containment    pass  named as an explicit carve-out
  safety-eval     configuration  FAIL  results are reported under sha256:9f2c1ab4e7d05836
                                       but the run was provisioned as sha256:41bd7e0c9a3f5528

Exit status is 1 when any predicate is unsatisfiable, so it gates a launch.

A manifest is YAML, written from values the harness already holds:

run_id: DL-v1
boundary:
  in_scope: ["10.10.1.0/24", "10.10.2.0/24", "10.10.3.0/24"]
  out_of_scope: ["10.20.0.0/16"]
  carve_out: []
configuration:
  declared: sha256:9f2c1ab4e7d05836
  actual: sha256:9f2c1ab4e7d05836
task:
  required_hosts: ["10.20.5.7"]

Every field is optional. A predicate with nothing to compare reports itself as unchecked rather than passing, so an empty manifest cannot read as a clean run.

After the fact

A scorecard records one published claim at every cell of the lattice.

decision-scope score examples/scorecards/

  Prep.    [documentation]  ok
  METR     [documentation]  ok
  Polyg.   [documentation]  ok
  Mythos   [documentation]  ok
  AISI     [documentation]  ok

  5 scorecard(s), 0 with problems

  Stage          Cells     D    PD    DA     F     U   N/A   N/R
  --------------------------------------------------------------
  declare           30    20     5           5
  check             30           5     1    12     5     2     5
  verify            30                 1     2    21     1     5

  declare satisfied: 20 of 30
  check satisfied: 0 of 30
  verify satisfied: 0 of 30

The operations are ordered, so a dimension whose declaration failed has unreachable rather than failed cells below it. score enforces that in both directions: a failed declaration requires N/R below it, and N/R is refused wherever the declaration did not fail.

decision-scope table emits the same matrix as a LaTeX tabular, with the satisfied counts recomputed from the cells rather than written by hand.

The lattice

decision-scope lattice prints this.

As a library

from pathlib import Path
from decision_scope import load_scorecard, audit, totals

cards = [load_scorecard(p) for p in Path("audits").glob("*.yaml")]
problems = [p for card in cards for p in audit(card)]
print(totals(cards)["check"].satisfied)

Files become models at the edge, so a malformed card raises CardFileError naming the file and the field rather than a KeyError from somewhere the caller cannot see. --json on check and score emits the same values for a harness that does not want to import.

What it will not do

It cannot detect a declaration that is well-formed and false. A manifest records what the operator declared and provisioned, which makes it accurate about the evaluation and silent about the world the evaluation ran in.

Claude Code plugin

/plugin marketplace add elliottower/decision-scope
/plugin install decision-scope@decision-scope

Development

uv run --with pydantic --with pyyaml --with pytest python -m pytest tests/

License

MIT