crust-score

A reference implementation of CRUS-T score: a normalized, programmatically computable runtime reliability score for agentic AI, from

Satyaki Ghosh Dastidar, Proportionate Governance in Practice: An IDEA–CRUS-T Framework for Risk-Tiered, Runtime-Governed AI Systems in Financial Services. Preprint, 2026.

CRUS-T scores an AI capability or execution trajectory on four dimensions — Consistency, Robustness, Uncertainty awareness, and Safety — each built from three standard, auditable sub-metrics, aggregated with a collapse-sensitive weighted geometric mean, and gated on a conservative statistical bound rather than a point estimate. Safety additionally carries a discontinuous hard gate: a single critical policy violation zeroes the governed safety score regardless of the continuous score, so it can never be averaged away.

This library implements the score itself (Section 5 of the paper) plus a thin tier-gating layer on top of it (Section 5.6–5.7) that decides continue/escalate/contain against a threshold vector you supply. It does not implement the paper's separate five-dimension risk-tiering method (Section 4 — Authority/Impact/Agency/Exposure/Recoverability → control tier); that assigns a design-time control tier, whereas this library is about the runtime reliability score that verifies a tier's assumptions continue to hold in production.

Status

Early (0.1.0). The math is unit-tested against the paper's equations and against the worked example in Appendix C. The weights, tolerances, and thresholds you configure are governance parameters, not statistical defaults — see Governance parameters below before using this for anything with real consequence.

Install

pip install -e .          # from a local checkout
pip install -e ".[dev]"   # + pytest, for running the test suite

Requires Python 3.10+. No required runtime dependencies.

How the code maps to the paper

Every public function's docstring cites the specific equation it implements, so you can go from a number in the paper straight to the code that produces it (and back).

What this library does not do for you

Several CRUS-T sub-metrics depend on a learned component the paper deliberately keeps out of the governed math: an entailment model for groundedness (S1), an embedding or entailment model for semantic similarity (C1/C2), a task-quality function Q(.) for Robustness. This library takes the outputs of those components (an entailment boolean, a similarity score, a quality score) as plain function arguments — it does not call any model itself. Section 5.5 of the paper is explicit about why: "a probabilistic verifier operating in the same semantic space as the system it polices is architecturally blind to fluent-but-wrong outputs," so whatever model you plug in needs to be validated and governed separately, on its own precision/recall for the violation class it's detecting.

Quickstart

from crus_t import (
    compute_consistency, compute_robustness, compute_uncertainty,
    governed_safety, PolicyObservation, safety_to_dimension_score,
    ReliabilityVector, composite,
    ThresholdVector, evaluate_gate,
    make_run_score, aggregate_run_scores,
)

# 1. Compute each dimension from its three sub-metrics (already observed
#    from your evaluation harness / production telemetry).
c = compute_consistency(c1=0.94, c2=0.88, c3=0.97)
r = compute_robustness(r1=0.85, r2=0.96, r3=0.80)
u = compute_uncertainty(u1=0.89, u2=0.81, u3=0.85)

safety_result = governed_safety(
    entailed_claims=98, total_claims=100,
    policy_observations=[PolicyObservation("no_stale_citation", severity=1.0, violation_count=0)],
    n_trajectories=100,
    authorized_calls=10, total_external_calls=10,
    critical_violation_observed=False,
    tau_safety=0.10,
)
s = safety_to_dimension_score(safety_result)

vector = ReliabilityVector(c, r, u, s)
print(vector.as_dict())          # {"C": 0.93, "R": 0.87, "U": 0.85, "S": ...}
print(composite(vector))         # single reporting number, Eq. (22)

# 2. Package it as a run and gate it against a tier's thresholds. Threshold
#    values are always your own governance input -- this library has no
#    built-in per-tier table.
thresholds = ThresholdVector(consistency=0.80, robustness=0.75, uncertainty=0.78, safety=0.90)
gate = evaluate_gate(vector.as_dict(), thresholds)
run = make_run_score(run_id="run-001", capability_id="drafting", vector=vector, gate=gate)
print(gate.zone)                 # RuntimeZone.CONTINUE / ESCALATE / CONTAIN

# 3. Later, roll many runs up into a batch view.
agg = aggregate_run_scores([run, ...])
print(agg.escalation_rate, agg.containment_rate, agg.axis_summaries["safety"].mean)

For a regulatory-grade decision, gate on a bootstrap lower bound instead of the point estimate (Eq. 24):

from crus_t import bootstrap_lower_bound

# `statistic` recomputes a dimension score from a resampled item list --
# wire this to your own evaluation-set records.
result = bootstrap_lower_bound(evaluation_items, statistic=my_dimension_statistic, seed=42)
gate = evaluate_gate({"C": result.lower_bound, ...}, thresholds)

See examples/worked_example.py for a runnable version of Appendix C's Drafting-capability example, and examples/quickstart.py for the snippet above.

Governance parameters

Every tolerance (tau), every sub-metric weight, every threshold vector, and the bootstrap confidence level are, per Section 5.1 of the paper, governance parameters: "set by the second line of defense from pooled validation evidence, versioned, and subject to challenge — not free parameters tuned to produce a desired outcome." crus_t.config.GovernanceConfig gives them one explicit, serializable home; ThresholdVector in crus_t.gating is a plain, explicit input with no built-in defaults. This library will not stop you from tuning a threshold until a report looks good — that governance discipline lives with your process, not the code.

Development

pip install -e ".[dev]"
pytest                 # run the test suite
pytest --cov=crus_t    # with coverage

License

MIT. See LICENSE.