Skip to main content

underwrite

underwrite turns local evaluation artifacts into a common observation, computes measurements under an explicit policy, and classifies change candidates against declared evidence. It keeps missing evidence visible: not_measured, indeterminate, and abstained are distinct outcomes.

It is a Python CLI and library. Artifact inspection checks bytes and declared hashes; release checks examine declared linkage and record presence. These checks do not establish independent provenance or grant deployment permission.

Install from source

Requires Python 3.12 or later and uv.

git clone https://github.com/ntts9990/underwrite.git
cd underwrite
uv sync --frozen --all-groups
uv run --frozen --no-sync underwrite --help

The repository includes the underwrite-core workspace package. No adjacent checkout is needed to install or use the example workflows.

PyPI release preparation

The functional 0.0.1 release is prepared for underwrite-core and underwrite; publication is not yet confirmed. The existing underwrite 0.0.0 entry is a reserved placeholder. Use the source installation above until both 0.0.1 releases are available. After publication, the planned CLI installation is:

uv tool install 'underwrite==0.0.1'
underwrite --help

Maintainers: create the GitHub environment pypi, restrict it to main, and configure a PyPI GitHub Trusted Publisher on each project with owner ntts9990, repository underwrite, workflow filename publish.yml, and environment pypi. For the new underwrite-core project, register a pending publisher under that project name; add the publisher to the existing underwrite project in its Publishing settings. No stored PyPI API token is required.

Once configured, run Publish PyPI manually from the main branch in GitHub Actions. It tests both packages, builds and validates the exact distributions, and installs the wheels in a clean environment before publishing core and then the application. Only the separate publish job receives OIDC permission. Retry with the same artifacts after a partial upload; existing registry files must match exactly. A changed artifact requires a new version. Confirm both PyPI project releases and a fresh indexed installation before describing the release as available.

The workflow follows the official uv publishing guide and PyPI Trusted Publisher guide.

Normalize an artifact

Choose a format and exact supported version explicitly:

Producer --format --version
DeepEval deepeval.test-run 4.1.1
Inspect AI inspect.eval-log 0.3.263
Langfuse langfuse.observations-v2 or langfuse.scores 4.35.0
OpenInference openinference.traces 0.1.38
OTLP JSON otlp-json.traces 0.160.0
promptfoo promptfoo.eval-output 0.123.0
Native evidence bundle underwrite.evidence-bundle v1

These are bounded payload profiles, not compatibility claims for every export from those producers. project is an explicit alias of ingest.

mkdir -p output
uv run --frozen --no-sync underwrite ingest \
  --format underwrite.evidence-bundle --version v1 \
  --file fixtures/examples/measurement/bundle.json --json \
  > output/observation.json
uv run --frozen --no-sync underwrite measure \
  --observation output/observation.json \
  --policy fixtures/examples/acceptance/policy-quality.json --json \
  > output/read-quality.json
uv run --frozen --no-sync underwrite measure \
  --observation output/observation.json \
  --policy fixtures/examples/acceptance/policy-latency.json --json \
  > output/read-latency.json

The native bundle declares schema_version: underwrite.evidence-bundle.v1 and carries readings, source claims, and availability records. An observation keeps those source claims separate from the measurement. Measurement requires a complete policy identifying inputs, subject, calibration, quorum, and statistical thresholds. A successful command exit means the read was produced; inspect the read's state before drawing a conclusion.

--http-body-file FILE reads an already captured HTTP body from a local file; it does not fetch a URL. Optional --source-ref and --source-locator describe the supplied source. When no source reference is given, byte identity supplies a content hash, not an independently verified origin. See command --help for input size and depth limits.

Malformed native input produces a typed error on stderr and no observation on stdout. Try a bundle with an empty run identifier:

mkdir -p output
printf '%s\n' '{"schema_version":"underwrite.evidence-bundle.v1","run_id":"","sources":[],"readings":[],"availability":[]}' \
  > output/malformed-bundle.json
if uv run --frozen --no-sync underwrite ingest \
  --format underwrite.evidence-bundle --version v1 \
  --file output/malformed-bundle.json --json \
  > output/malformed.stdout 2> output/malformed.stderr; then
  ingest_exit=0
else
  ingest_exit=$?
fi
printf 'exit: %s\n' "$ingest_exit"
wc -c < output/malformed.stdout
cat output/malformed.stderr

This reports exit 1, stdout size 0, and this stderr JSON:

{"schema":"instrument_error.v1","command":"ingest","code":"INVALID_INPUT","reason":"MALFORMED_EVIDENCE_BUNDLE_PAYLOAD","exit_code":1}

Classify a candidate

accept binds a candidate to declared claims and policy-matched reads. It reports whether the supplied evidence supports the requested acceptance basis and retains uncertainty or missing measurements in the decision.

uv run --frozen --no-sync underwrite accept \
  --candidate fixtures/examples/acceptance/candidate.json \
  --claims fixtures/examples/acceptance/claims.json \
  --read output/read-quality.json \
  --read output/read-latency.json --json

Repeat --read for each required claim. Exit 0 means a classification was produced, not approval. This example produces screened, requires human review, and keeps merge_authorized false. The inputs are synthetic local examples; the reads above are computed from the supplied bundle and policies.

Other checks

  • inspect CASE.json --json: inspect one local artifact manifest and its projection.
  • check RELEASE.json --json: check a release manifest's declared evidence linkage.
  • absence --root DIR --glob '*.json' --min 1 --json: require a nonempty population.
  • boundary scan DIR --json: scan JSON and JSONL strings for configured verdict vocabulary.
  • drift check --pins-root PINS --siblings-root SOURCES --json: compare a user-supplied pin registry against local source checkouts.
  • pins refresh --repo NAME --pins-root PINS --siblings-root SOURCES: refresh a configured repository's pins from local bytes.

Pin commands are generic utilities; no preconfigured source registry is included. Allowlist and exemption evaluation requires an explicit --now when applicable. Malformed inputs and missing paths remain configuration errors rather than passes. Use each command's --help for its options and exit behavior. JSON contracts live in contracts/; each command family has its own output and error schema.

Development and license

See CONTRIBUTING.md for the checks and contribution process, SECURITY.md for private vulnerability reporting, and AGENTS.md for technical boundaries. The core package uses only the Python standard library; application adapters own local I/O and schema validation.

underwrite is licensed under Apache-2.0. LICENSING.md records the adapted policy templates' CC licenses.

Metadata

Release files for underwrite 0.0.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for underwrite 0.0.1
File Size Uploaded
underwrite-0.0.1.tar.gz 137.4 kB Details

Built distribution (wheel)

Table of built distributions (wheels) for underwrite 0.0.1
File Interpreter ABI Platform
underwrite-0.0.1-py3-none-any.whl Python 3 none any Details

Total release size: 307.7 kB

Release files / underwrite-0.0.1.tar.gz

Download URL underwrite-0.0.1.tar.gz
Size 137.4 kB
Tags Source
SHA-256 checksum
How to use checksums
85cdc103d1bb2786eff2f204f0decb6fd135852fc8d9acdaabf205fea1d3c294
BLAKE2b-256 checksum
How to use checksums
d1bf342cb80f00ea4a8cb20f07a3b1d76464ec6f2ede3ab5a2e07736570d6c49
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.11.29 {"installer":{"name":"uv","version":"0.11.29","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Release files / underwrite-0.0.1-py3-none-any.whl

Download URL underwrite-0.0.1-py3-none-any.whl
Size 170.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums
b448244313eb8f0606048306a1a6f729215faa1a9c5b451964588f5a5ae92051
BLAKE2b-256 checksum
How to use checksums
62502b2e165340c50697c5ea48ac65ffb5cb75d40f1d35c6586dc779b6b2179b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.11.29 {"installer":{"name":"uv","version":"0.11.29","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Release history Release notifications | RSS feed

This release

0.0.1 This release

2 release files

0.0.0

2 release files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page