SetSpec

Every versioned data contract that crosses an application boundary: benchmark results, capability evidence, event/error envelopes, prompt records.

Status: 0.2.0 — Phases 1–2 complete. The envelope, version negotiation and serialization core are implemented and tested; model.identity, machine.profile, benchmark.result, benchmark.run_summary, capability.evidence and benchmark.evidence_bundle are registered in SUPPORTED_SCHEMAS, but draft: Phase 4 may still reshape a field once FreeWeight has produced real results against these payloads. Which schemas are still provisional is readable at runtime from setspec.DRAFT_SCHEMAS, not only stated here — freezing one is a deletion from that set. Event and error envelopes arrive in Phase 3. See the development plan for what each phase adds.

Part of the Local AI Suite.

Install

pip install setspec

Quickstart

Write a document, then read it back:

from setspec import GeneratorInfo, SchemaVersion, dump_envelope, load_envelope

generator = GeneratorInfo(name="freeweight", version="1.0.0")
document = dump_envelope(
    {"tokens_per_second": 42.0},
    schema="benchmark.result",
    version=SchemaVersion(1, 0),
    generator=generator,
)

envelope = load_envelope(document, expect="benchmark.result", supported=[SchemaVersion(1, 0)])
assert envelope.payload == {"tokens_per_second": 42.0}

dump_envelope returns canonical JSON: byte-identical for equal input, on every platform and Python version, so a document can be hashed and diffed as well as read.

Payload types come in two halves generated from one definition — a strict Out for writers and a preserving In for readers (ADR-0009 rule 4):

from setspec import PayloadDefinition, payload_models
from setspec.serialization import MeasurementField


class ResultFields(PayloadDefinition):
    reading: MeasurementField
    unit: str


ResultOut, ResultIn = payload_models(ResultFields)

# A reader keeps what a newer writer added, so a re-export loses nothing.
received = ResultIn.model_validate({"reading": 1.5, "unit": "ms", "confidence": 0.87})
assert received.extras == {"confidence": 0.87}

A measurement this environment cannot provide is UNSUPPORTED, which serializes as the string "unsupported" — never null, never 0 (ADR-0016).

Phase 2's payload types live in their own versioned modules, not the top-level package, so that a future benchmark.result 2.0 can coexist with v1 rather than racing it for one name (ADR-0009 rule 6):

from setspec.capability.v1 import CapabilityEvidenceOut

evidence = CapabilityEvidenceOut.model_validate(
    {
        "model": {
            "provider_kind": "ollama",
            "provider_model_name": "qwen3.5:9b-q8_0",
            "artifact_digest": None,
            "identity_confidence": "name_only",
            "canonical_id": "ollama/qwen3.5:9b-q8_0@unknown",
            "observed_at": "2026-08-20T09:00:00.000Z",
        },
        "runtime_profile_hash": "a" * 16,
        "machine_fingerprint": "b" * 64,
        "capability_id": "coding.python",  # unenumerated specialization of the known root "coding"
        "score": 0.82,
        "confidence": 0.71,
        "sample_count": 40,
        "excluded_count": 2,
        "dispersion": 0.09,
        "measured_at": "2026-08-20T00:00:00.000Z",
        "computed_at": "2026-08-22T00:00:00.000Z",
        "policy_version": "1.0",
        "vocabulary_version": "1.0",
        "environment": {"provider_kind": "ollama", "provider_version": "0.32.13"},
    }
)
assert evidence.capability_id == "coding.python"

See docs/packages/setspec/spec.md §7 for the full public API and §20 for the acceptance criteria.

Documentation

Project documentation lives under docs/. Start with docs/README.md.

Development

python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pre-commit install
pytest -m "not live and not performance"

See CONTRIBUTING.md for the full workflow and SECURITY.md for how to report a vulnerability.

License

Apache-2.0 — see LICENSE.