7 projects
toolkit-rag-quality
Retrieval regression testing in CI: trec_eval-equal metrics, per-query diffs, significance tests and leakage checks.
toolkit-mmqa
Dataset pre-flight QA gate: train/eval and benchmark contamination, record and file dedup, image and audio checks, with a signable in-toto report.
toolkit-policy-test-bench
LLM safety and compliance evidence: policy tests, garak/promptfoo/PyRIT import, control mapping and CI gates.
toolkit-ml-provenance
AI-BOM in one command: CycloneDX 1.6 ML-BOMs, unsafe-pickle scanning, OpenSSF model signing and DSSE/Sigstore file signing for ML models.
toolkit-eval-harness
Eval regression gate with signed evidence: paired-bootstrap compare, importers for promptfoo, Inspect AI and DeepEval, in-toto report envelopes.
toolkit-cost-optimizer
LLM spend and routing analyzer: ingest gateway, trace and usage exports, price requests by token class, and find the cheapest routing that keeps quality.
toolkit-data-contracts
Contracts for LLM-shaped data: JSON Schema contracts, LLM presets, ODCS interop and drift checks.
