assert-ai 0.1.0

YAML-driven safety evaluation pipeline with LiteLLM-backed stages

ASSERT.

Adaptive Spec-driven Scoring for Evaluation and Regression Testing
Local-first. Framework-agnostic. Trace-aware.

🚀 Get started | 🔌 View supported targets | 📘 CLI Reference | 🧪 Examples

Why ASSERT?

Most AI systems start with a specification: product requirements, policies, system prompts, or launch criteria describing what the system should and should not do.

But evaluation often starts elsewhere: generic scorers, predefined benchmarks, or manual test cases that drift from the original intent.

ASSERT closes that gap. It turns your specified behaviors in natural language into structured, executable evaluations that can be reviewed, run, scored, and improved over time.

From the natural language specification, the ASSERT pipeline derives behavior categories, generates single-turn and multi-turn test cases, inferences them against your target, and uses an LLM judge to score each conversation against your policies.

What you get with ASSERT

• Spec-driven coverage - test cases are generated from your product requirements and context, not a generic benchmark. You specify the behaviors that you want to test for
• Test any model endpoint via integrations with LiteLLM, supporting 100+ model endpoints from platform providers such as Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sagemaker, HuggingFace, VLLM, NVIDIA NIM.
• Test any agent or multi-agent system via integrations with OpenInference. Evaluate a LangGraph agent, a CrewAI / OpenAI Agents SDK / DSPy / LlamaIndex / AutoGen system, custom multi-agent orchestration, a Python callable, or a hosted model — without rewriting the evaluation orchestration pipeline.
• Agent trace-grounded judgment - the recommended integration captures OpenTelemetry spans (Phoenix/OpenInference auto-instruments 33+ frameworks in two lines, or you can emit your own with the OTel SDK) so the judge can cite tool calls, routing, model calls, and latency as evidence — not just the final response.
• Portable artifacts - every stage writes JSON/JSONL files locally for inspection, CI, and sharing.
• Bundled local viewer - browse runs side-by-side, pin a baseline, drill into per-behavior dimension breakdowns, and read judge justifications cited against the captured traces.

Get started

Quick install

pip install -e ".[otel,langgraph]"       # install
cp .env.example .env                     # add your provider key
assert-ai run --config examples/travel_planner_langgraph/eval_config.yaml

🌐 Project website ↗	📝 Technical blog ↗	🚀 Quickstart guide ↗	📚 Documentation ↗
Learn about ASSERT	Read the Command Line post	Follow the full walkthrough	Browse concepts and guides

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos is subject to those third party's policies.

Telemetry

This project does not collect or send telemetry to Microsoft by default. Runs write local artifacts under artifacts/results/, and optional OpenTelemetry trace capture is controlled by your configuration and local collector setup, such as Phoenix.

If you configure a target, judge, trace collector, or model provider to send data to an external service, the prompts, responses, traces, metadata, and other evaluation artifacts sent to that service are governed by that service's terms and your configuration.

Disclaimer: Risks and limitations of ASSERT

See the full section in the Concept Doc.