aicert 0.1.0

CI for LLM JSON outputs - Validate, test, and measure LLM outputs

aicert

Reliability tooling for structured LLM outputs.

Measure, validate, and enforce JSON stability in CI.

---

Why aicert?

LLMs are probabilistic systems.

If your application depends on structured JSON generated by an LLM, that output becomes an implicit contract within your system. Model updates, prompt changes, or configuration tweaks can cause that contract to drift — often without immediate visibility.

Traditional tests typically validate a single run. They do not measure variability across repeated executions.

Aicert treats LLM output like any other production dependency:

• Validate against a JSON Schema
• Measure stability across repeated runs
• Track latency and variability
• Enforce reliability thresholds in CI

Instead of assuming structured outputs remain stable, you can verify that they do.

---

What It Enables

Aicert helps you:

• Detect schema breakage before deployment
• Quantify output variability across runs
• Catch regressions caused by model or prompt changes
• Enforce reliability standards automatically in CI
• Treat prompts and structured outputs as testable infrastructure

It converts non-deterministic behavior into measurable signals.

---

Who It’s For

Aicert is designed for teams shipping structured LLM outputs into production systems:

• Backend APIs powered by LLMs
• Extraction and classification pipelines
• Decision-support systems
• Any workflow where JSON output drives downstream logic

If structured LLM output is part of your system contract, Aicert provides guardrails.

---

Installation

pip install aicert

For development:

pip install -e .

---

What It Measures

• Compliance — % of outputs matching JSON Schema
• Stability — % of identical outputs across repeated runs
• Latency — P50 / P95 response times
• Similarity — Structural or semantic similarity
• CI Gating — Threshold-based pass/fail enforcement

---

Basic Workflow

1. Create aicert.yaml

project: my-project

providers:
  - id: openai-gpt4
    provider: openai
    model: gpt-4
    temperature: 0.1

prompt_file: prompt.txt
cases_file: cases.jsonl
schema_file: schema.json

runs: 50
concurrency: 10
timeout_s: 30

validation:
  extract_json: true
  allow_extra_keys: false

thresholds:
  min_stability: 85
  min_compliance: 95
  p95_latency_ms: 5000

ci:
  runs: 10
  save_on_fail: true

---

2. Run Stability Tests

aicert stability aicert.yaml

---

3. Enforce in CI

aicert ci aicert.yaml

Non-zero exit codes indicate threshold failures.

---

Commands

aicert init
aicert doctor aicert.yaml
aicert run aicert.yaml
aicert stability aicert.yaml
aicert compare aicert.yaml
aicert ci aicert.yaml
aicert diff <run_a> <run_b>
aicert report <run_dir>

---

Aicert Pro

Aicert Pro turns measurement into enforcement.

The core package tells you how your LLM behaves. Pro defines what behavior is acceptable — and blocks regressions automatically.

When structured outputs drive production systems, “informational metrics” are not enough. You need a reliability boundary.

Aicert Pro adds:

• Baseline locking — capture a known-good state
• Regression enforcement — fail CI when stability or compliance degrades
• Schema and prompt drift detection — detect structural changes immediately
• Cost regression limits — prevent silent cost creep
• Policy-backed CI gating — enforce standards across commits

Pro is designed for teams that treat LLM output as production infrastructure.

---

What Changes With Pro?

Without Pro:

You measure stability. You review results manually. Regressions can slip through if thresholds are ignored.

With Pro:

Known-good behavior is locked. Deviations are compared automatically. CI fails when output contracts drift. Reliability becomes enforceable policy. Pro moves you from observation to control.

Installing Aicert Pro

After purchasing a subscription:

pip install aicert-pro

Set your license key:

export AICERT_LICENSE="your_signed_license_key"

Verify:

aicert-pro license verify

Use baseline enforcement:

aicert-pro baseline save aicert.yaml
aicert-pro baseline check aicert.yaml

---

Licensing

Aicert Pro uses signed offline license keys.

• No SaaS dependency
• No account login
• CI-friendly
• Time-limited per subscription

---

Pricing

• $29 / month
• $290 / year (2 months free)

Purchase: https://mfifth.github.io/aicert/

Questions: [e-mail]

---

GitHub Actions Example

- run: pip install aicert
- run: aicert ci aicert.yaml
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

Pro enforcement in CI:

- run: pip install aicert-pro
- run: aicert-pro baseline check aicert.yaml
  env:
    AICERT_LICENSE: ${{ secrets.AICERT_LICENSE }}

---

Exit Codes

Code	Meaning
0	Success
2	Threshold failure
3	Config/schema error
4	Provider/auth error
5	License error (Pro only)

---

License

Core package (aicert) is MIT licensed. Aicert Pro is commercial software.