Skip to main content

llm-eval-ci

CI PyPI Python License: MIT

pytest for LLMs and agents, in CI. Deterministic, offline-first evals that fail your build when a model output regresses — stdlib only, zero runtime dependencies, no API key.

llm-eval run ./examples/basic.json --report ./report.md --json ./report.json
# exit code != 0 as soon as one case breaks a threshold

Install (30 seconds)

pip install llm-eval-ci

From source:

git clone https://github.com/shenn667/llm-eval-ci.git
cd llm-eval-ci
pip install -e .

Requirements: Python 3.10+, nothing else at runtime (argparse, json, re, time from the stdlib). No network calls, no API keys.

10-line example

from llm_eval import Suite

suite = Suite.from_json("examples/basic.json")          # load cases
result = suite.run(lambda p: {"output": "Hello, world!",  # your ModelFn
                              "usage": {"tokens": 12, "cost_usd": 0.0}})
print(result.summary())        # 3/4 passed, 1 failed: [missing_keyword]
result.write_markdown("report.md")
result.write_json("report.json")
assert result.ok               # False -> CI step fails

Bring your own model with one function: ModelFn(prompt) -> {"output": str, "usage": {"tokens": int, "cost_usd": float}}. For CI and demos, StubModel replays canned prompt-to-response fixtures from JSON — fully offline and reproducible.

CLI demo

Three pedagogical suites ship in examples/. Green one first:

python -m llm_eval run examples/basic.json --report report.md --json report.json
echo $?   # 1 — 3 pass, 1 clean failure on purpose (see below)

Then the red ones — each fails the command with exit code 1, which is the point:

python -m llm_eval run examples/injection_pii.json --report report.md --json report.json
echo $?   # 1 — injection obeyed + PII leak detected
python -m llm_eval run examples/latency_cost.json --report report.md --json report.json
echo $?   # 1 — token / latency / cost budgets exceeded

Same suites as pytest, one test per case with native JUnit output for CI:

pytest --llm-eval=examples/basic.json

Sample report.md produced by the CLI:

# llm-eval report — 3/5 passed (60%)

| case | status | latency | cost |
|------|--------|---------|------|
| within_budgets | PASS | 5 ms | $0.0001 |
| too_many_tokens | FAIL (budget_tokens) | 5 ms | $0.0001 |
| too_slow | FAIL (budget_latency) | 300 ms | $0.0001 |
| too_expensive | FAIL (budget_cost) | 5 ms | $0.0500 |
| usage_not_declared | FAIL (budget_tokens) | 5 ms | n/a |

## Detections
- `pii_leaked_email`: PII pattern matched (email) — value redacted: ***

report.json mirrors it in machine-readable form: {version, cases: [{id, status, failed_checks[], latency_ms, usage}]}. Secret values are always masked as *** in both reports.

Checks

check what it does
contains / not_contains substring present / absent in output
regex re.search match on output
json_valid output parses as JSON
json_schema_lite in-house type/required subset (no jsonschema dependency)
refusal output matches a refusal template (EN + FR)
budget_tokens / budget_latency / budget_cost usage vs max_tokens / wall-clock vs max_latency_ms / cost_usd vs max_cost_usd; missing usage fails closed
no_pii_leak FR + EN regexes (email, phone, IBAN, simplified ssn); hits are masked in reports
no_injection_obey prompt matches known injection patterns and output is not a refusal

Case-file format is JSON (array of cases) or JSONL (one case per line) — never YAML, so there is nothing to install. Each case carries a note field stating its expected verdict; see SPEC_CASES.md for the exact schema.

Limitations & Roadmap

Honest limits of v0.1.0:

  • No LLM-as-judge, on purpose. A model grading a model costs money per run, is non-deterministic (flaky CI), and needs network + secrets — the opposite of a gate that must be fast, free, and reproducible. Deterministic string/schema/budget checks catch regressions; qualitative review stays human.
  • no_injection_obey and no_pii_leak are pattern heuristics (regex + refusal templates), not guarantees. A novel phrasing can bypass them; the security fixtures document known bypass classes. They are tripwires, not proofs.
  • No semantic similarity or embeddings: paraphrased-but-correct answers fail contains/regex. Write checks on invariants (keys, formats, refusals), not on wording.
  • Stub latency (delay_ms) only simulates slowness; with real models, budget_latency is wall-clock and can flake on loaded runners — set generous thresholds in CI.
  • No streaming, no server storage, no web dashboard, no provider adapters (OpenAI & co. are a v0.2.0 opt-in, outside the stdlib-only core).

Development

pip install -e .          # dev install
pytest                    # >=30 tests, 100% on checks/

CI (GitHub Actions, <60 s, Python 3.10–3.14, no secrets): lint + pytest + CLI demo. LICENSE is MIT. Releases are tagged vX.Y.Z; PyPI publication is manual and requires founder approval.

License

MIT — see LICENSE.

Metadata

Release files for llm-eval-ci 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for llm-eval-ci 0.1.0
File Size Uploaded Provenance Actions
llm_eval_ci-0.1.0.tar.gz 40.9 kB With provenance Details for llm_eval_ci-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for llm-eval-ci 0.1.0
File Interpreter ABI Platform Actions
llm_eval_ci-0.1.0-py3-none-any.whl Python 3 none any Details for llm_eval_ci-0.1.0-py3-none-any.whl

Total release size: 75.9 kB

Release files / llm_eval_ci-0.1.0.tar.gz

Download URL llm_eval_ci-0.1.0.tar.gz
Size 40.9 kB
Tags Source
SHA-256 checksum
How to use checksums
13b12565bd4060e5871d74e95abdd8068468b76970cf2be6105f0926bc6411ea
BLAKE2b-256 checksum
How to use checksums
da48a42df453461747fd65e146759b7a5ab0e996492114430cee01e3f203fabb
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

Release files / llm_eval_ci-0.1.0-py3-none-any.whl

Download URL llm_eval_ci-0.1.0-py3-none-any.whl
Size 34.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
34a8f3af72de368f1607c4886077928641976c2eebf76203c39bdfca0e8b42ae
BLAKE2b-256 checksum
How to use checksums
210f69620a9af2522340564f5c88e4bf81547a3ce9c94ec873e59b84d772a3f8
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

Security for llm-eval-ci 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for llm_eval_ci-0.1.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page