llm-eval-ci
pytest for LLMs and agents, in CI. Deterministic, offline-first evals that fail your build when a model output regresses — stdlib only, zero runtime dependencies, no API key.
llm-eval run ./examples/basic.json --report ./report.md --json ./report.json
# exit code != 0 as soon as one case breaks a threshold
Install (30 seconds)
pip install llm-eval-ci
From source:
git clone https://github.com/shenn667/llm-eval-ci.git
cd llm-eval-ci
pip install -e .
Requirements: Python 3.10+, nothing else at runtime (argparse, json, re, time from the stdlib). No network calls, no API keys.
10-line example
from llm_eval import Suite
suite = Suite.from_json("examples/basic.json") # load cases
result = suite.run(lambda p: {"output": "Hello, world!", # your ModelFn
"usage": {"tokens": 12, "cost_usd": 0.0}})
print(result.summary()) # 3/4 passed, 1 failed: [missing_keyword]
result.write_markdown("report.md")
result.write_json("report.json")
assert result.ok # False -> CI step fails
Bring your own model with one function: ModelFn(prompt) -> {"output": str, "usage": {"tokens": int, "cost_usd": float}}. For CI and demos, StubModel replays canned prompt-to-response fixtures from JSON — fully offline and reproducible.
CLI demo
Three pedagogical suites ship in examples/. Green one first:
python -m llm_eval run examples/basic.json --report report.md --json report.json
echo $? # 1 — 3 pass, 1 clean failure on purpose (see below)
Then the red ones — each fails the command with exit code 1, which is the point:
python -m llm_eval run examples/injection_pii.json --report report.md --json report.json
echo $? # 1 — injection obeyed + PII leak detected
python -m llm_eval run examples/latency_cost.json --report report.md --json report.json
echo $? # 1 — token / latency / cost budgets exceeded
Same suites as pytest, one test per case with native JUnit output for CI:
pytest --llm-eval=examples/basic.json
Sample report.md produced by the CLI:
# llm-eval report — 3/5 passed (60%)
| case | status | latency | cost |
|------|--------|---------|------|
| within_budgets | PASS | 5 ms | $0.0001 |
| too_many_tokens | FAIL (budget_tokens) | 5 ms | $0.0001 |
| too_slow | FAIL (budget_latency) | 300 ms | $0.0001 |
| too_expensive | FAIL (budget_cost) | 5 ms | $0.0500 |
| usage_not_declared | FAIL (budget_tokens) | 5 ms | n/a |
## Detections
- `pii_leaked_email`: PII pattern matched (email) — value redacted: ***
report.json mirrors it in machine-readable form: {version, cases: [{id, status, failed_checks[], latency_ms, usage}]}. Secret values are always masked as *** in both reports.
Checks
| check | what it does |
|---|---|
contains / not_contains |
substring present / absent in output |
regex |
re.search match on output |
json_valid |
output parses as JSON |
json_schema_lite |
in-house type/required subset (no jsonschema dependency) |
refusal |
output matches a refusal template (EN + FR) |
budget_tokens / budget_latency / budget_cost |
usage vs max_tokens / wall-clock vs max_latency_ms / cost_usd vs max_cost_usd; missing usage fails closed |
no_pii_leak |
FR + EN regexes (email, phone, IBAN, simplified ssn); hits are masked in reports |
no_injection_obey |
prompt matches known injection patterns and output is not a refusal |
Case-file format is JSON (array of cases) or JSONL (one case per line) — never YAML, so there is nothing to install. Each case carries a note field stating its expected verdict; see SPEC_CASES.md for the exact schema.
Limitations & Roadmap
Honest limits of v0.1.0:
- No LLM-as-judge, on purpose. A model grading a model costs money per run, is non-deterministic (flaky CI), and needs network + secrets — the opposite of a gate that must be fast, free, and reproducible. Deterministic string/schema/budget checks catch regressions; qualitative review stays human.
no_injection_obeyandno_pii_leakare pattern heuristics (regex + refusal templates), not guarantees. A novel phrasing can bypass them; the security fixtures document known bypass classes. They are tripwires, not proofs.- No semantic similarity or embeddings: paraphrased-but-correct answers fail
contains/regex. Write checks on invariants (keys, formats, refusals), not on wording. - Stub latency (
delay_ms) only simulates slowness; with real models,budget_latencyis wall-clock and can flake on loaded runners — set generous thresholds in CI. - No streaming, no server storage, no web dashboard, no provider adapters (OpenAI & co. are a v0.2.0 opt-in, outside the stdlib-only core).
Development
pip install -e . # dev install
pytest # >=30 tests, 100% on checks/
CI (GitHub Actions, <60 s, Python 3.10–3.14, no secrets): lint + pytest + CLI demo. LICENSE is MIT. Releases are tagged vX.Y.Z; PyPI publication is manual and requires founder approval.
License
MIT — see LICENSE.
Metadata
Release files for llm-eval-ci 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| llm_eval_ci-0.1.0.tar.gz | 40.9 kB | With provenance | Details for llm_eval_ci-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| llm_eval_ci-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for llm_eval_ci-0.1.0-py3-none-any.whl |
Total release size: 75.9 kB
Release files / llm_eval_ci-0.1.0.tar.gz
| Download URL | llm_eval_ci-0.1.0.tar.gz |
|---|---|
| Size | 40.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
13b12565bd4060e5871d74e95abdd8068468b76970cf2be6105f0926bc6411ea
|
|
BLAKE2b-256 checksum How to use checksums |
da48a42df453461747fd65e146759b7a5ab0e996492114430cee01e3f203fabb
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.
Transparency logRelease files / llm_eval_ci-0.1.0-py3-none-any.whl
| Download URL | llm_eval_ci-0.1.0-py3-none-any.whl |
|---|---|
| Size | 34.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
34a8f3af72de368f1607c4886077928641976c2eebf76203c39bdfca0e8b42ae
|
|
BLAKE2b-256 checksum How to use checksums |
210f69620a9af2522340564f5c88e4bf81547a3ce9c94ec873e59b84d772a3f8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.
Transparency logSecurity for llm-eval-ci 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for llm_eval_ci-0.1.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.