ankora
Turn the traces you already capture into regression tests that fail your CI when quality silently drops.
Teams have observability but no tests. ~89% of teams run some form of LLM observability, but only ~52% run evals — and quality is the No. 1 blocker to shipping LLM features to production. So prompt tweaks, model-version bumps, and silent provider changes ship undetected until users complain.
ankora closes that gap from the supply side: it replays the traces you
already have as a deterministic regression suite, scores the outputs, and
exits non-zero when quality regresses — so a GitHub Action can block the
merge. Local-first, bring-your-own-keys, no account, no telemetry.
60-second quickstart (no API keys required)
Install from source with uv:
git clone https://github.com/ankora/ankora
cd ankora
uv sync
Coming once published to PyPI:
uv tool install ankora(orpip install ankora).
Now run the fully offline demo — it uses the built-in deterministic echo
provider and deterministic scorers, so no network and no keys:
bash examples/run_demo.sh
You'll watch the full loop run → baseline set → gate and see the CI contract both ways: a green gate (exit 0) when the run matches the baseline, then a red gate (exit 1) after a Case is deliberately broken:
==> 3. Gate against the baseline (clean — expect exit 0)
No regressions — gate passed.
clean gate exit code: 0
...
==> 5. Gate again (regression — expect non-zero exit)
1 regression(s) detected — failing the gate.
broken gate exit code: 1
That's the whole idea: a regression makes the command exit non-zero.
The core loop
In your own repo, the loop is four commands. Point target.provider at
openai/anthropic in ankora.yaml for real replays (keys come from your
env, below), or keep the keyless echo provider to try it out.
# 0. Scaffold ankora.yaml + an evals/ directory
ankora init
# 1. Turn an OpenTelemetry GenAI or Langfuse trace export into regression Cases
# (format is auto-detected; force it with --format otel|langfuse)
ankora ingest traces.json --out evals/
# 2. Replay + score the suite; saves a run under .ankora/runs/
ankora run
# 3. Promote a good run to the baseline
ankora baseline set <run_id>
# 4. The CI entrypoint: replay, diff vs baseline, exit non-zero on regression
ankora gate
Inspect any two runs read-only (never fails the build):
ankora diff <baseline_run_id> <current_run_id>
Every command has --help; --config points at a non-default ankora.yaml,
--target provider:model overrides the target, and --concurrency bounds
parallel replays.
Configuration (ankora.yaml)
version: 1
suites: ["evals/**/*.yaml"]
target:
provider: openai # openai | anthropic | echo (keyless, for demos/CI)
model: gpt-4o-mini
providers:
openai: {api_key_env: OPENAI_API_KEY} # keys read from env, never inlined
scorers:
- type: exact # deterministic, no key needed
threshold: 1.0
- type: regex
pattern: '"country"'
- type: json_schema
schema: {type: object, required: [city, country]}
- type: llm_judge # needs a provider key
judge: {provider: openai, model: gpt-4o}
rubric: "Score 1 if factually consistent with the reference, else 0."
threshold: 0.7
- type: embedding_similarity
model: {provider: openai, model: text-embedding-3-small}
threshold: 0.85
gate:
fail_on: regression # "regression" (vs baseline) or "absolute" (vs thresholds)
baseline: .ankora/baseline.json
OpenAI-compatible endpoints
The openai provider can talk to any OpenAI-compatible endpoint — Google Gemini's
OpenAI-compat API, OpenRouter, Groq, Together, or a local Ollama / LM Studio
server — by setting base_url on the provider. Leave it unset to hit
api.openai.com as usual. Keys are still read from the env var you name; ankora
never sees or stores them.
Gemini (free tier) — get a key from Google AI Studio:
target:
provider: openai
model: gemini-2.0-flash
providers:
openai:
api_key_env: GEMINI_API_KEY
base_url: https://generativelanguage.googleapis.com/v1beta/openai/
scorers:
- type: llm_judge # judge over the same endpoint
judge: {provider: openai, model: gemini-2.0-flash}
rubric: "Score 1 if factually consistent with the reference, else 0."
threshold: 0.7
- type: exact # deterministic, no model needed
threshold: 1.0
export GEMINI_API_KEY=... # from Google AI Studio
ankora run
OpenRouter — one key, hundreds of models (openrouter.ai):
target:
provider: openai
model: openai/gpt-4o-mini # any OpenRouter model slug
providers:
openai:
api_key_env: OPENROUTER_API_KEY
base_url: https://openrouter.ai/api/v1
Embeddings caveat: many OpenAI-compatible endpoints expose only chat completions, not the
/embeddingsroute. On those, prefer thellm_judgeand deterministic (exact,regex,json_schema) scorers; useembedding_similarityonly against a provider whose endpoint actually serves embeddings.
Wire it into CI
Add a workflow to your repo that runs ankora gate on pull requests. Provider
keys come from repo secrets — ankora reads them from the environment and
never sees or stores your tokens.
# .github/workflows/ankora.yml
name: ankora
on: pull_request
jobs:
gate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: astral-sh/setup-uv@v5
- run: uv tool install ankora # once published to PyPI
- run: ankora gate
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
A composite action that wraps those steps ships at
.github/actions/ankora-gate.
Commit .ankora/baseline.json (or promote a run with ankora baseline set)
so CI has something to compare against.
Why it's different
- Local-first & offline. Runs on your laptop or a CI runner. No account, no login, no hosted service — and no telemetry, ever.
- Bring-your-own-keys. Replays use your provider keys from env vars. We never carry token cost and never see your tokens.
- Neutral & framework-agnostic. Reads open formats (OpenTelemetry GenAI semantic conventions first). Your suite is plain YAML checked into your repo — no lock-in to a framework, provider, or storage backend.
- It fails your CI. The whole point:
ankora gateexits non-zero on regression, so a quality drop blocks the merge instead of reaching users.
v1 scope — and what's next
Shipped in v1:
init— scaffoldankora.yaml+evals/ingest— OpenTelemetry GenAI and Langfuse traces → regression Cases (format auto-detected; override with--format {otel,langfuse,auto})run— deterministic replay + scoring, persisted runsdiff— per-case comparison of two runsgate— replay + baseline diff + non-zero exit on regression (the CI entrypoint)baseline set— promote a run to the baseline- Providers:
openai,anthropic, and a keylessechoprovider for demos/CI - Scorers:
exact,regex,json_schema(deterministic),embedding_similarity,llm_judge
Coming next (not built yet — no false promises):
- A scheduled drift watch (
runon a cron against a live endpoint) - Multi-step agent-trajectory record/replay with tool mocking
v1 targets single-turn LLM replay; recorded tool calls are kept as reference data but not yet re-executed.
Development
uv sync
uv run pytest # all provider calls are mocked; no live API calls
uv run ruff check
uv run ruff format --check
Releasing
Maintainers: see RELEASING.md. Publishing to PyPI happens automatically when you cut a GitHub Release, via Trusted Publishing (OIDC — no API tokens stored in the repo).
License
Metadata
Release files for ankora 0.2.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| ankora-0.2.1.tar.gz | 258.2 kB | With provenance | Details for ankora-0.2.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| ankora-0.2.1-py3-none-any.whl | Python 3 | none | any | Details for ankora-0.2.1-py3-none-any.whl |
Total release size: 305.6 kB
Release files / ankora-0.2.1.tar.gz
| Download URL | ankora-0.2.1.tar.gz |
|---|---|
| Size | 258.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
976c6beb704952a0dad43c18f41c14a6354fb11386093456a6398347a0854cb2
|
|
BLAKE2b-256 checksum How to use checksums |
86d77d2564949612369d308fd73a38e5959a86098e7e59df42b14ab4758614c8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 6, 2026.
Transparency logRelease files / ankora-0.2.1-py3-none-any.whl
| Download URL | ankora-0.2.1-py3-none-any.whl |
|---|---|
| Size | 47.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
e055920b6ea017a2ee6fc4e2946f17fb373e16726231b666c2712ed79116360d
|
|
BLAKE2b-256 checksum How to use checksums |
03e535e2e73bf97ad199f3a5785ffa7c8548ca1469d142feb3830707ce7ac1bd
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 6, 2026.
Transparency logSecurity for ankora 0.2.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for ankora-0.2.1.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 6, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.