Skip to main content

ek

ek (Evaluation Kit) — a framework for building Knowledge Evaluation systems, evaluating the outputs of information-extraction systems. OCR is treated as the noisiest special case of a general problem, so the core is source-agnostic and the OCR pieces are optional.

import ek

ek.score("hello wrld", "hello world")  # -> Score(value=0.0909..., metric='cer')
ek.score("hello wrld", "hello world", metric="wer").value  # 0.5
ek.evaluate(
    [("ct", "cat"), ("dg", "dog")], metric="cer"
).aggregate  # 0.333... (global CER)

What it does

Evaluating an extraction splits along two axes — is there a gold answer (reference-based) or not (reference-free), and are we scoring one item or a whole corpus. ek gives you both halves through two facades over one shared typed schema:

  • score() / evaluate() — reference-based: compare against gold, one item or a corpus, the metric chosen by output type (string → CER/WER, record → field-F1), aggregated correctly (global error-rate accumulation, micro-F1; never a naive mean) with optional per-slice cuts.
  • estimate_quality() — reference-free: gather signals → calibrate → validate → decide accept/flag/block, with no gold answer.

Everything swappable is a strategy injected with a smart default, so the simple call works out of the box and every layer stays replaceable.

Evaluate an OCR engine

The first concrete instance: measure OCR accuracy over a gold corpus. ek consumes ocracy's normalized OcrResult, so it can benchmark any of its ~16 engines — or any image -> OcrResult callable of your own.

import ek.ocr

gold = {
    "inv-1": {
        "image": "scan.png",
        "reference_text": "INVOICE 2024",
        "slice": "invoices",
    }
}
report = ek.ocr.evaluate_ocr(
    "ocrmac",
    gold,
    metric="cer",
    normalize=["lower", "collapse_whitespace"],
    persist=True,
)
report.aggregate  # corpus CER
report.per_slice  # CER per document slice
report.detail["per_item"]  # prediction, reference, score, confidence per document

Gold corpora, results, and runs persist to local dol stores under ~/.local/share/ek/.

Evaluate an AI agent — in cost per successful task

The second instance. Agent evaluation is the same 2×2, but the evaluated object is an episode (tool calls + observations ending in a final state) and the unit that matters is cost per successfully completed task, not cost per token — because tokens spent on a failed episode are pure waste.

from ek.agents import TaskSpec, run_suite, per_million

tasks = [
    TaskSpec("t1", input="2+2", gold="4", slice="easy"),
    TaskSpec("t2", input="17*23", gold="391", slice="hard"),
]

report = run_suite(my_agent, tasks, k=8, price=per_million(3.0, 15.0))

report.pass_hat_k  # reliability: succeeds on ALL 8 trials (the production number)
report.pass_at_k  # capability: succeeds on ANY of 8 trials
report.success_ci  # a Wilson interval — a point estimate is not a result
report.cost[
    "cost_per_success"
]  # Cost-of-Pass: dollars per *successful* task (inf if none)
report.per_slice  # ...cut by difficulty

Two numbers, not one: an agent that "usually works" is not shippable, and pass^k is what exposes it (a 90%-reliable agent is only ~43% reliable across 8 tries). Agent scores are stochastic, so the regression gate compares intervals, not points, and refuses to compare runs whose user-simulator or suite version changed:

from ek.agents import agent_regression_gate, save_agent_baseline

save_agent_baseline(report, "v1")
assert agent_regression_gate(
    new_report, "v1"
)  # fails only on a *real* regression, not noise

Also included: BFCL-style tool-call correctness (cost-weighted by the tool grammar, so a wrong argument to a destructive tool costs more), an order-sensitive trajectory distance, and an LLM-as-judge signal — with judge_validation(), because an unvalidated judge is a liability, not a metric. None of this needs an extra: ek.agents is pure-python, and the bridges duck-type, so it scores an Inspect/DeepEval run without importing either. With ek[agents] installed, run_inspect(task, model=..., epochs=k) runs an Inspect AI eval and returns graded episodes ready for reliability(..., k=k) (use model="mockllm/model" to try it offline).

Install

pip install ek            # lean, permissive core (dol, config2py, jiwer, rapidfuzz)
pip install "ek[ocr]"     # + the ocracy OCR fleet (install engines via ocracy extras)
pip install "ek[agents]"  # + external agent harnesses to *run* (inspect-ai, deepeval, ragas)
pip install "ek[agreement]"  # + uqlm self-consistency (heavy: torch/transformers)
pip install "ek[all]"     # + the permissive capability tiers (metrics, calibration, ...)

Heavier or copyleft/non-commercial libraries are never installed by default, and a CI license gate enforces it; see the extras in pyproject.toml.

CLI

ek cer "hello wrld" "hello world"     # character error rate
ek wer "hello wrld" "hello world"     # word error rate
ek pass-k 10 9 --k 8                  # pass@k (capability) vs pass^k (reliability)
ek cost-per-success 12.50 5           # dollars per successfully completed task
ek where                              # the local data folder
ek check tesseract                    # what an OCR engine needs to run

For contributors

The architecture, conventions, and the research behind the design are documented for agents and humans in AGENTS.md, the dev skills under skills/, and the research reports under misc/docs/.

Metadata

Release files for ek 0.1.15

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for ek 0.1.15
File Size Uploaded Provenance Actions
ek-0.1.15.tar.gz 547.6 kB Without provenance Details for ek-0.1.15.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for ek 0.1.15
File Interpreter ABI Platform Actions
ek-0.1.15-py3-none-any.whl Python 3 none any Details for ek-0.1.15-py3-none-any.whl

Total release size: 764.7 kB

Release files / ek-0.1.15.tar.gz

Download URL ek-0.1.15.tar.gz
Size 547.6 kB
Tags Source
SHA-256 checksum
How to use checksums
db93d28d72ab2bf60554cd666e62177a2433ea735f4166ee7a505852c178e83a
BLAKE2b-256 checksum
How to use checksums
707d2b7e018561187ed2a5abac8cb5f73efcb5b92b3fe55e6bdbd664344eb857
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.3 {"installer":{"name":"uv","version":"0.13.3","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / ek-0.1.15-py3-none-any.whl

Download URL ek-0.1.15-py3-none-any.whl
Size 217.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
122ab4b6903235570e131f7fff7e887c3b16842556b2301a771e2faa18c22b51
BLAKE2b-256 checksum
How to use checksums
046e4ae4e74d22ac07f29b82c8e93b5bd6e1002a0fd7397e0d3334b31f7850d3
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.3 {"installer":{"name":"uv","version":"0.13.3","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for ek 0.1.15

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page