omni-bench-core
The shared core of the omni benchmarks. Measuring a system on documents is always one loop: send each row of a dataset to the system, save its answer, and grade the answers against the gold. A task package (omni-parse-bench, omni-extract-bench, or your own) writes the parts that differ, and this package runs the loop.
from omni_bench_core import benchmark
b = benchmark.Benchmark(SomeTask(), "hf://datasets/<org>/<name>", [SomeScorer()])
b.run(SomeAdapter(mode="accurate"), out="runs", sample=20) # plan, predict, grade
b.grade("mine/") # a folder of your own predictions
What you write, from least to most:
| to | write |
|---|---|
| run a benchmark | Benchmark(task, dataset, scorers).run(SomeAdapter(...), out), once per system |
| add a system | an Adapter: a frozen dataclass of its options, and call(input) -> Call (or raise SystemFailed) |
| add a way of grading | a Scorer: a frozen dataclass, gold(row), score(pred, gold) -> Score (or raise Unscored) and summarize(lines), which returns headline and headline_is |
| add a task | a Task: rows(dataset) and input(row), a row type, and its adapters and scorers |
Where to read next:
OVERVIEW.md: the pieces one at a time, with diagrams. Start here if you are new;tests/test_core.py: a whole toy task, with a test of every rule (from a clone:PYTHONPATH=. python tests/test_core.py); and- the class docstrings in
omni_bench_core/: what each method is for, and when you would write it.
Two limits are deliberate:
- a dataset is a folder or an
hf://address, never rows made in code, because a run folder records which dataset its answers are for; and - an adapter calls one row at a time; a model that batches is served, and its adapter calls the server.
1 maintainer
Metadata
1 maintainer
Release files for omni-bench-core 0.7.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| omni_bench_core-0.7.0.tar.gz | 55.3 kB | With provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| omni_bench_core-0.7.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 98.1 kB
Release files / omni_bench_core-0.7.0.tar.gz
| Download URL | omni_bench_core-0.7.0.tar.gz |
|---|---|
| Size | 55.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
2761651c21c20b81ab6b20f3172b1de41aa7b9fae7235e1da384846d83055f29
|
|
BLAKE2b-256 checksum How to use checksums |
2d9f52331c08fd2b9f1e8be066cea5ab699c47efe1f8e833caa226b553a9cee1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 20, 2026.
Transparency logRelease files / omni_bench_core-0.7.0-py3-none-any.whl
| Download URL | omni_bench_core-0.7.0-py3-none-any.whl |
|---|---|
| Size | 42.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
268966bd534f3676e29de17a951b221cea048d83f2ea5126a8d0f3a23913a01e
|
|
BLAKE2b-256 checksum How to use checksums |
41ead280df9ed5b24fb037663f63a856f9317c5982ab94060bb86d3557ca2496
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 20, 2026.
Transparency logSecurity for omni-bench-core 0.7.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
All 2 files in this release have been uploaded with provenance metadata.
Attestation for omni_bench_core-0.7.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 20, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
