Skip to main content

hanzo-research

File a measurement made in Python to Hanzo's research plane (/v1/research) and its benchmark arena (/v1/benchmark), with what it takes to believe or repeat it: the harness commit, the model and its weights' SHA-256, the host, CPU and GPU, the device and dtype, the Python and package versions, the date. The Python peer of the Rust crate hanzo-research (hanzoai/ml): the same wire records, the same RFC 8785 digests, the same lineage and WAL records. Standard library only, plus hanzo-login (itself standard library only), the one place a Hanzo tool learns who it acts as.

pip install hanzo-research        # or: uv run --with hanzo-research python measure.py

The credential is HANZO_API_KEY, else what hanzo auth token prints (the CLI at ~/.local/bin/hanzo, never one found on PATH; a CLI token is fetched again when it expires), else the legacy apiKey in ~/.hanzo/config.json, last. Records land in the org the server reads from the token (a person's home org, the first of its signed orgs; a machine token's owner; an API key's own org) and in the token's project (its project claim, else default). Research(org="lux") acts in another org a person's token belongs to (X-Org-Id); a switch the server would not honour, where it would file under the home org instead, is refused before any request, as is a credential sent over plain http to another machine.

Worked example

Measure something, file it with its provenance, make it public, read it back.

import hashlib
import platform
import time

from hanzo_research import Benchmark, Research, Run, Split, capture, now

r = Research()
bench = Benchmark(
    id="demo.sha256",
    title="SHA-256 over 16 MiB",
    splits=[Split("16mib", 30, "30 hashes of 16 MiB of zeros, one process, after one discarded")],
    metric="p50_ms",
    metrics=["p50_ms", "p95_ms", "mb_s"],
)
p = capture(device="cpu")  # harness commit, host, CPU, GPU, Python, packages, date
start = now()
data = bytes(16 << 20)
hashlib.sha256(data).digest()
ms = []
for _ in range(30):
    t = time.perf_counter()
    hashlib.sha256(data).digest()
    ms.append(1000 * (time.perf_counter() - t))
ms.sort()

run = Run(
    id=f"demo.sha256-16mib-cpu-{p.host}",
    benchmark=bench.id,
    split="16mib",
    system="hashlib",
    version=platform.python_version(),
    questions=30,
    answered=30,
    when=start,
    ended=now(),
)
run.put("all", "p50_ms", ms[14], n=30, of=30)
run.put("all", "p95_ms", ms[28], n=30, of=30)
run.put("all", "mb_s", 16 / (ms[14] / 1000))
p.apply(run)  # fills commit and by, appends the provenance sentence and its digest to notes

r.record(benchmarks=[bench], runs=[run])  # the definition, then the run: both private
r.grant(run.id, visibility="public", publishable=True)
back = r.get_run(run.id, org=r.org)  # the public read: no credential
print(back.completion, back.measure("all", "p50_ms").value, back.commit)

completion is the server's: complete only when answered == questions and ended is set; partial, inconsistent or unknown otherwise. A model run names its weights: capture(model="kai-1.3", weights="/models/kai-1.3", device="cuda", dtype="bf16") hashes a file, or a directory: one holding SHA256SUMS is checked against it (every file listed and matching, none unlisted) and named by that file's digest, as a Kai release is pinned; any other is digested as its sha256sum listing. apply sets system to the model and version to <model>@<sha256>. A dirty harness tree is recorded with the SHA-256 of git diff HEAD and the contents of the untracked files, so the number stays pinned to the code that made it; a git that fails records unknown, never clean.

The research plane: Research

call route
record(benchmarks=, studies=, runs=, papers=) POST /v1/research/{benchmarks,studies,runs,papers}, one batch a non-empty kind, in that order: {"runs": [...]}
benchmarks(id=, project=), get_benchmark(id) GET /v1/research/benchmarks
runs(id=, key=, benchmark=, system=, study=, completion=, since=, until=, project=) GET /v1/research/runs: your org's runs, private ones included, with the completion census and the fields runs leave unstated
get_run(id, project=) one canonical run in a project of your org (default the client's: an id is unique only within a project)
runs(org=...), get_run(id, org=...) the same route with ?org= and no credential: that org's public runs
studies(...), papers(..., cites=) GET /v1/research/{studies,papers}
compare(a, b) GET /v1/research/compare: the reasons the two runs may not be compared, the config they differ in, the deltas
grant(id, visibility=, publishable=, trainable=), grant(sha256=, visibility=) POST /v1/research/grants: on every version of the id; a write is always private until this
revise(record) a new version marked corrected, the run's grants carried to it (a new version lands private)
retract(id, why) a new version marked retracted: the id leaves every listing, earlier versions stay
ingest, query, totals, projects, experiment(...) the experiment plane: /v1/research/{experiments,totals,projects}
upload_artifact, upload_json_artifact, artifacts, get_artifact /v1/research/artifacts[/<sha256>]: snapshot or report bytes, addressed by the server's hash
url(run_id, org=) the public read address of one run: the source a claim cites
measure(identity, fn, force=False) GET /v1/research/runs?key=&completion=complete, then fn and POST /v1/research/runs only on a miss or with force

Every write is a batch of at most 20,000 and is idempotent by content: the same record again records nothing. A mapping is sent as given; a Run is sent with every field. A refusal raises Denied (401, 403), Missing (404), Invalid (400, 409, 422), Limited (413, 429) or Unavailable (502 to 504, no answer), each carrying the server's own sentence; a 503 is asked again after 2, 4 and 8 seconds. A NaN or infinity never leaves: json refuses it before the request.

Measure once

A run's key is the SHA-256 of what decides its number: benchmark, split, system, version, k, temperature, max_tokens, embedder, reader and the four digests (IDENTITY), and nothing about who ran it, when, or at which commit. The server derives it on every run and filters on it; key(run) computes the same bytes before anything is spent. measure asks for a complete run under that key and calls fn only on a miss:

identity = Run(
    id="locomo-all-k20-enso-flash",
    benchmark="locomo",
    split="all",
    system="enso-flash",
    version="1.2.0",
    k=20,
    temperature=0.0,
    dataset_digest=digest,
)
run = r.measure(identity, lambda: evaluate())  # returns measures, questions, answered, ended
r.measure(identity, evaluate, force=True)  # measure again regardless
r.runs(key=key(identity))  # every run of that measurement

A partial run is not a hit. fn may not change an identity field: a result filed under a configuration it did not measure is the false hit this exists to prevent.

The arena: Arena (or Research().arena)

call route
catalog() GET /v1/benchmark/catalog
leaderboard(benchmark) GET /v1/benchmark/leaderboard: each model's measured score (its latest run) beside a published claim. Only claims a SuperAdmin wrote under org admin, and public, reach it.
claims(benchmark=, model=, provider=, source=, protocol=, org=) GET /v1/benchmark/claims: every org's public claims, labelled, and your org's private ones
post_claims(rows, visibility=) POST /v1/benchmark/claims: under your acting org; each row names its protocol and its source (for a measured score, Research.url(run_id)), or it is refused here
history(benchmark, model=), compare(a, b, benchmark=) GET /v1/benchmark/{history,compare}: read the arena's per-item attempts plane, not the research runs
presets(), post_preset(preset) GET/POST /v1/benchmark/presets: validated and echoed as enso-<name>, not stored
post_run(benchmarks, model) POST /v1/benchmark/runs: per benchmark, the model's latest run the arena holds, or that nothing was queued (the arena has no executor); spends nothing

The command

hanzo-research (or python -m hanzo_research); every answer is JSON on stdout.

hanzo-research post runs.jsonl --grant public --publishable   # a run per line
hanzo-research post plan.json                                 # {"benchmarks": [...], "runs": [...]}
hanzo-research post plan.json --dry                           # print the bodies, file nothing
hanzo-research list runs --benchmark demo.sha256
hanzo-research list runs --public hanzo                       # hanzo's public runs, no credential
hanzo-research get demo.sha256-16mib-cpu-dgx --public hanzo
hanzo-research grant RUN_ID ... --visibility public --publishable
hanzo-research retract RUN_ID --why "read on the wrong split"
hanzo-research compare RUN_A RUN_B
hanzo-research provenance --model kai-1.3 --weights /models/kai-1.3 --device cpu
hanzo-research arena leaderboard --benchmark gpqa_diamond

It is not hanzo research: the hanzo command is the native CLI, and no Python package in this workspace may install one.

Records beside the API

  • canonical.canonicalize(value), canonical.digest(value): RFC 8785 bytes and their SHA-256, equal to the crate's rfc8785_canonicalize/rfc8785_digest on every string, integer within 2^53 and float from 1e-6 to 1e21. Past that the crate writes Rust's Display (1e21 as 22 digits, -0.0 as -0), which is not RFC 8785; this package writes the RFC's 1e+21 and 0. tests/vectors/rust.json holds the crate's own output.
  • lineage: DataPurpose, LineagePolicy (a child never less restrictive than a parent), ContaminationError, verify_promotion_eligibility, and the crate's evidence records (EnvironmentV1, GateSpec, Claim, TeacherInteraction, PairEvidence, DeterminismReport, ArtifactRef, ResearchEvent), each with the crate's digest.
  • wal.ResearchWal: an fsynced, hash-chained event log. A log the crate wrote verifies here (wal.verify), and the reverse.

Tests

uv run --no-project --with pytest pytest pkg/hanzo-research              # no network
uv run --no-project --with pytest pytest pkg/hanzo-research -m integration  # api.hanzo.ai, no credential

Metadata

Release files for hanzo-research 0.3.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for hanzo-research 0.3.1
File Size Uploaded Provenance Actions
hanzo_research-0.3.1.tar.gz 64.1 kB Without provenance Details for hanzo_research-0.3.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for hanzo-research 0.3.1
File Interpreter ABI Platform Actions
hanzo_research-0.3.1-py3-none-any.whl Python 3 none any Details for hanzo_research-0.3.1-py3-none-any.whl

Total release size: 108.9 kB

Release files / hanzo_research-0.3.1.tar.gz

Download URL hanzo_research-0.3.1.tar.gz
Size 64.1 kB
Tags Source
SHA-256 checksum
How to use checksums
b0631d5d2fb9a4c486ac947c26ce9d30a86aefba51a1abc5198ddbd3e8f43f0d
BLAKE2b-256 checksum
How to use checksums
77e0bce8d93694f669eeb41eda864f491d5ebf249cfac20c4ec3cc704e919661
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.9

Release files / hanzo_research-0.3.1-py3-none-any.whl

Download URL hanzo_research-0.3.1-py3-none-any.whl
Size 44.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
6c13c0fdbcdb1b148f039f225655d4e056fb01f1cad227e05f4999fa6e8b72e2
BLAKE2b-256 checksum
How to use checksums
509748d15771edc6a9f41d9a7050413a90c919f3395267dcd2f2d840334306c9
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.9

Security for hanzo-research 0.3.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page