llama-stack-provider-csoai: a remote::csoai eval provider
Apache-2.0. This is an out-of-tree Llama Stack eval provider, and it takes no network access.
- A benchmark lists directories of CSOAI signed evidence batches (
batch.json,batch.signed.json,events.jsonl). - Each eval job verifies every directory offline against a pinned DID document.
- The job returns one row per event. The row carries the event's state word, or INVALID / UNVERIFIABLE_KEY when the batch does not verify.
aggregated_resultscounts state words. It holds no score and no average.- The candidate model in the request is never called.
Install
pip install llama-stack-provider-csoai # Python >= 3.12, llama-stack 0.7.x
curl -o did.json https://csoai.org/.well-known/did.json # pin the issuer's keys; verification never fetches them
Add the provider to your run config with the module: form:
apis:
- eval
providers:
eval:
- provider_id: csoai
provider_type: remote::csoai
module: llama_stack_provider_csoai
config:
did_json: ${env.CSOAI_DID_JSON:=./did.json}
Then register a benchmark whose metadata.bundles lists your batch directories, and run a job:
llama stack run run.yaml &
curl -X POST localhost:8321/v1alpha/eval/benchmarks -H 'content-type: application/json' \
-d '{"benchmark_id":"csoai-evidence","dataset_id":"csoai-evidence","scoring_functions":["csoai::state"],"provider_id":"csoai","metadata":{"bundles":["./batch-dir"]}}'
curl -X POST localhost:8321/v1alpha/eval/benchmarks/csoai-evidence/jobs -H 'content-type: application/json' \
-d '{"benchmark_config":{"eval_candidate":{"type":"model","model":"none-no-model-is-called","sampling_params":{}}}}'
curl localhost:8321/v1alpha/eval/benchmarks/csoai-evidence/jobs/<job_id>/result
A Helm chart that runs this provider in a Llama Stack server is in the repository at https://councilof.ai/helm/:
helm repo add csoai https://councilof.ai/helm/
helm install csoai-evidence csoai/llama-stack-csoai-eval
What the fixture job returns
The source distribution carries fixtures signed with a published test key (they attest nothing). A job over them returns these rows, in this order:
- CONSISTENT
- DIVERGENT (the negative-control case)
- UNMEASURED
- INVALID (a one-word edit made after signing)
- UNVERIFIABLE_KEY
The older external_providers_dir layout is also shipped, in providers.d/remote/eval/csoai.yaml.
Dependency pin. llama-stack 0.7.3 declares mcp>=1.23.0. With the resolver's choice, mcp 2.2.0 (read 30 Sep 2026), the server fails at import: cannot import name 'McpError' from 'mcp'. Pinning mcp<2 makes it start. The declared range and the import disagree.
Not measured.
- Registering this provider on a hosted OpenShift AI cluster was not done, because it needs an account. That registration is UNMEASURED.
- The Helm chart was rendered and linted; it has not been installed on a live cluster. That install is UNMEASURED.
Limit. A VALID batch shows who signed these bytes. It does not show that any claim inside is true.
This provider is our code. It is not a Red Hat or Meta integration, and no vendor has reviewed it. More: https://councilof.ai/connect/
Metadata
Release files for llama-stack-provider-csoai 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| llama_stack_provider_csoai-0.1.0.tar.gz | 16.0 kB | Without provenance | Details for llama_stack_provider_csoai-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| llama_stack_provider_csoai-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for llama_stack_provider_csoai-0.1.0-py3-none-any.whl |
Total release size: 27.2 kB
Release files / llama_stack_provider_csoai-0.1.0.tar.gz
| Download URL | llama_stack_provider_csoai-0.1.0.tar.gz |
|---|---|
| Size | 16.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
f4b8b722bfabff914d75a5b31fc96ae31813fbb269dbc4adaaf491f4b065495a
|
|
BLAKE2b-256 checksum How to use checksums |
108d091fe8b5df05c469dcf8cba71c933384ab9d7cafa153966587c583180524
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.11.10
|
Release files / llama_stack_provider_csoai-0.1.0-py3-none-any.whl
| Download URL | llama_stack_provider_csoai-0.1.0-py3-none-any.whl |
|---|---|
| Size | 11.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
a2eac5b5189e0abf6248ace331e096c16a0e761a08b0863b6eeaf14a5282337a
|
|
BLAKE2b-256 checksum How to use checksums |
174fc1765082bad9294ce77578745ad0b42f60d2a6986969f9e208c25b633e7f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.11.10
|
Security for llama-stack-provider-csoai 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.