valcore 0.0.4

A small, self-contained local tool for developing, improving, and running agentic evaluations. Author evaluators in a web UI, run them over datasets from the command line, and gate CI on their accuracy.

Install

With Homebrew:

brew install duncankmckinnon/tap/valcore

Or as a uv tool:

uv tool install valcore

Either way you get an valcore command on your PATH.

Web UI

valcore serve opens a dark-themed web UI with four surfaces:

• Overview — the landing page, summarizing what you have and pointing to the next step.
• Evaluators — author, version, and validate LLM-as-judge evaluators.
• Datasets — build and edit the datasets evaluators run over, by hand or generated from a description.
• Runs — inspect completed runs, their metrics, and per-row scores, and compare runs against each other.

Quickstart

Store your gateway API key, then start the app:

valcore config set-key sk-...
valcore serve

serve starts the web UI and API on http://127.0.0.1:8000 and opens a browser (pass --no-browser to skip that, or --port to bind elsewhere). Author evaluators and datasets in the UI, then drive runs from the command line. Both can be written by hand or generated from a description; a generated result is an editable draft either way.

Seeding one from the other

An evaluator and a dataset have to agree on columns, so rather than retype that shape you can seed either one from the other and let the model fill in content you describe.

Generate a dataset from an evaluator version and the dataset always gets that version's required columns; you can add extra columns by naming them, and per-column notes say what each should contain. Suggested labels are optional — ask for them when you want the model to propose ground truth, and the label space comes from the evaluator.

Generate an evaluator from a dataset and it is drafted against that dataset's columns, with per-column notes saying how each factors into the assessment. The result is an editable draft, not a saved version, so you review and adjust it before keeping it.

A dataset needs no labels to be scored: an ordinary run just records the judge's output. Labels are only required for a validation run, which compares the judge against them to measure agreement.

Models and the gateway

valcore reaches models through the Pydantic AI Gateway. That is currently the only route: there is no direct-to-provider client and no per-provider API key, so a gateway key is required before anything that calls a model will run.

Model strings are always gateway/<provider>:<model>:

gateway/anthropic:claude-sonnet-5      # the default
gateway/anthropic:claude-opus-4-5
gateway/anthropic:claude-haiku-4-5
gateway/openai:gpt-5
gateway/google:gemini-2.5-pro

Valid providers are anthropic, openai, google, google-cloud, bedrock, and groq. A string that does not match this shape is rejected before any request is made, so a bare claude-sonnet-5 fails fast with a clear error rather than at call time.

The key is stored in ~/.valcore/config.toml (mode 0600) and exported as PYDANTIC_AI_GATEWAY_API_KEY when a command runs. An already-exported environment variable always wins over the stored key, which is what you want in CI:

export PYDANTIC_AI_GATEWAY_API_KEY=sk-...

Run valcore config set-key with no argument to be prompted without echoing the key.

Override the default model, highest precedence first: an explicit argument, VALCORE_DEFAULT_MODEL, model in config.toml, then the built-in default.

On other providers. Routing everything through one gateway keeps model access to a single credential and a single validated string format. It also means valcore inherits whatever the gateway supports and nothing else. Provider routing is confined to one module, so widening this later — direct provider clients, a self-hosted or OpenAI-compatible endpoint, local models — is a change to that resolution layer and the config schema rather than a change to how evaluators, datasets, or runs work.

Commands

Evaluators, versions, and datasets are addressable by name or by a unique id prefix; an ambiguous value is an error that lists the candidates. Pass --db PATH on the group to point at a SQLite database other than the default under ~/.valcore.

run accepts --version (defaults to the active version), --kind, --concurrency, --watch (one line per completed row), --json, and --min-accuracy. Progress goes to stderr and results go to stdout, so redirecting stdout yields clean JSON. The CLI talks to SQLite directly, so run works whether or not serve is up.

Agent skills

valcore ships a skill document that teaches a coding agent how to drive it — the data model, the author/validate/run/export loop, the evaluator-dataset compatibility rules, and a full CLI reference. Install it into whichever agent you use:

valcore skills install --claude          # ./.claude/skills/
valcore skills install --claude --copilot
valcore skills install                   # ./.agents/skills/, discoverable by any client
valcore skills install --claude --global # ~/.claude/skills/

Flags are additive and nothing is implicit — --claude --copilot writes exactly those two directories and leaves .agents/ alone. Add --global for home-level directories instead of the current repository.

Copying is the default: an already-identical skill is skipped, and one you have edited prompts before being overwritten (--force to skip the prompt). Use --symlink to link to the packaged copy instead, so upgrading valcore upgrades the installed skill.

Run valcore skills list to see what is bundled and where each copy currently lives.

Using valcore in CI

run --json emits a single object with run metadata, metrics, and per-row scores, and --min-accuracy turns a validation run into a pass/fail gate:

valcore run my-evaluator my-dataset \
  --kind validation \
  --min-accuracy 0.9 \
  --json > run.json

Exit codes:

--min-accuracy requires a categorical accuracy metric; numeric or unlabeled runs have no accuracy and error rather than silently passing.

~/.valcore

All state lives under ~/.valcore (mode 0700). Set VALCORE_HOME to relocate it.

~/.valcore/              0700
  config.toml              0600  gateway key + defaults
  valcore.db             SQLite (plus -wal, -shm)
  logs/                    serve logs

Development

uv sync                    # install dependencies into a local venv
uv run pytest              # run the test suite

The web UI is a Vite + React SPA under web/:

cd web
npm install
npm run dev                # Vite dev server, proxying the API

To build the SPA into the wheel, run npm run build and copy web/dist/ into src/valcore/web_dist/ before uv build; the release workflow does this automatically on a v* tag.