Valkyrie

Valkyrie orchestrates scalable, reproducible evaluations for AI agents. Run it on infrastructure that Vals AI hosts, or on your own machine with a local Tracker.

Guides, the CLI reference and the Python SDK reference are published at docs.valkyrie.vals.ai.

Quickstart

Run Claude Code on one Terminal-Bench task with Vals AI's hosted Valkyrie. You need uv, a Vals AI account (email contact@vals.ai), a Modal account, and an Anthropic API key.

1. Install the CLI. It installs as valkyrie, with the short alias valk:

   uv tool install valkyrie-cli
   

   If you installed the CLI from GitHub before, remove that install first with uv tool uninstall valkyrie.

2. Sign in:

   valkyrie login
   

3. Connect Modal by following Set up Modal. It's set once per organization, so skip it if someone in yours already did.

4. Install Claude Code from the public agent registry, and store your Anthropic API key at the prompt:

   valkyrie agent install \
     https://github.com/vals-ai/public-agent-registry/tree/main/examples/claude_code-v1.0.0
   
   valkyrie secret set anthropic-api-key
   

5. Install Terminal-Bench 2 from Harbor as the dataset tb/tb2:

   valkyrie dataset install tb/tb2 terminal-bench/terminal-bench-2 --location harbor
   

6. Run one task:

   valkyrie run start --agent claude_code-v1.0.0 --dataset tb/tb2 \
     --model anthropic/claude-sonnet-4-6 --provider modal \
     -s ANTHROPIC_API_KEY anthropic-api-key --slice 0:1 --concurrency 1
   

7. Watch it with valkyrie tui.

The quickstart guide walks through each step and how to read the result.

Browse runs in the terminal

valkyrie tui lists your organization's runs with live status, so everyone signed in to the same organization sees the same runs. Open one to see its tasks, errors first, next to the highlighted task's log.

• / filters the list, e groups a run's errors by message, and s picks how far back the list goes.
• y copies the matching valkyrie command, to run yourself or hand to an agent.
• valkyrie tui --fake opens it on sample runs, with no account needed.

Where to go next

Citation

If you use Valkyrie in your research, cite the paper:

@inproceedings{forzano2026valkyrie,
  author    = {Forzano, Jarett and Almatov, Omar and Nashold, Langston and Ravi, Nikil and Kassian, Orestes},
  title     = {Valkyrie: A Microservice-Based Framework for Scalable Evaluation of AI Agents},
  booktitle = {Proceedings of the 1st ACM Conference on Agentic and AI Systems},
  series    = {CAIS '26},
  year      = {2026},
  month     = {may},
  address   = {San Jose, CA, USA},
  publisher = {ACM},
  location  = {New York, NY, USA},
  numpages  = {5},
  doi       = {10.1145/3786335.3813231},
  url       = {https://doi.org/10.1145/3786335.3813231}
}