Valkyrie

Valkyrie orchestrates scalable, reproducible evaluations for AI agents. Run it on infrastructure that Vals AI hosts, or on your own machine with a local Tracker.

Guides, the CLI reference and the Python SDK reference are published at docs.valkyrie.vals.ai.

Quickstart

Note: Valkyrie can be invoked using either valkyrie or the alias valk. For example: valkyrie run start or valk run start.

Install the CLI:

uv tool install --exclude-newer "7 days" --exclude-newer-package certifi=false --exclude-newer-package valkyrie-cli=false --exclude-newer-package valkyrie-sdk=false valkyrie-cli

If you installed the CLI from GitHub before, remove that install first:

uv tool uninstall valkyrie

Sign in with your Vals account:

valkyrie login

Install an agent from the public agent registry:

valkyrie agent install https://github.com/vals-ai/public-agent-registry/tree/main/agents/mini_swe_agent

Store your model provider's API key as a secret. The CLI prompts for the value:

valkyrie secret set anthropic-key

Start a benchmark run, passing the secret to the agent as ANTHROPIC_API_KEY with -s:

valkyrie run start \
  --agent mini_swe_agent \
  --benchmark swebench \
  --model anthropic/claude-sonnet-4-6 \
  -s ANTHROPIC_API_KEY anthropic-key \
  -k config benchmarks/swebench.yaml \
  --slice "0:10" \
  --connect

Where to go next

Citation

If you use Valkyrie in your research, cite the paper:

@inproceedings{forzano2026valkyrie,
  author    = {Forzano, Jarett and Almatov, Omar and Nashold, Langston and Ravi, Nikil and Kassian, Orestes},
  title     = {Valkyrie: A Microservice-Based Framework for Scalable Evaluation of AI Agents},
  booktitle = {Proceedings of the 1st ACM Conference on Agentic and AI Systems},
  series    = {CAIS '26},
  year      = {2026},
  month     = {may},
  address   = {San Jose, CA, USA},
  publisher = {ACM},
  location  = {New York, NY, USA},
  numpages  = {5},
  doi       = {10.1145/3786335.3813231},
  url       = {https://doi.org/10.1145/3786335.3813231}
}