Nasute

A self-hosted eval harness for RAG (retrieval-augmented generation: a system that looks things up and then writes an answer) and for language models. It asks a system questions with known answers, has a second AI mark the answers claim by claim, counts, and keeps every run. It tells you whether a change made things better, worse or no different, with a 95% range on every number. Named after the nasute, the soldier termite that guards the colony.

Plain Python 3.11 or newer, standard library only. Any system you can ask a question over HTTP is a target, and anything else takes one Python function.

The demo report on a phone: the front page, and a run page with its outcomes at a glance.

Quick start

Install. Nasute uses only Python's standard library, so pipx is the way to install it; there is no Docker image, and none is needed.

pipx install nasute                     # or, until it is on PyPI: pipx install git+https://github.com/bobbyely/nasute

Or run it from a clone, with no install. Then type python3 -m nasute wherever this page says nasute:

git clone https://github.com/bobbyely/nasute && cd nasute
export PYTHONPATH="$PWD/src"            # lets python3 find the code from any folder
python3 -m nasute --help

Changing the code? Install it editable in a virtual environment instead (python3 -m venv .venv && .venv/bin/pip install -e .); see CONTRIBUTING.md.

Try the demo, with no account and no model. In one terminal start the stand-in system and judge, and in another make a folder set up for it and run:

nasute demo-server                      # leave it running
nasute init try --demo && cd try        # in a second terminal
nasute run --profile smoke

Each question gets a line as it is marked, such as [2/3] water-boils ... right (0.0 s). The run ends with its scores, each with a 95% range, and a link to its page in the report.

Point it at your own model. Run nasute init mytest (without --demo) and edit [targets.mine] and [judge] in mytest/nasute.toml. Ready-made settings: a model behind an OpenAI-compatible API and a model run by Ollama. Then nasute run --profile smoke from that folder.

Open the report. Open report/index.html in a browser, or run nasute serve and visit http://127.0.0.1:8765/. The front page shows each dataset with its systems ranked; click through to a dataset and then to a run, or tick runs to compare them. To have the printed link use your own web address, set report_url (see settings).

Progress prints as short lines at a terminal and as JSON lines when the output goes to a file or a pipe, for a program that reads logs; --log text or --log json picks one (commands).

The full walk-through is the Quick start.

How do I…

• test my own endpoint or model?
• write questions, and change them safely? Or import them from a spreadsheet.
• tag questions? Tags group the scores in the report, by topic and by difficulty.
• compare two models, or two runs?
• mark a baseline, or label a run?
• see why a question failed?
• take the results into a spreadsheet?
• use a hosted model as the judge?
• check the judge marks fairly?
• run a long test overnight?

Documentation

All of it is in docs/: a Quick start, How Nasute works, how-to guides, a reference and explanations.

This repo

Contributing and security

See CONTRIBUTING.md. To report a security problem, see SECURITY.md.

Licence

MIT (see LICENSE). The vendored chart libraries keep their own MIT licences, in src/nasute/assets/.