ai-yardstick 0.0.1

A CLI tool for running and managing LLM evaluations

ai-yardstick

A CLI tool for running and managing LLM evaluations

See my own evals

Installation

Install this tool using pip:

pip install ai-yardstick

Usage

Create a new eval with:

ai-yardstick create EVAL_NAME

Then edit prompts.csv, models.csv and tests.csv with your test cases.

Run an eval with:

ai-yardstick run path/to/config.yaml

For help, run:

ai-yardstick --help

You can also use:

python -m llm_evals_cli --help

Development

To contribute to this tool, first checkout the code. Then create a new virtual environment:

cd ai-yardstick
python -m venv venv
source venv/bin/activate

Now install the dependencies and test dependencies:

pip install -e '.[test]'

To run the tests:

python -m pytest