thabit 0.2.4

Thabit: evaluate multiple LLMs on your data

Thabit

Evaluate multiple LLM models with the same data to determine which one is better for your use case.

How to run

pip3 install thabit

Test

pytest tests

Build

pip3 install -e .

Contribute

Docs

TODO:

• Validate the input dataset.

• UI for adding/editing config.

• Visulaise Output (using UI).

• Run eval per dataset (add folders for dataset and for evals). This is to simplify visualising results later using the UI.

  root
  ├── datasets
  │ └── a
  └── evals
    └── a