Thabit: evaluate multiple LLMs on your data
Project description
Thabit
Evaluate multiple LLM models with the same data to determine which one is better for your use case.
How to run
pip3 install thabit
Test
pytest tests
Build
pip3 install -e .
Contribute
Docs
TODO:
-
More logs.
-
Validate the input dataset.
-
Util folder for Validating Dataset, versioning datasets.
-
UI for adding a dataset.
-
UI for adding/editing config.
-
Visulaise Output (using UI).
-
Run eval per dataset (add folders for dataset and for evals). This is to simplify visualising results later using the UI.
root ├── datasets │ └── a └── evals └── a
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
thabit-0.1.5.tar.gz
(17.7 kB
view hashes)
Built Distribution
thabit-0.1.5-py3-none-any.whl
(21.3 kB
view hashes)