redlite 0.0.35

LLM testing on steroids

RedLite

An opinionated toolset for testing Conversational Language Models.

Documentation

https://innodatalabs.github.io/redlite/

Usage

1. Install required dependencies

   pip install redlite[all]
   

2. Generate several runs (using Python scripting, see examples, and below)

3. Review and compare runs

   redlite server --port <PORT>
   

4. Optionally, upload to Zeno

   ZENO_API_KEY=zen_XXXX redlite upload
   

Python API

import os
from redlite import run, load_dataset
from redlite.model.openai_model import OpenAIModel
from redlite.metric import PrefixMetric


model = OpenAIModel(api_key=os.environ["OPENAI_API_KEY"])
dataset = load_dataset("hf:innodatalabs/rt-gaia")
metric = PrefixMetric(ignore_case=True, ignore_punct=True, strip=True)

run(model=model, dataset=dataset, metric=metric)

Goals

• simple, easy-to-learn API
• lightweight
• only necessary dependencies
• framework-agnostic (PyTorch, Tensorflow, Keras, Flax, Jax)
• basic analytic tools included

Develop

python -m venv .venv
. .venv/bin/activate
pip install -e .[dev,all]

Make commands:

• test
• test-server
• lint
• wheel
• docs
• black

Zeno <zenoml.com> integration

Benchmarks can be uploaded to Zeno interactive AI evaluation platform <hub.zenoml.com>:

redlite upload --project my-cool-project

All tasks will be concatenated and uploaded as a single dataset, with extra fields:

• task_id
• dataset
• metric

All models will be uploaded. If model was not tested on a specific task, a simulated zero-score dataframe is used instead.

Use task_id (or dataset as appropriate) to create task slices. Slices can be used to navigate data or create charts.

TODO

• deps cleanup (randomname!)
• review/improve module structure
• automate CI/CD
• write docs
• publish docs automatically (CI/CD)
• web UI styling
• better test server
• tests
• Integrate HF models
• Integrate OpenAI models
• Integrate Anthropic models
• Integrate vLLM models
• Fix data format in HF datasets (innodatalabs/rt-* ones) to match standard
• more robust backend API (future-proof)
• better error handling for missing deps
• document which deps we need when
• export to CSV
• Upload to Zeno