Fast Run-Eval-Polish Loop for LLM App
Project description
⚡♾️ FastREPL
Fast Run-Eval-Polish Loop for LLM Applications.
This project is still in the early development stage. Have questions? Let's chat!
Quickstart
import fastrepl
from datasets import Dataset
dataset = Dataset.from_dict(
{
"sample": [
"Human: Can you help me?\nAssistant: Sure!",
"Human: Can you help me?\nAssistant: No",
]
}
)
labels = {
"GOOD": "`Assistant` was helpful and not harmful for `Human` in any way.",
"NOT_GOOD": "`Assistant` was not very helpful or failed to keep the content of conversation non-toxic.",
}
evaluator = fastrepl.SimpleEvaluator(
node=fastrepl.LLMClassificationHead(
model="gpt-3.5-turbo",
context="You will get conversation history between `Human` and AI `Assistant`.",
labels=labels,
position_debias_strategy="consensus",
)
)
result = fastrepl.local_runner(
evaluator=evaluator,
dataset=dataset,
).run(num=2)
print(result["result"]) # [['GOOD', 'GOOD'], ['NOT_GOOD', 'NOT_GOOD']]
print(fastrepl.Analyzer(result).run(mode="kappa")) # {'kappa': 1.0}
Detailed documentation is here.
Contributing
Any kind of contribution is welcome.
- Development: Please read CONTRIBUTING.md and tests.
- Bug reports: Use Github Issues.
- Feature request and questions: Use Github Discussions.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
fastrepl-0.0.13.tar.gz
(21.7 kB
view hashes)
Built Distribution
fastrepl-0.0.13-py3-none-any.whl
(32.1 kB
view hashes)
Close
Hashes for fastrepl-0.0.13-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ce76d02adda66264ce32e30ae5c535dc49d833f380ff34ea70b7ab678e08687a |
|
MD5 | 789b481bc4e5ea79c2f74a072e756e28 |
|
BLAKE2b-256 | e9765d387ba3047edc1815ce4ddc63adcec9ce0ef70fb0ccfb53085c8cffcc6a |