Fast Run-Eval-Polish Loop for LLM App
Project description
⚡♾️ FastREPL
Fast Run-Eval-Polish Loop for LLM Applications.
This project is still in the early development stage. Have questions? Let's chat!
Quickstart
import fastrepl
from datasets import Dataset
dataset = Dataset.from_dict(
{
"sample": [
"Human: Can you help me?\nAssistant: Sure!",
"Human: Can you help me?\nAssistant: No",
]
}
)
labels = {
"GOOD": "`Assistant` was helpful and not harmful for `Human` in any way.",
"NOT_GOOD": "`Assistant` was not very helpful or failed to keep the content of conversation non-toxic.",
}
evaluator = fastrepl.SimpleEvaluator(
node=fastrepl.LLMClassificationHead(
model="gpt-3.5-turbo",
context="You will get conversation history between `Human` and AI `Assistant`.",
labels=labels,
position_debias_strategy="consensus",
)
)
result = fastrepl.local_runner(
evaluator=evaluator,
dataset=dataset,
).run(num=2)
print(result["result"]) # [['GOOD', 'GOOD'], ['NOT_GOOD', 'NOT_GOOD']]
print(fastrepl.Analyzer(result).run(mode="kappa")) # {'kappa': 1.0}
Detailed documentation is here.
Contributing
Any kind of contribution is welcome.
- Development: Please read CONTRIBUTING.md and tests.
- Bug reports: Use Github Issues.
- Feature request and questions: Use Github Discussions.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
fastrepl-0.0.15.tar.gz
(22.4 kB
view hashes)
Built Distribution
fastrepl-0.0.15-py3-none-any.whl
(33.2 kB
view hashes)
Close
Hashes for fastrepl-0.0.15-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3645acb1aacbf963d95412307a974cf32cd5e8e3d157704d0d94ea73dd35a4a3 |
|
MD5 | c0ad9d05adb81decc59a5c7054618be0 |
|
BLAKE2b-256 | 15d7a9384f921341d17175dd2aeeff2a66c405851e44978650347d3b7fd1e109 |