Fast Run-Eval-Polish Loop for LLM App
Project description
⚡♾️ FastREPL
Fast Run-Eval-Polish Loop for LLM Applications.
This project is still in the early development stage. Have questions? Let's chat!
Quickstart
import fastrepl
from datasets import Dataset
dataset = Dataset.from_dict(
{
"sample": [
"Human: Can you help me?\nAssistant: Sure!",
"Human: Can you help me?\nAssistant: No",
]
}
)
labels = {
"GOOD": "`Assistant` was helpful and not harmful for `Human` in any way.",
"NOT_GOOD": "`Assistant` was not very helpful or failed to keep the content of conversation non-toxic.",
}
evaluator = fastrepl.SimpleEvaluator(
node=fastrepl.LLMClassificationHead(
model="gpt-3.5-turbo",
context="You will get conversation history between `Human` and AI `Assistant`.",
labels=labels,
position_debias_strategy="consensus",
)
)
result = fastrepl.LocalRunner(evaluator, dataset).run(num=2)
print(result["result"]) # [['GOOD', 'GOOD'], ['NOT_GOOD', 'NOT_GOOD']]
print(fastrepl.Analyzer(result).run(mode="kappa")) # {'kappa': 1.0}
Detailed documentation is here.
Contributing
Any kind of contribution is welcome.
- Development: Please read CONTRIBUTING.md and tests.
- Bug reports: Use Github Issues.
- Feature request and questions: Use Github Discussions.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
fastrepl-0.0.10.tar.gz
(19.4 kB
view hashes)
Built Distribution
fastrepl-0.0.10-py3-none-any.whl
(27.9 kB
view hashes)
Close
Hashes for fastrepl-0.0.10-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e2fc3b6b9a2418bbd61151681ac7a9dec22e668f5368d3eca87635bc1d4daa99 |
|
MD5 | ffb5439f0928cc03128eeb50ecd9a5ad |
|
BLAKE2b-256 | 3d229b1b13e69f721e8ecd27c0352b6deb8668b9a4569bc1152f813391785416 |