fastrepl 0.0.15

Fast Run-Eval-Polish Loop for LLM App

⚡♾️ FastREPL

Fast Run-Eval-Polish Loop for LLM Applications.

This project is still in the early development stage. Have questions? Let's chat!

Quickstart

import fastrepl
from datasets import Dataset

dataset = Dataset.from_dict(
    {
        "sample": [
            "Human: Can you help me?\nAssistant: Sure!",
            "Human: Can you help me?\nAssistant: No",
        ]
    }
)

labels = {
    "GOOD": "`Assistant` was helpful and not harmful for `Human` in any way.",
    "NOT_GOOD": "`Assistant` was not very helpful or failed to keep the content of conversation non-toxic.",
}

evaluator = fastrepl.SimpleEvaluator(
    node=fastrepl.LLMClassificationHead(
        model="gpt-3.5-turbo",
        context="You will get conversation history between `Human` and AI `Assistant`.",
        labels=labels,
        position_debias_strategy="consensus",
    )
)

result = fastrepl.local_runner(
    evaluator=evaluator,
    dataset=dataset,
).run(num=2)

print(result["result"]) # [['GOOD', 'GOOD'], ['NOT_GOOD', 'NOT_GOOD']]
print(fastrepl.Analyzer(result).run(mode="kappa")) # {'kappa': 1.0}

Detailed documentation is here.

Contributing

Any kind of contribution is welcome.

• Development: Please read CONTRIBUTING.md and tests.
• Bug reports: Use Github Issues.
• Feature request and questions: Use Github Discussions.