llmtask 0.0.2

LLM Downstream Task Evaluation Tool.

LLM Task

Introduction

Quick Start

pip install llmtask

import random

from llmtask import TaskGenerator

choices = ("A", "B", "C", "D")

TG = TaskGenerator("mmlu", max_shot=4)

for task in TG:
    TG.feedback(random.choice(choices))

print(TG.summary())