Scorebook

A Python library for Model evaluation

Scorebook provides a flexible and extensible framework for evaluating models such as large language models (LLMs). Easily evaluate any model using evaluation datasets from Hugging Face such as MMLU-Pro, HellaSwag, and CommonSenseQA, or with data from any other source. Evaluations calculate scores for any number of specified metrics such as accuracy, precision, and recall, as well as any custom defined metrics, including LLM as a judge (LLMaJ).

Use Cases

Scorebook's evaluations can be used for:

• Model Benchmarking: Compare different models on standard datasets.
• Model Optimization: Find optimal model configurations.
• Iterative Experimentation: Reproducible evaluation workflows.

Key Features

• Model Agnostic: Evaluate any model, running locally or deployed on the cloud.
• Dataset Agnostic: Create evaluation datasets from Hugging Face datasets or any other source.
• Extensible Metric Engine: Use the Scorebook's built-in or implement your own.
• Hyperparameter Sweeping: Evaluate over multiple model hyperparameter configurations.
• Adaptive Evaluations: Run Trismik's ultra-fast adaptive evaluations.
• Trismik Integration: Upload evaluations to Trismik's platform.

Installation

pip install scorebook

Scoring Models Output

Scorebooks score function can be used to evaluate pre-generated model outputs.

Score Example

from scorebook import score
from scorebook.metrics import Accuracy

# 1. Prepare a list of generated model outputs and labels
model_predictions = [
    {"input": "What is 2 + 2?", "output": "4", "label": "4"},
    {"input": "What is the capital of France?", "output": "London", "label": "Paris"},
    {"input": "Who wrote Romeo and Juliette?", "output": "William Shakespeare", "label": "William Shakespeare"},
    {"input": "What is the chemical symbol for gold?", "output": "Au", "label": "Au"},
]

# 2. Score the model's predictions against labels using metrics
results = score(
    items = model_predictions,
    metrics = Accuracy,
)

Score Results:

{
    "aggregate_results": [
        {
            "dataset": "scored_items",
            "accuracy": 0.75
        }
    ],
    "item_results": [
        {
            "id": 0,
            "dataset": "scored_items",
            "input": "What is 2 + 2?",
            "output": "4",
            "label": "4",
            "accuracy": true
        }
        // ... additional items
    ]
}

Classical Evaluations

Running a classical evaluation in Scorebook executes model inference on every item in the dataset, then scores the generated outputs using the dataset’s specified metrics to quantify model performance.

Classical Evaluation example:

from scorebook import evaluate, EvalDataset
from scorebook.metrics import Accuracy

# 1. Create an evaluation dataset
evaluation_items = [
    {"question": "What is 2 + 2?", "answer": "4"},
    {"question": "What is the capital of France?", "answer": "Paris"},
    {"question": "Who wrote Romeo and Juliet?", "answer": "William Shakespeare"}
]

evaluation_dataset = EvalDataset.from_list(
    name = "basic_questions",
    items = evaluation_items,
    input = "question",
    label = "answer",
    metrics = Accuracy,
)

# 2. Define an inference function - This is a pseudocode example
def inference_function(inputs: List[Any], **hyperparameters):

    # Create or call a model
    model = Model()
    model.temperature = hyperparameters.get("temperature")

    # Call model inference
    model_outputs = model(inputs)

    # Return outputs
    return model_outputs

# 3. Run evaluation
evaluation_results = evaluate(
    inference_function,
    evaluation_dataset,
    hyperparameters = {"temperature": 0.7}
)

Evaluation Results:

{
    "aggregate_results": [
        {
            "dataset": "basic_questions",
            "temperature": 0.7,
            "accuracy": 1.0,
            "run_completed": true
        }
    ],
    "item_results": [
        {
            "id": 0,
            "dataset": "basic_questions",
            "input": "What is 2 + 2?",
            "output": "4",
            "label": "4",
            "temperature": 0.7,
            "accuracy": true
        }
        // ... additional items
    ]
}

Adaptive Evaluations with evaluate

To run an adaptive evaluation, use a Trismik adaptive dataset The CAT algorithm dynamically selects items to estimate the model’s ability (θ) with minimal standard error and fewest questions.

Adaptive Evaluation Example

from scorebook import evaluate, login

# 1. Log in with your Trismik API key
login("TRISMIK_API_KEY")

# 2. Define an inference function
def inference_function(inputs: List[Any], **hyperparameters):

    # Create or call a model
    model = Model()

    # Call model inference
    outputs = model(inputs)

    # Return outputs
    return outputs

# 3. Run an adaptive evaluation
results = evaluate(
    inference_function,
    datasets = "trismik/headQA:adaptive",    # Adaptive datasets have the ":adaptive" suffix
    project_id = "TRISMIK_PROJECT_ID",       # Required: Create a project on your Trismik dashboard
    experiment_id = "TRISMIK_EXPERIMENT_ID", # Optional: An identifier to upload this run under
)

Adaptive Evaluation Results

{
    "aggregate_results": [
        {
            "dataset": "trismik/headQA:adaptive",
            "experiment_id": "TRISMIK_EXPERIMENT_ID",
            "project_id": "TRISMIK_PROJECT_ID",
            "run_id": "RUN_ID",
            "score": {
                "theta": 1.2,
                "std_error": 0.20
            },
            "responses": null
        }
    ],
    "item_results": []
}

Metrics

Tutorials

For local more detailed and runnable examples:

pip install scorebook[examples]

The tutorials/ directory contains comprehensive tutorials as notebooks and code examples:

• tutorials/notebooks: Interactive Jupyter Notebooks showcasing Scorebook's capabilities.
• tutorials/examples: Runnable Python examples incrementally implementing Scorebook's features.

Run a notebook:

jupyter notebook tutorials/notebooks

Run an example:

python3 tutorials/examples/1-score/1-scoring_model_accuracy.py

Contributing

We welcome contributions! Please see CONTRIBUTING.md for guidelines.

License

This project is licensed under the MIT License - see the LICENSE file for details.

About

Scorebook is developed by Trismik to simplify and speed up your LLM evaluations.