lm-pub-quiz 0.3.1

Evaluate language models using multiple choice items

LM Pub Quiz

Evaluate language models using multiple choice items

---

This library implements a knoweledge probing approach which uses LM's inherent ability to estimate the log-likelihood of any given textual statement. For more information visit the LM Pub Quiz website.

See also

• Website
• Documentation
• BEAR Dataset
• Paper

Getting started

This short guide should get you started. For more detailed information visit the documentation.

Installing the Package

You can install the package via pip:

pip install lm-pub-quiz

For alternatives methods of installing the package, visit the documentation.

Example Usage

from lm_pub_quiz import Dataset, Evaluator

# Load the dataset
dataset = Dataset.from_name("BEAR")

# Load the model
evaluator = Evaluator.from_model(
    "gpt2",
    model_type="CLM",
)

# Run the evaluation and save the
results = evaluator.evaluate_dataset(
    dataset,
    template_index=0,
    save_path="gpt2_results",
    batch_size=32,
)

# If the results are analyzed in a different session, they can be loaded from the file system
# results = DatasetResults.from_path("gpt2_results")

print("=== Overall score ===")
print(results.get_metrics("accuracy"))

Contributing

We welcome any questions, comments, or even PRs to this project to improve the package.

We use hatch to manage this project. For the most comfortable development experience, please first install hatch using pip or pipx.

Then, to propose a change to the library,

• test your code locally using hatch run all:test
• format the code according to our formatting guidelines using hatch run lint:fmt,
• check type- and style-consistency using hatch run lint:all, and
• finally create a pull request describing the changes you propose.

For work on the documentation, use hatch run serve-docs to run a local documentation server.