lm-pub-quiz 0.2.1

Evaluate language models using multiple choice items

LM Pub Quiz

Evaluate language models using multiple choice items

---

This library implements a knoweledge probing approach which uses LM's inherent ability to estimate the log-likelihood of any given textual statement. For more information visit the LM Pub Quiz website.

See also

• Website
• Documentation
• BEAR Dataset
• Paper

Getting started

This short guide should get you started. For more detailed information visit the documentation.

Installing the Package

You can install the package via pip:

pip install lm-pub-quiz

or clone the repository and install the package using the -e flag to make changes to the source code:

pip install -e lm-pub-quiz  # Modify the path to the repository if necessary

For alternatives methods of installing the package, visit the documentation.

Example Usage

from lm_pub_quiz import Dataset, Evaluator

dataset_path = "<BEAR data path, e.g. ./transformer-knowledge-probe/data/BEAR>"
result_save_path = "<BEAR results save path>"
model_name = "gpt2"

# Load the BEAR dataset from its specific location
dataset = Dataset.from_path(dataset_path)

# Run the BEAR evaluator and save the results
evaluator = Evaluator.from_model(model_name, model_type="CLM", device="cuda")
results = evaluator.evaluate_dataset(dataset, save_path=result_save_path, batch_size=32)

Contributing

We welcome any questions, comments, or event PRs to this project to improve the package.

We use hatch to manage this project. To run the test cases, run hatch run test or hatch run all:test (to test on multiple python versions). In order to check the formatting and correct typing, run hatch run lint:all.