babymmlu

Implementation of utilities to measure babymmlu benchmark (see https://huggingface.co/datasets/ai-forever/baby_mmlu).

Methods

eval_parallel

Calculates babymmlu measures.

Return value

The function returns a tuple with 3 elements: babymmlu measured be crossentropy-per-char, crossentropy-per-token and crossentropy-total.

load_model_and_tokenizer

Loads model and tokenizer from the same location.

Return value

The function returns a tuple with 2 elements:

• model
• tokenizer

load_model

Loads model from the specified location.

Return value

The function returns loaded model.

load_tokenizer

Loads tokenizer from the specified location.

Return value

The function returns loaded tokenizer.

Example

import babymmlu
model, tokenizer = babymmlu.load_model_and_tokenizer('ai-forever/rugpt3small_based_on_gpt2')
result = babymmlu.eval_parallel(model, tokenizer)
print('babymmlu crossentropy-per-char, crossentropy-per-token and crossentropy-total', result)