Project description

Media Profile Summaries Evaluation

pytest style

Automated evaluation of media profile summaries. Datasets are expected to include the predicted summary, the input evidence (prompt), and the expected summary.

Setup

poetry install --with dev

Usage

poetry run python scripts/evaluate.py

You must have access to an instance of bowler hat.
See bowlerhat section in config and bowlerhat.Client for more information
evaluate.py is configured with hydra (python evaluate.py --help for config options)
Datasets are loaded from file or hugging face hub via datasets.load_dataset

Project details

These details have not been verified by PyPI

View statistics for this project via Libraries.io, or by using our public dataset on Google BigQuery

Release history Release notifications | RSS feed

This version

0.1.1

Sep 19, 2023

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

media_profile_summaries_evaluation-0.1.1.tar.gz (5.8 kB view hashes)

Uploaded Sep 19, 2023 Source

Built Distribution

media_profile_summaries_evaluation-0.1.1-py3-none-any.whl (7.1 kB view hashes)

Uploaded Sep 19, 2023 Python 3

Hashes for media_profile_summaries_evaluation-0.1.1.tar.gz

Hashes for media_profile_summaries_evaluation-0.1.1.tar.gz
Algorithm	Hash digest
SHA256	`9e1f81918d05cb806a0076cad8274fadbb7a5a37f89cf717fabfc669f5f9803c`
MD5	`ef4b1b3d54d978ca00868149ea1429f0`
BLAKE2b-256	`cc8305b160ffe3dcd860f7bb903bddace85151ab1ce0d98e7a6cf4287bf65c15`

Hashes for media_profile_summaries_evaluation-0.1.1-py3-none-any.whl

Hashes for media_profile_summaries_evaluation-0.1.1-py3-none-any.whl
Algorithm	Hash digest
SHA256	`71ebf1bc0de0952e42a989c4f4127a5b1eb27479feca218e04be821b3374ff0d`
MD5	`4a3e31a22f9e2ce0d7d8bea395a4a136`
BLAKE2b-256	`34257d6be2d47409e21b0a0e535368fa61947e04c1f8f5e0f47f6e29d125c37a`