anyeval

Universal Evaluation for Gen AI

Usage

pip install anyeval

# run evaluation for a parquet file or directory.
anyeval run [parquet_file|dir]

Design

Data Schema (Parquet)

Columns

id: is a UUID column. dataset: is a String column. label: is a JSON column. key value must be String.
input: is a JSON column. key must be String. can include media data.
output: is a JSON column. key must be String. can include media data.

media data key must start by '@[file_type]->key'
only @image->, @video->, @audio-> file types are supported.

Example:

{
  "prompt": "make the image to video",
  "@image->input_image": "fs://path/to/image.jpg"
}

{
  "inference_time": "1.234",
  "inference_device": "NVIDIA GeForce RTX 4090",
  "@video->output_video": "fs://path/to/output.mp4",
  "@video->output_video_thumbnail": "s3://path/to/output.jpg"
}

created_at: is a timestamp column.
updated_at: is a timestamp column.