aicard-eval 0.1.0

Evaluation module for aicard.

AICard-Eval

This is a package created under the AI-CODE and it's part of the transparency services for AI model cards. Its purpose is to provide a single tool for evaluating AI models. The output is standardized and ment (but not restricted) to be used as an import for aicard package.

Notice: This is an alpha version. Suported cases are text and image binary, multiclass, and multilabel classifications

⚡ Quickstart

To install use:

conda create -n aicard-eval python=3.11
conta activate aicard-eval
pip install -e .

Follow the script bellow. The aicard-eval will choose the correct metrics corresponding to your case. For more examples see the examples/ folder.

You can use datasets and models from service providers e.g. huggingface or you can use your local models and datasets. Supported datasets types are: .csv, .tsv, .json, .jsonl, .xml, .yml, .yaml, .parquet, .feather, .pickle and supported image types are .jpg, .jpeg, .png, .gif, .bmp, .tiff, .tif

import aicard_eval
from datasets import load_dataset
from transformers import pipeline

# 1) Load your model
classifier = pipeline(task="text-classification", model="SamLowe/roberta-base-go_emotions", top_k=None)

# 2) Load your dataset
dataset = load_dataset("google-research-datasets/go_emotions", split='test')
class_names = dataset.features["labels"].feature.names


# 3) Define a function to handle the dataset
def pipeline(data):
        sentences = [text for text in data['text']]
        model_outputs = classifier(sentences)
        out = []
        for sample in model_outputs:
            flat = {d['label']: d['score'] for d in sample}
            out.append([flat[name] for name in class_names])
        return out

# 4) call the aicard-eval evaluation function
metrics = aicard_eval.evaluate(
    data=dataset,
    pipeline=pipeline,
    task=aicard_eval.tasks.vision.image_classification,
    batch_size=32)

print(metrics)
# {'package version': '0.1.0', 
# 'datetime': '2025-Nov-14 14:54', 
# 'task': 'Text Classification', 
# 'metrics': {
#     'precision_macro': 0.5090416420534856, 
#     'precision_micro': 0.5741662060070021, 
#     'recall_macro': 0.46497245851260965, 
#     'recall_micro': 0.5741662060070021, 
#     'top1_acc_micro': 0.5741662060070021, 
#     'top1_acc_macro': 0.5741662060070021, 
#     'top1_acc_weighted': 0.5741662060070021, 
#     'f1_macro': 0.4661938061623436, 
#     'f1_micro': 0.5741662060070021, 
#     'auc_roc_macro': 0.9286682043487104, 
#     'auc_roc_weighted': 0.9099991445506153}, 
# 'batch_size': 32, 
# 'hardware': 'CPU: AMD Ryzen 7 7800X3D 8-Core Processor, RAM: 15.62 GB, CUDA: | NVIDIA-SMI 580.102.01 Driver Version: 581.57  CUDA Version: 13.0|', 
# 'execution_time': 'inference: 34.68s, metrics: 45.76ms', 
# 'num_classes': 28}