aicard 0.3.3

Compute and organize model cards locally or online.

AI Card

This SDK contains a collection of methods to create, manage, and edit AI model cards. Cards can be stored either locally or in an online service (that can also be self-hosted). We finally provide methods to populate fields by analyzing datasets by computing well-known measures or calling AI assistants.

Alpha version - Current apis and functionalities are unstable.

⚡ Quickstart

INstall aicard in a virtual environment like below. If you are a developer working on this repository, clone it and install it locally per pip install -e package instead.

python -m venv .venv
source .venv/bin/activate
pip install aicard

Create your first model card like below. You can also visit our public service or self-host a copy of your own to maintain a database of cards and modify fields through a UI. The aicard library makes that hosting possible, allows programmatic management of cards, and passes data to the service from your local environment. Those data comprise mainly quantitative evaluation.

# demo.py
import aicard as aic

card = aic.ModelCard()
card.title = "Model Card"
card.model.name = "Llama"
card.model.overview = "This is a model overview. Freely add <b>html</b> or *markdown*."
card.model.version = "3.2"
card.considerations.use_case = "text generation"

print(card)

> python demo.py
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃                                  Model Card                                  ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
completion        🧩🧩⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️                                        

                                     model                                      
name               Llama                                                        
overview           This is a model overview. Freely add html or markdown.       
version            3.2                                                       

                                 considerations                                 
use case           text generation   

🧠 Assistants

There are broadly two types of assistants for model evaluation: evaluation ones that can quantitatively evaluate your models based on popular measures, and LLM-based ones that fill or enrich qualitative fields of your cards. The last type can also run through services too.

Perform manual numerical assessment across a wide variety of available tasks holding popular evaluation methodologies and measures. If you need customization, you can create your own tasks too.

# experiment.py
from datasets import load_dataset
from huggingface_hub import dataset_info
from transformers import pipeline
import aicard as aic

dataset = load_dataset("google-research-datasets/go_emotions", split = 'test')
info = dataset_info("google-research-datasets/go_emotions")
class_names = info.card_data['dataset_info'][1]['features'][1]['sequence']['class_label']['names']
classifier = pipeline(task="text-classification", model="SamLowe/roberta-base-go_emotions", top_k=None)

def pipeline(data):
    sentences = [text for text in data['text']]
    model_outputs = classifier(sentences)
    out = []
    for sample in model_outputs:
        flat = {d['label']: d['score'] for d in sample}
        out.append([flat[name] for name in class_names.values()])
    return out

metrics = aic.evaluation.evaluate(
    data=dataset,
    pipeline=pipeline,
    task=aic.evaluation.tasks.nlp.text_classification,
    batch_size=32,
    as_card=True
)

print(metrics)

> python experiment.py
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃                         Text Classification Results                          ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
completion        🧩⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️                                         

                                    analysis                                    
analysis           Evaluation was conducted at 2025-08-19 for text              
classification with 32 batch size. A pipeline function runs the model.          
metrics            The following metrics were computed at 2025-08-19:           
- precision_macro: 0.575                                                        
- precision_micro: 0.685                                                        
- recall_macro: 0.396                                                           
- recall_micro: 0.511                                                           
- f1_macro: 0.450                                                               
- f1_micro: 0.586                                                               
- auc_roc_macro: 0.929   

The default as_card=True creates a model card that is partially filled with analysis data automatically obtained from the evaluation process. Otherwise, computed metrics are returned as a dictionary of (name, value) pairs. The default case's card can be merged into an existing card. Merge commands accept an argument to signify whether non-empty fields of the base card should be replaced or (if not) have the new card's contents be appended.

card.merge(metrics, replace=True)

You have the option to collaborate with LLM assistants too! These let you fill in qualitative aspects of the model card from a software's repository. In the simplest case, the assistants will be owned by you. Buf, if you want persistent storage of your cards, you can collaborate with the public or self-hosted service too (see below). The assistant will create a copy of the model card with changes applied.

card = card.assistant(
    repository="myproject/", # your model's git repository or working directory here
    model=mc.assistants.olama,
    dotenv=".env", # assistant configuration file
)

The AI assistant can also create a simplified version of your model card that is friendlier to laypeople to read and parse through.

card = card.gist(
    model=aic.assistants.olama,
    dotenv=".env" # assistant configuration file
)

📡 Connecting to a server

You can link to a public or self-hosted server for persistent card storage. Here we will link to the public server, with can be found at URL (to be decided). To work with a server you need to log in first and create a card.

# the credentials bellow are the default for self-hosted servers
conn = aic.connect("URL").login(username="admin", password="admin")  
card = conn.create()  # no argument for a brand new card

Alternatively, connect with a dotenv file holding those fields:

conn = aic.connect("URL").login(env=".env")  
card = conn.create()

# .env
USER=admin
PASS=admin

You can also search for cards. Cards that you do not own do not come with any connection attached, and therefore cannot be used as contexts. To simplify usage, the default argument value owned_only=True retrieves only the cards owned by you.

for card in conn.search("Model Card", owned_only=True, top=10):
    print(card.title)  # treat it as a normal card

Submit local card modifications to the server by calling card.commit(). The card keeps track of the connection. If you fail to do this throughout your program, you will eventually get an assertion error. Call card.detach() to safely detach a card from a connection without commiting pending changes (this disables further commits). The best practice is to use the card as a context when making changes that require a commit, like below:

conn = aic.connect("http://127.0.0.1:5000", username="admin", password="admin")
with conn.create(card) as card:
    card.title = "Updated model card name"

Connections can also substitute aic as the environment of assistants. For safety, you will get an error if you try to call a connection gist or metric computation without commiting the card first, as the server operate in its own copy.

card = card.gist(
    model=aic.assistants.olama,
    dotenv=conn  # use the connection as assistant configuration
)

Server load may delay the above snippet, as it blocks until notified by the sever. Finally, the server may not support the assistant, for example if it is a custom one.

🛠️ Self-hosting a server

If you have aicard installed, you can immediately self-host a server. To do so, create a dictionary of assistants and start a flask service. This will set up everything the first time, including a database. If you do not plan to expose the server externally, you can login with the default administrator credentials, like above. If you want console instead of persistent logging, skip the log_file argument. Do note that, if an .env file is provided, then there will be an attempt to retrieve missing keyword arguments from there. Below is an example service whose assistant is primarily used for testing:

from aicard.service import serve, TestAssistant

app = serve({"tassist": TestAssistant()}, env=".env")
app.run()

# .env
INDEX=/apidocs # redirect_index
USER=admin  # admin_username 
PASS=admin  # admin_password
LOG=log.txt # log_file

📜 License

TBD