modelagnosticsafeaipackage 0.3.0

MODEL AGNOSTIC SAFE AI package to measure risks of AI models WITHOUT CONSIDERING TYPE OF THE MODEL

MODEL AGNOSTIC SAFE AI

The increasing widespread of Artificial Intelligence (AI) applications implies the formalisa- tion of an AI risk management model which needs methodological guidelines for an effec- tive implementation. To fill the gap, Giudici and Raffinetti (2023) introduced a S.A.F.E. risk management model which derives from the proposed international regulations four main compliance principles: Security, Accuracy, Fairness and Explainability, that can be measured for any AI application. The primary motivation for developing safeaipackage is providing a unified framework that can evaluate these AI risks. The important aspect of the S.A.F.E. model is that it proposes metrics that are interrelated, are standardised, and have a common mathematical root: the Lorenz Zonoid tool (see, e.g. Koshevoy and Mosler 1996; Lorenz 1905). Despite the advantages deriving from the S.A.F.E approach, it suffers from being computationally intensive as it requires the construction of all the models’ configurations. Moreover, the recent regulatory debate has further expanded the notion of Security, distinguishing the internal resilience of an AI system: its robustness; from the external resilience of the ecosystem which surrounds it: environmental, social and governance sustainability see, e.g. International Standard Organisation 2023. In line with this evolution and the need of providing a unified and computationally efficient method, in this contribution we suggest to combine different statistical metrics able to measure the Security (robustness), Accuracy, Fairness and Explainability of highly complex machine learning models. We remark that the definition of robustness that we adopt in this paper derives from that being employed by AI regulators and standard setters, for which an AI system should achieve an appropriate level of robustness, to be resilient to internal anomalies and external attacks. For consistency, the proposed measures will be based on the Lorenz and concordance curves (see, e.g. Giudici and Raffinetti 2011). This will allow to integrate all measures into an agnostic score that can be employed to assess the trustworthiness of any AI application.

The revisited S.A.F.E. approach will be called RGB where RGB stands for “Rank Graduation Box”. The use of the term “box” is motivated by the need of emphasizing that our proposal is always in progress so that, like a box, it can be constantly filled by innovative tools addressed to the measurement of the new future requirements necessary for the safety condition of AI-systems.

Install

Simply use:

pip install safeaipackage

Example

In the folder "examples", we provide two notebooks related to a classification and a regression problem applied to the employee dataset. This dataset can also be downloaded from this folder.

Citations

The proposed measures in this package came primarily out of research by Paolo Giudici, Emanuela Raffinetti, and Golnoosh Babaei in the Statistical laboratory at the University of Pavia. This package is based on the following papers. If you use safeaipackage in your research we would appreciate a citation to our papers:

• Giudici, P., & Raffinetti, E. (2024). RGA: a unified measure of predictive accuracy. Advances in Data Analysis and Classification, 1-27.
• Raffinetti, E. (2023). A rank graduation accuracy measure to mitigate artificial intelligence risks. Quality & Quantity, 57(Suppl 2), 131-150.