AL360° Trustworthy AI Toolbox

The AL360° Trustworthy AI Toolbox is an end-to-end suite of tools designed to assess, develop, and deploy AI systems in a safe, trustworthy, and ethical manner. With this toolbox, stakeholders of AI systems can better understand their models, make trustworthy data-driven decisions, and take corrective actions to ensure fairness, transparency, and accountability in AI development.

This toolbox integrates several open-source tools and libraries into a unified experience, enabling users to identify model errors, evaluate fairness, understand predictions, and optimize decision-making. It is aimed at both technical users such as data scientists and engineers and non-technical stakeholders such as business leaders and policy-makers.

---

Key Features

1. Error Analysis: Identify cohorts where models underperform using Error Analysis.
2. Fairness Assessment: Evaluate fairness across sensitive attributes with Fairlearn.
3. Model Interpretability: Understand model predictions through InterpretML.
4. Counterfactual Analysis: Explore counterfactual scenarios with DiCE.
5. Causal Analysis: Conduct causal inference with EconML.
6. Data Balance Diagnostics: Visualize and mitigate imbalances in datasets.

---

Repository Overview

Repository	Description
Affectlog360	Central repository containing the AL360° Trustworthy AI tools including dashboards for model interpretability, fairness, error analysis, and data balance analysis.
Affectlog360 Mitigations	Library for applying mitigation techniques to improve model fairness and performance.
Trustworthy-AI-Tracker	JupyterLab extension for managing, tracking, and comparing machine learning experiments.
Affectlog360 GenBit	Library to measure and mitigate gender bias in Natural Language Processing (NLP) models.

---

Installation

You can install the AL360° Trustworthy AI Toolbox using the following command:

pip install al360_taiwidgets

Note: If you're running this in Jupyter, restart the kernel after installation to ensure the tools are loaded correctly.

---

AL360° Trustworthy AI Dashboard

The AL360° Trustworthy AI Dashboard is the core component of the toolbox. It provides an intuitive interface that allows users to explore model performance, errors, fairness, and feature importance, as well as generate actionable insights for decision-making. It integrates multiple tools into a cohesive flow that helps practitioners quickly diagnose issues and improve models.

Capabilities:

• Error Detection: Find and analyze cohorts where the model exhibits higher error rates.
• Fairness: Understand how the model impacts different demographic groups and mitigate disparities.
• Interpretability: Explain model predictions both globally and locally.
• Causal Analysis: Apply What-If scenarios to see how changes in features impact outcomes.
• Data Balance: Visualize data distributions and feature imbalances.

Dashboard Flows:

You can customize the AL360° dashboard to suit different use cases:

• Model Overview -> Error Analysis -> Data Explorer: Identify model errors and diagnose root causes in the data.
• Model Overview -> Fairness Assessment -> Data Explorer: Diagnose fairness issues by understanding feature distributions.
• Model Overview -> Error Analysis -> Counterfactuals Analysis: Use counterfactuals to understand what changes would alter individual predictions.
• Data Explorer -> Causal Inference: Explore causal relationships in the data to make data-driven decisions.

Explore different flows and usage examples in our detailed guides:

• Housing Model Debugging
• Diabetes Progression Model

---

Use Cases

The tools within the AL360° Trustworthy AI Toolbox support various model types and AI use cases, including:

• Tabular Data Models: Used in structured datasets like financial records, medical datasets, etc.
• Text Models: NLP models for classification, question-answering, and more.
• Vision Models: Analyze computer vision models for object detection, classification, and more.

Check out the following notebooks for example use cases:

• Housing Model Debugging (Tabular)
• SQuAD Question Answering (NLP)
• Fridge Object Detection (Vision)

---

Supported Models and Frameworks

The AL360° Trustworthy AI Toolbox supports models trained on datasets in the following formats:

• Python: numpy.ndarray, pandas.DataFrame, iml.datatypes.DenseData, scipy.sparse.csr_matrix.
• Deep Learning: PyTorch, TensorFlow, Keras models.
• Scikit-learn Pipelines: Any model with a predict or predict_proba method compatible with Scikit-learn's API.

---

Jupyter Integration

The AL360° Trustworthy AI Toolbox provides seamless integration with Jupyter notebooks for managing, tracking, and comparing models. You can explore your model's fairness, interpretability, and data balance directly within JupyterLab.

---

Contributing

We welcome contributions to the AL360° Trustworthy AI Toolbox. Please read our CONTRIBUTING.md guide for more details on how to get involved.

---

License

This project is licensed under the MIT License.

---

Useful Links

• Get Started with the AL360° Dashboard
• Explore the AL360° Toolbox Tour