rashomon-pdp 1.1

No project description provided

Rashomon-PDP

Rashomon-PDP is a python package that allows for the use of the Rashomon Partial Dependence Profile (PDP) framework to aggregate and plot the explanation performance of multiple near-optimal models. It currently integrates with the models in libraries H2O and AutoGluon. It also integrates with DALEX's tool package, along with having a base module that lets developers integrate their own models into the package.

Installation

Use the package manager pip to install the rashomon_toolkit package.

pip install rashomon-pdp

For a specific version of the package, go to release history for the specific version, or us the following command where x is the desired version.

pip install rashomon-pdp==x

Dependencies

Core Python dependencies are listed in requirements.txt and include:

• h2o 3.38.0
• matplotlib 3.7
• numpy 2.2.0
• openml 0.15.0
• pandas 2.2.3
• scikit-learn 1.3.0
• seaborn 0.13.2
• autogluon 1.5.0
• streamlit 1.57.0

Note: Java is required as well for the package to run properly, so please install the latest version to your device.

Note: rpy2 and and R PATH are required in order to save a dataset in R.

Quick Start

For simple tasks and running the package quickly, you can use the package's built-in UI.

To access the UI, rom the repository root run:

streamlit run rashomon_toolkit/rashomon_toolkit/web.py

Then open your browser to your local host (normally http://localhost:8501). You can upload CSV data directly, modify parameters like the metrics, and interact with the PDP plot display.

Usage

For PDP selection:

• make_PDP((model, feature: str = None, metric: Metric = None, framework: str = 'h2o', **kwargs): base PDP constructer that constructs a PDP based on the model loaded, using a given metric and framework.
  Metrics include:
• Accuracy
• Recall
• Precision
• F1
  Framework can be user-created library to use locally or one of the pre-defined frameworks:
• h2o_PDP
• gluon_PDP
• dalex_PDP

Base PDP Methods:

• read_data(path) - Loads a dataset from a .csv or .arff file into a pandas DataFrame.
• data_split(data, ratio=0.2, seed=None, **kwargs) - Splits the input data into training and test sets using scikit-learn.
• train(predictors, responses, **kwargs) - Fits the underlying model on the chosen predictors and response column.
• predict(sample, model=None) - Returns model predictions for the supplied sample.
• get_models() - Returns the available trained models for PDP evaluation.
• change_feature(feature) - Switches the PDP control feature to a different column name.
• change_metric(metric) - Replaces the metric used to rank models during Rashomon set generation.
• change_sample(sample) - Sets the sample dataset used for later PDP and Rashomon calculations.
• _modify_feature(feature_value, sample=None) - Temporarily replaces the selected feature in the sample with a fixed value so PDP can be computed.
• _revert_feature(original_values) - Restores the original values of the selected feature after PDP evaluation.
• _get_sample(sample=None) - Returns the active sample, creating one from the provided input if needed.
• _get_respones(sample=None) - Extracts the actual response values from the current sample for metric evaluation.
• get_scores(sample=None, models=None, metric=None, **kwargs) - Computes model scores using the selected metric.
• get_rashomon_set(sample=None, models=None, scores=None, epsilon=0.2, **kwargs) - Builds the Rashomon set of models whose score is within range of the best score.
• get_rashomon_ratio() - Returns the fraction of models that are in the Rashomon set.
• pdp(feature_value, sample=None, model=None) - Computes the PDP value for a single model at one feature setting.
• __call__(feature_value, sample=None, models=None, **kwargs) - Computes the aggregated Rashomon PDP value over the selected Rashomon set.
• bootstrap(feature_value, sample=None, n_boots=50, alpha=0.05, models=None, **kwargs) - Calculates Rashomon PDP over bootstrap samples and returns the PDP and confidence interval.

Model integration helpers

• to_combine(pdp, sample, **kwargs) - Takes PDP models and scores so they can be merged.
• concat(pdp_list, sample=None, **kwargs) - Combines multiple PDP objects into one larger collection of models and scores.

Plotting helpers

• render_pdp(data, x_col, y_col, group_col, title, coverage_rate, mean_ci_width, y_label='response', description=None, save_path=None) - Draws a comparison plot of the best model and the Rashomon set, including confidence intervals and summary metrics.
• render_single_pdp(data, x_col, y_col, title, y_label='response', color='#4659a7', description=None) - Draws a single PDP curve with confidence intervals.

Contributing

Contribution credit to:

• Laith Agbaria
• CJ Reitter
• Varun Brahmadin
• Marit Paul
• Bing Steens
• Gideon Mazzola

License

MIT