A package of various specified distribution shift patterns of out-of-distributoin generalization problem on tabular data, and tools for diagnosing model performance are integrated.
Project description
WhyShift
: A Benchmark with Specified Distribution Shift Patterns
Tsinghua University, Columbia University
WhyShift
is a python package that provides a benchmark with various specified distribution shift patterns on real-world tabular data. And tools to diagnose performance degradation are integrated in it, including performance degradation decomposition and risky region identification. Our testbed highlights the importance of future research that builds an understanding of how distributions differ. For more details, please refer to our paper.
If you find this repository useful in your research, please cite the following paper:
@inproceedings{liu2023need,
title={On the Need for a Language Describing Distribution Shifts: Illustrations on Tabular Datasets},
author={Jiashuo Liu and Tianyu Wang and Peng Cui and Hongseok Namkoong},
booktitle={Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
year={2023}
}
For settings utilizing ACS Income, Public Coverage, Mobility datasets
get_data(task, state, year, need_preprocess, root_dir)
functiontask
values: 'income', 'pubcov', 'mobility'
- examples:
from whyshift import get_data # for ACS Income X, y, feature_names = get_data("income", "CA", True, './datasets/acs/', 2018) # for ACS Public Coverage X, y, feature_names = get_data("pubcov", "CA", True, './datasets/acs/', 2018) # for ACS Mobility X, y, feature_names = get_data("mobility", "CA", True, './datasets/acs/', 2018)
- support
state
values:- ['AL', 'AK', 'AZ', 'AR', 'CA', 'CO', 'CT', 'DE', 'FL', 'GA', 'HI', 'ID', 'IL', 'IN', 'IA', 'KS', 'KY', 'LA', 'ME', 'MD', 'MA', 'MI', 'MN', 'MS', 'MO', 'MT', 'NE', 'NV', 'NH', 'NJ', 'NM', 'NY', 'NC', 'ND', 'OH', 'OK', 'OR', 'PA', 'RI', 'SC', 'SD', 'TN', 'TX', 'UT', 'VT', 'VA', 'WA', 'WV', 'WI', 'WY', 'PR']
For settings utilizing US Accident, Taxi datasets
- download data files:
# US Accident: https://www.kaggle.com/datasets/sobhanmoosavi/us-accidents # Taxi https://www.kaggle.com/competitions/nyc-taxi-trip-duration
- put data files in dir
./datasets/
- accident:
./datasets/Accident/US_Accidents_Dec21_updated.csv
- taxi:
./datasets/Taxi/{city}_clean.csv
- accident:
- pass the
path to the data file
ofget_data
function - example:
from whyshift import get_data # for US Accident X, y, _ = get_data("accident", "CA", True, './datasets/Accident/US_Accidents_Dec21_updated.csv') # for Taxi X, y, _ = get_data("taxi", "nyc", True, './datasets/Taxi/train.csv')
- support
state
values:- for US Accident: ['CA', 'TX', 'FL', 'OR', 'MN', 'VA', 'SC', 'NY', 'PA', 'NC', 'TN', 'MI', 'MO']
- for Taxi: ['nyc', 'bog', 'uio', 'mex']
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file whyshift-0.1.3.tar.gz
.
File metadata
- Download URL: whyshift-0.1.3.tar.gz
- Upload date:
- Size: 40.0 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.9.18
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7126191ff46dd8d2c457fd488998aa705ce53bd2bc753f919550c3e68fdd601c |
|
MD5 | 1c420fffae5a399dfea96370a7833d69 |
|
BLAKE2b-256 | cd4c5011305701768ea6bd704af850b03ceda54e9a59210a86e01f46a2ec8f55 |
File details
Details for the file whyshift-0.1.3-py3-none-any.whl
.
File metadata
- Download URL: whyshift-0.1.3-py3-none-any.whl
- Upload date:
- Size: 41.4 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.9.18
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1d6209bff1ea0f6fe937e1f9bf84b55f75bd4ec292d963fd8ac5a22b5c5ab6f0 |
|
MD5 | c31d305af0d89fce201bb4d2bca8a197 |
|
BLAKE2b-256 | 6396b9c18e1e210d4e8b78f34bc780f32f2bc7ea5884e0de88b22fcc6e33978c |