Skip to main content

A package of various specified distribution shift patterns of out-of-distributoin generalization problem on tabular data, and tools for diagnosing model performance are integrated.

Project description

WhyShift: A Benchmark with Specified Distribution Shift Patterns

Jiashuo Liu*, Tianyu Wang*, Peng Cui, Hongseok Namkoong

Tsinghua University, Columbia University

WhyShift is a python package that provides a benchmark with various specified distribution shift patterns on real-world tabular data. And tools to diagnose performance degradation are integrated in it, including performance degradation decomposition and risky region identification. Our testbed highlights the importance of future research that builds an understanding of how distributions differ. For more details, please refer to our paper.

If you find this repository useful in your research, please cite the following paper:

@inproceedings{liu2023need,
  title={On the Need for a Language Describing Distribution Shifts: Illustrations on Tabular Datasets},
  author={Jiashuo Liu and Tianyu Wang and Peng Cui and Hongseok Namkoong},
  booktitle={Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
  year={2023}
}

For settings utilizing ACS Income, Public Coverage, Mobility datasets

  • get_data(task, state, year, need_preprocess, root_dir) function
    • task values: 'income', 'pubcov', 'mobility'
  • examples:
    from whyshift import get_data
    # for ACS Income
    X, y, feature_names = get_data("income", "CA", True, './datasets/acs/', 2018)
    # for ACS Public Coverage
    X, y, feature_names = get_data("pubcov", "CA", True, './datasets/acs/', 2018)
    # for ACS Mobility
    X, y, feature_names = get_data("mobility", "CA", True, './datasets/acs/', 2018)
    
  • support state values:
    • ['AL', 'AK', 'AZ', 'AR', 'CA', 'CO', 'CT', 'DE', 'FL', 'GA', 'HI', 'ID', 'IL', 'IN', 'IA', 'KS', 'KY', 'LA', 'ME', 'MD', 'MA', 'MI', 'MN', 'MS', 'MO', 'MT', 'NE', 'NV', 'NH', 'NJ', 'NM', 'NY', 'NC', 'ND', 'OH', 'OK', 'OR', 'PA', 'RI', 'SC', 'SD', 'TN', 'TX', 'UT', 'VT', 'VA', 'WA', 'WV', 'WI', 'WY', 'PR']

For settings utilizing US Accident, Taxi datasets

  • download data files:
    # US Accident:
    https://www.kaggle.com/datasets/sobhanmoosavi/us-accidents
    # Taxi
    https://www.kaggle.com/competitions/nyc-taxi-trip-duration
    
  • put data files in dir ./datasets/
    • accident: ./datasets/Accident/US_Accidents_Dec21_updated.csv
    • taxi: ./datasets/Taxi/{city}_clean.csv
  • pass the path to the data file of get_data function
  • example:
    from whyshift import get_data
    # for US Accident
    X, y, _ = get_data("accident", "CA", True, './datasets/Accident/US_Accidents_Dec21_updated.csv')
    # for Taxi
    X, y, _ = get_data("taxi", "nyc", True, './datasets/Taxi/train.csv')
    
  • support state values:
    • for US Accident: ['CA', 'TX', 'FL', 'OR', 'MN', 'VA', 'SC', 'NY', 'PA', 'NC', 'TN', 'MI', 'MO']
    • for Taxi: ['nyc', 'bog', 'uio', 'mex']

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

whyshift-0.1.3.tar.gz (40.0 kB view details)

Uploaded Source

Built Distribution

whyshift-0.1.3-py3-none-any.whl (41.4 kB view details)

Uploaded Python 3

File details

Details for the file whyshift-0.1.3.tar.gz.

File metadata

  • Download URL: whyshift-0.1.3.tar.gz
  • Upload date:
  • Size: 40.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.9.18

File hashes

Hashes for whyshift-0.1.3.tar.gz
Algorithm Hash digest
SHA256 7126191ff46dd8d2c457fd488998aa705ce53bd2bc753f919550c3e68fdd601c
MD5 1c420fffae5a399dfea96370a7833d69
BLAKE2b-256 cd4c5011305701768ea6bd704af850b03ceda54e9a59210a86e01f46a2ec8f55

See more details on using hashes here.

File details

Details for the file whyshift-0.1.3-py3-none-any.whl.

File metadata

  • Download URL: whyshift-0.1.3-py3-none-any.whl
  • Upload date:
  • Size: 41.4 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.9.18

File hashes

Hashes for whyshift-0.1.3-py3-none-any.whl
Algorithm Hash digest
SHA256 1d6209bff1ea0f6fe937e1f9bf84b55f75bd4ec292d963fd8ac5a22b5c5ab6f0
MD5 c31d305af0d89fce201bb4d2bca8a197
BLAKE2b-256 6396b9c18e1e210d4e8b78f34bc780f32f2bc7ea5884e0de88b22fcc6e33978c

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page