Skip to main content

Explainable Leaderboards for Natural Language Processing

Project description

ExplainaBoard: An Explainable Leaderboard for NLP



License GitHub stars PyPI Code Style Integration Tests

What is ExplainaBoard?

When developing a natural language processing (NLP or AI) system, often one of the hardest things is to understand where your system is working and where it is failing, and deciding what to do next. ExplainaBoard is a tool that inspects your system outputs, identifies what is working and what is not working, and helps inspire you with ideas of where to go next.

It offers a number of different ways with which you can evaluate and understand your data:

  1. Single-system Analysis: What is a system good or bad at?
  2. Pairwise Analysis: Where is one system better (worse) than another?
  3. Data Bias Analysis: What are the characteristics of different evaluated datasets?
  4. Common Errors: What are common mistakes that top-5 systems made?
  5. Fine-grained Error Analysis: where do errors occur?
  6. System Combination: Is there potential complementarity between different systems?

Using Explainaboard

ExplainaBoard can be used online or offline.

Online Usage

Browse the web interface, which gives you the ability to browse outputs and upload your own system outputs.

Offline Usage

First, follow the installation directions below, then take a look at our CLI examples.

Install Method 1 - Standard Use: Simple installation from PyPI (Python 3 only)

pip install --upgrade pip  # recommending the newest version of pip.
pip install explainaboard
python -m spacy download en_core_web_sm  # if you plan to use the TextClassificationProcessor

Install Method 2 - Development: Install from the source and develop locally (Python 3 only)

# Clone current repo
git clone https://github.com/neulab/ExplainaBoard.git
cd ExplainaBoard

# Install the required dependencies and dev dependencies
pip install ."[dev]"
pre-commit install
  • Testing: To run tests, you can run python -m unittest.
  • Linting and Code Style: This project uses flake8 (linter) and black (formatter). They are enforced in the pre-commit hook and in the CI pipeline.
    • run python -m black . to format code
    • run flake8 to lint code
    • You can also configure your IDE to automatically format and lint the files as you are writing code.

After trying things out in the CLI, you can read how to add new features, tasks, or file formats.

Acknowledgement

ExplainaBoard is developed by Carnegie Mellon University, Inspired Cognition Inc., and other collaborators. If you find it useful in research, you can cite it in papers:

@inproceedings{liu-etal-2021-explainaboard,
    title = "{E}xplaina{B}oard: An Explainable Leaderboard for {NLP}",
    author = "Liu, Pengfei and Fu, Jinlan and Xiao, Yang and Yuan, Weizhe and Chang, Shuaichen and Dai, Junqi and Liu, Yixin and Ye, Zihuiwen and Neubig, Graham",
    booktitle = "Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations",
    month = aug,
    year = "2021",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.acl-demo.34",
    doi = "10.18653/v1/2021.acl-demo.34",
    pages = "280--289",
}

We thanks all authors who share their system outputs with us: Ikuya Yamada, Stefan Schweter, Colin Raffel, Yang Liu, Li Dong. We also thank Vijay Viswanathan, Yiran Chen, Hiroaki Hayashi for useful discussion and feedback about ExplainaBoard.

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

explainaboard-0.9.4.tar.gz (93.7 kB view details)

Uploaded Source

Built Distribution

explainaboard-0.9.4-py2.py3-none-any.whl (130.3 kB view details)

Uploaded Python 2 Python 3

File details

Details for the file explainaboard-0.9.4.tar.gz.

File metadata

  • Download URL: explainaboard-0.9.4.tar.gz
  • Upload date:
  • Size: 93.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.0 CPython/3.9.13

File hashes

Hashes for explainaboard-0.9.4.tar.gz
Algorithm Hash digest
SHA256 bb420c5950a1236245809c346027a4c27340956006bc5ee856cd44a4a38db2b7
MD5 0b6361f13a79e86e522f9b462d1d8ca1
BLAKE2b-256 dcd64c73c3cdc2b395eb38f152036deff82f41a000100d473c0c47238287ac6c

See more details on using hashes here.

File details

Details for the file explainaboard-0.9.4-py2.py3-none-any.whl.

File metadata

File hashes

Hashes for explainaboard-0.9.4-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 67c35bc408dae1f01a88fc5ae993381d0c7bf6e7dd37e005c891c277e2ffb6b2
MD5 681bf8bcb8c28e8dc2dff8acdef5640a
BLAKE2b-256 25f139ecaa8a716e05b17a89c89f362126df7344a15233b6f9ecb5c8e63c187c

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page