Skip to main content

Explainable Leaderboards for Natural Language Processing

Project description

ExplainaBoard: An Explainable Leaderboard for NLP

Introduction | Web Tool | API Tool | Download | Paper | Video | Bib



License GitHub stars PyPI Code Style Integration Tests

What is ExplainaBoard?

When developing a natural language processing (NLP or AI) system, often one of the hardest things is to understand where your system is working and where it is failing, and deciding what to do next. ExplainaBoard is a tool that inspects your system outputs, identifies what is working and what is not working, and helps inspire you with ideas of where to go next.

It offers a number of different ways with which you can evaluate and understand your data:

  1. Single-system Analysis: What is a system good or bad at?
  2. Pairwise Analysis: Where is one system better (worse) than another?
  3. Data Bias Analysis: What are the characteristics of different evaluated datasets?
  4. Common Errors: What are common mistakes that top-5 systems made?
  5. Fine-grained Error Analysis: where do errors occur?
  6. System Combination: Is there potential complementarity between different systems?

How do you get started?

  1. Browse Outputs Online: We provide an online interface where you can browse and understand outputs from various state-of-the-art systems.
  2. Upload Your Own Outputs: You can also upload your own system outputs through the online interace, and we will process them so you can view them alongside other systems.
  3. Run Your Own Analyses Offline: You can follow the installation instructions below to run analyses on your own system data offline. We also plan to support command-line upload to the web interface soon!

Web-based Toolkit

We have deployed ExplainaBoard as a Web toolkit, which includes 9 NLP tasks, 40 datasets and 300 systems. Detailed information is as follows.

Quick Installation

Method 1: Simple installation from PyPI (Python 3 only)

pip install explainaboard
python -m spacy download en_core_web_sm  # if you plan to use the TextClassificationProcessor

Method 2: Install from the source and develop locally (Python 3 only)

# Clone current repo
git clone https://github.com/neulab/ExplainaBoard.git
cd ExplainaBoard

# Install the package
pip install .

Then, you can run following examples via bash

Example for CLI

  • text-classification:
explainaboard --task text-classification --system_outputs ./data/system_outputs/sst2/sst2-lstm.tsv

See detailed description and more supported tasks.

Example for Python SDK

from explainaboard import TaskType, get_loader, get_processor

path_data = "./explainaboard/tests/artifacts/test-summ.tsv"
loader = get_loader(TaskType.summarization, data = path_data)
data = loader.load()
processor = get_processor(TaskType.summarization, data = data)
analysis = processor.process()
analysis.write_to_directory("./")

Acknowledgement

We thanks all authors who share their system outputs with us: Ikuya Yamada, Stefan Schweter, Colin Raffel, Yang Liu, Li Dong. We also thank Vijay Viswanathan, Yiran Chen, Hiroaki Hayashi for useful discussion and feedback about ExplainaBoard.

Project details


Release history Release notifications | RSS feed

This version

0.6.8

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

explainaboard-0.6.8.tar.gz (523.3 kB view details)

Uploaded Source

Built Distribution

explainaboard-0.6.8-py2.py3-none-any.whl (556.3 kB view details)

Uploaded Python 2 Python 3

File details

Details for the file explainaboard-0.6.8.tar.gz.

File metadata

  • Download URL: explainaboard-0.6.8.tar.gz
  • Upload date:
  • Size: 523.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.10.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.10

File hashes

Hashes for explainaboard-0.6.8.tar.gz
Algorithm Hash digest
SHA256 ba6bed4b4c646328a5b6edabb64ac19f65ff2ed2dfe67525bcc139e06b6aaba5
MD5 a502453b74216a912f6411996d13b93d
BLAKE2b-256 26c571c51191ee3023fc21eb9bf6d3077041407644d8cc3d414e1bcb8f960ca4

See more details on using hashes here.

File details

Details for the file explainaboard-0.6.8-py2.py3-none-any.whl.

File metadata

  • Download URL: explainaboard-0.6.8-py2.py3-none-any.whl
  • Upload date:
  • Size: 556.3 kB
  • Tags: Python 2, Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.10.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.10

File hashes

Hashes for explainaboard-0.6.8-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 37e058ddeaba2cb1c3250e0efb9095fe0e178a850be99bc7c65ef85bbb468466
MD5 4bc6e1dcd0987df221514bbdf2da235b
BLAKE2b-256 a32721e0f1ed0d71440a1977f012346df454fa8a69bd64e172e6bbffc358dff6

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page