Skip to main content

Explainable Leaderboards for Natural Language Processing

Project description

ExplainaBoard: An Explainable Leaderboard for NLP

Introduction | Web Tool | API Tool | Download | Paper | Video | Bib



License GitHub stars PyPI Code Style

Introduction

ExplainaBoard is an interpretable, interactive and reliable leaderboard with seven (so far) new features (F) compared with generic leaderboard.

  • F1: Single-system Analysis: What is a system good or bad at?
  • F2: Pairwise Analysis: Where is one system better (worse) than another?
  • F3: Data Bias Analysis: What are the characteristics of different evaluated datasets?
  • F5: Common errors: What are common mistakes that top-5 systems made?
  • F6: Fine-grained errors: where will errors occur?
  • F7: System Combination: Is there potential complementarity between different systems?

Usage

We not only provide a Web-based Interactive Toolkit but also release an API that users can flexible evaluate their systems offline, which means, you can play with ExplainaBoard at following levels:

  • U1: Just playing with it: You can walk around, track NLP progress, understand relative merits of different top-performing systems.
  • U2: We help you analyze your model: You submit your model outputs and deploy them into online ExplainaBoard
  • U3: Do it by yourself: You can process your model outputs by yourself using our API.

API-based Toolkit: Quick Installation

Method 1: Simple installation from PyPI (Python 3 only)

pip install explainaboard

Method 2: Install from the source and develop locally (Python 3 only)

# Clone current repo
git clone https://github.com/neulab/ExplainaBoard.git
cd ExplainaBoard

# Requirements
pip install -r requirements.txt

# Install the package
python setup.py install

Then, you can run following examples via bash

Example for CLI

  • text-classification:
explainaboard --task text-classification --system_outputs ./data/system_outputs/sst2/sst2-lstm.tsv
  • named-entity-recognition:
  explainaboard --task named-entity-recognition --system_outputs ./data/system_outputs/conll2003/conll2003.elmo
  • extractive-qa:
    explainaboard --task extractive-qa --system_outputs ./data/system_outputs/squad/testset-en.json
  • summarization:
    explainaboard --task summarization --system_outputs ./data/system_outputs/cnndm/cnndm_mini.bart
  • text-pair-classification:
    explainaboard --task text-pair-classification --system_outputs ./data/system_outputs/snli/snli.bert
  • hellaswag
    explainaboard --task hellaswag --system_outputs ./data/system_outputs/hellaswag/hellaswag.random

Example for Python SDK

from explainaboard import TaskType, get_loader, get_processor

path_data = "./explainaboard/tests/artifacts/test-summ.tsv"
loader = get_loader(TaskType.summarization, data = path_data)
data = loader.load()
processor = get_processor(TaskType.summarization, data = data)
analysis = processor.process()
analysis.write_to_directory("./")

Web-based Toolkit: Quick Learning

We deploy ExplainaBoard as a Web toolkit, which includes 9 NLP tasks, 40 datasets and 300 systems. Detailed information is as follows.

So far, ExplainaBoard covers following tasks

Task Sub-task Dataset Model Attribute
Sentiment 8 40 2
Text Classification Topics 4 18 2
Intention 1 3 2
Text-Span Classification Aspect Sentiment 4 20 4
Text pair Classification NLI 2 6 7
NER 3 74 9
Sequence Labeling POS 3 14 4
Chunking 3 14 9
CWS 7 64 7
Structure Prediction Semantic Parsing 4 12 4
Text Generation Summarization 2 36 7

Submit Your Results

You can submit your system's output by this form following the format description.

Download System Outputs

We haven't released datasets or corresponding system outputs that require licenses. But If you have licenses please fill in this form and we will send them to you privately. (Description of output's format can refer here If these system outputs are useful for you, you can cite our work.

Currently Covered Systems

So far, ExplainaBoard support more than 10 NLP tasks, including sequence classification, labeling, extraction and generation. Click here to see more.

Acknowledgement

We thanks all authors who share their system outputs with us: Ikuya Yamada, Stefan Schweter, Colin Raffel, Yang Liu, Li Dong. We also thank Vijay Viswanathan, Yiran Chen, Hiroaki Hayashi for useful discussion and feedback about ExplainaBoard.

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

explainaboard-0.5.0.tar.gz (519.6 kB view details)

Uploaded Source

Built Distribution

explainaboard-0.5.0-py2.py3-none-any.whl (550.2 kB view details)

Uploaded Python 2 Python 3

File details

Details for the file explainaboard-0.5.0.tar.gz.

File metadata

  • Download URL: explainaboard-0.5.0.tar.gz
  • Upload date:
  • Size: 519.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/3.7.0 pkginfo/1.5.0.1 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.7.6

File hashes

Hashes for explainaboard-0.5.0.tar.gz
Algorithm Hash digest
SHA256 d26027c2b23b26979fc245299f4725d2e0aa4d1ef93809341fe3563cc36a33e3
MD5 713a21c48e2a9f66cd3848c7158086dd
BLAKE2b-256 913fb9f0324c7ca84cf1c7af7590e1059a29d579474a7fa2cb80375434092ab3

See more details on using hashes here.

File details

Details for the file explainaboard-0.5.0-py2.py3-none-any.whl.

File metadata

  • Download URL: explainaboard-0.5.0-py2.py3-none-any.whl
  • Upload date:
  • Size: 550.2 kB
  • Tags: Python 2, Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/3.7.0 pkginfo/1.5.0.1 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.7.6

File hashes

Hashes for explainaboard-0.5.0-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 67f60aba43cb4db6b820791cfbebd8cba30fcaeb8cca3f58630e672a5f6adeb4
MD5 9cd6ae2f19b754f07c212cc4481505a6
BLAKE2b-256 40aa47b55ddfe5ef6a363799737dfaf462eb5556566c0c893b96b1435204b670

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page