Explainable Leaderboards for Natural Language Processing
Project description
ExplainaBoard: An Explainable Leaderboard for NLP
Introduction | Web Tool | API Tool | Download | Paper | Video | Bib
Introduction
ExplainaBoard is an interpretable, interactive and reliable leaderboard with seven (so far) new features (F) compared with generic leaderboard.
- F1: Single-system Analysis: What is a system good or bad at?
- F2: Pairwise Analysis: Where is one system better (worse) than another?
- F3: Data Bias Analysis: What are the characteristics of different evaluated datasets?
- F5: Common errors: What are common mistakes that top-5 systems made?
- F6: Fine-grained errors: where will errors occur?
- F7: System Combination: Is there potential complementarity between different systems?
Usage
We not only provide a Web-based Interactive Toolkit but also release an API that users can flexible evaluate their systems offline, which means, you can play with ExplainaBoard at following levels:
- U1: Just playing with it: You can walk around, track NLP progress, understand relative merits of different top-performing systems.
- U2: We help you analyze your model: You submit your model outputs and deploy them into online ExplainaBoard
- U3: Do it by yourself: You can process your model outputs by yourself using our API.
API-based Toolkit: Quick Installation
Method 1: Simple installation from PyPI (Python 3 only)
pip install explainaboard
Method 2: Install from the source and develop locally (Python 3 only)
# Clone current repo
git clone https://github.com/neulab/ExplainaBoard.git
cd ExplainaBoard
# Requirements
pip install -r requirements.txt
# Install the package
python setup.py install
Then, you can run following examples via bash
Example for CLI
- text-classification:
explainaboard --task text-classification --system_outputs ./data/system_outputs/sst2/sst2-lstm.tsv
- named-entity-recognition:
explainaboard --task named-entity-recognition --system_outputs ./data/system_outputs/conll2003/conll2003.elmo
- extractive-qa:
explainaboard --task extractive-qa --system_outputs ./data/system_outputs/squad/testset-en.json
- summarization:
explainaboard --task summarization --system_outputs ./data/system_outputs/cnndm/cnndm_mini.bart
- text-pair-classification:
explainaboard --task text-pair-classification --system_outputs ./data/system_outputs/snli/snli.bert
- hellaswag
explainaboard --task hellaswag --system_outputs ./data/system_outputs/hellaswag/hellaswag.random
Example for Python SDK
from explainaboard import TaskType, get_loader, get_processor
path_data = "./explainaboard/tests/artifacts/test-summ.tsv"
loader = get_loader(TaskType.summarization, data = path_data)
data = loader.load()
processor = get_processor(TaskType.summarization, data = data)
analysis = processor.process()
analysis.write_to_directory("./")
Web-based Toolkit: Quick Learning
We deploy ExplainaBoard as a Web toolkit, which includes 9 NLP tasks, 40 datasets and 300 systems. Detailed information is as follows.
So far, ExplainaBoard covers following tasks
Task | Sub-task | Dataset | Model | Attribute |
---|---|---|---|---|
Sentiment | 8 | 40 | 2 | |
Text Classification | Topics | 4 | 18 | 2 |
Intention | 1 | 3 | 2 | |
Text-Span Classification | Aspect Sentiment | 4 | 20 | 4 |
Text pair Classification | NLI | 2 | 6 | 7 |
NER | 3 | 74 | 9 | |
Sequence Labeling | POS | 3 | 14 | 4 |
Chunking | 3 | 14 | 9 | |
CWS | 7 | 64 | 7 | |
Structure Prediction | Semantic Parsing | 4 | 12 | 4 |
Text Generation | Summarization | 2 | 36 | 7 |
Submit Your Results
You can submit your system's output by this form following the format description.
Download System Outputs
We haven't released datasets or corresponding system outputs that require licenses. But If you have licenses please fill in this form and we will send them to you privately. (Description of output's format can refer here If these system outputs are useful for you, you can cite our work.
Currently Covered Systems
So far, ExplainaBoard support more than 10 NLP tasks, including sequence classification, labeling, extraction and generation. Click here to see more.
Acknowledgement
We thanks all authors who share their system outputs with us: Ikuya Yamada, Stefan Schweter, Colin Raffel, Yang Liu, Li Dong. We also thank Vijay Viswanathan, Yiran Chen, Hiroaki Hayashi for useful discussion and feedback about ExplainaBoard.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file explainaboard-0.5.0.tar.gz
.
File metadata
- Download URL: explainaboard-0.5.0.tar.gz
- Upload date:
- Size: 519.6 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.4.1 importlib_metadata/3.7.0 pkginfo/1.5.0.1 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.7.6
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | d26027c2b23b26979fc245299f4725d2e0aa4d1ef93809341fe3563cc36a33e3 |
|
MD5 | 713a21c48e2a9f66cd3848c7158086dd |
|
BLAKE2b-256 | 913fb9f0324c7ca84cf1c7af7590e1059a29d579474a7fa2cb80375434092ab3 |
File details
Details for the file explainaboard-0.5.0-py2.py3-none-any.whl
.
File metadata
- Download URL: explainaboard-0.5.0-py2.py3-none-any.whl
- Upload date:
- Size: 550.2 kB
- Tags: Python 2, Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.4.1 importlib_metadata/3.7.0 pkginfo/1.5.0.1 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.7.6
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 67f60aba43cb4db6b820791cfbebd8cba30fcaeb8cca3f58630e672a5f6adeb4 |
|
MD5 | 9cd6ae2f19b754f07c212cc4481505a6 |
|
BLAKE2b-256 | 40aa47b55ddfe5ef6a363799737dfaf462eb5556566c0c893b96b1435204b670 |