Skip to main content
Python Software Foundation 20th Year Anniversary Fundraiser  Donate today!

NLP error analysis.

Project description


This opensourcing is work-in-progress!

Errudite is an interactive tool for scalable, reproducible, and counterfactual error analysis. Errudite provides an expressive domain-specific language for extracting relevant features of linguistic data, which allows users to visualize data attributes, group relevant instances, and perform counterfactual analysis across all available validation data.

Getting Started

  1. Watch this video demo that contains the highlights of Errudite's functions & use cases
  2. Get set up quickly
  3. Try Errudite's user interface on machine comprehension
  4. Try the tutorials on JupyterLab notebooks
  5. Read the documentation


If you are interested in this work, please see our ACL 2019 research paper and consider citing our work:

    title = {Errudite: Scalable, Reproducible, and Testable Error Analysis},
    author = {Wu, Tongshuang and Ribeiro, Marco Tulio and Heer, Jeffrey and Weld Daniel S.},
  booktitle={the 57th Annual Meeting of the Association for Computational Linguistics (ACL 2019)},
    year = {2019},
    url = {},

Quick Start



Errudite requires Python 3.6.x. The package is avaiable through pip: Just install it in your Python environment and you're good to go!

# create the virtual environment
virtualenv --no-site-packages -p python3.6 venv
# activate venv
source venv/bin/activate
# install errudite
pip install errudite

Install from source

You can also install Errudite by cloning our git repository:

git clone

Create a Python 3.6 virtual environment, and install Errudite in editable mode by running:

pip install --editable .

This will make errudite available on your system but it will use the sources from the local clone you made of the source repository.

GUI Server

Errudite has a UI wrapped for Machine Comprehension and Visual Question Answering tasks. The interface integrates all the key analysis functions (e.g., inspecting instance attributes, grouping similar instances, rewriting instances), It also provides exploration support such as visualizing data distributions, suggesting potential queries, and presenting the grouping and rewriting results. While not strictly necessary, it makes their application much more straightforward.

To get a taste of GUI for the machine comprehension task, you should first download a cache folder for preprocessed SQuAD instances, which will help you skip the process of running your own preprocessing:

python -m

                A folder name. Currently, we allow downloading the following:
                squad-100, squad-10570.
    cache_path  A local path where you want to save the cache folder to.

Then, we need to start the server:

# the model relies on Allennlp, so make sure you install that first.
pip install allennlp==0.8.4
source venv/bin/activate
python -m errudite.server

                A yaml config file path.

The config file looks like the following (or in config.yml):

task: qa # the task, should be "qa" and "vqa".
cache_path: {cache_path}/{cache_folder_name}/ # the cached folder.
model_metas: # a model.
- name: bidaf
  model_class: bidaf # an implemented model class
  model_path: # a local model file path
  # an online path to an Allennlp model
  description: Pretrained model from Allennlp, for the BiDAF model (QA)
attr_file_name: null # It set, to load previously saved analysis.
group_file_name: null
rewrite_file_name: null

Then visit http://localhost:5000/ in your web browser.

JupyterLab Tutorial

Besides used in a GUI, errudite also serves as a general python package. The tutorial goes through:

  1. Preprocessing the data, and extending Errudite to different tasks & predictors
  2. Creating data attributes and data groups with a domain specific language (or your customized functions).
  3. Creating rewrite rules with the domain specific language (or your customized functions).

To go through the tutorial, do the following steps:

# clone the repo
git clone
# initial folder: errudite/
# create the virtual environment
virtualenv --no-site-packages -p python3.6 venv
# activate venv
source venv/bin/activate

# run the default setup script
pip install --editable .

# get to the tutorial folder, and start!
cd tutorials
pip install -r requirements_tutorials.txt
jupyter lab

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for errudite, version 0.0.3
Filename, size File type Python version Upload date Hashes
Filename, size errudite-0.0.3-py3-none-any.whl (3.7 MB) File type Wheel Python version py3 Upload date Hashes View
Filename, size errudite-0.0.3.tar.gz (127.3 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page