Skip to main content

Unified Multilingual Robustness Evaluation Toolkit for Natural Language Processing

Project description

Textflint Logo

Unified Multilingual Robustness Evaluation Toolkit for Natural Language Processing

Github Runner Covergae Status Website License GitHub release (latest by date)

TextFlint is a multilingual robustness evaluation platform for natural language processing, which unifies text transformation, sub-population, adversarial attack,and their combinations to provide a comprehensive robustness analysis. So far, TextFlint supports 13 NLP tasks.

If you're looking for robustness evaluation results of SOTA models, you might want the TextFlint IO page.

Features

  • Full coverage of transformation types, including 20 general transformations, 8 subpopulations and 60 task-specific transformations, as well as thousands of their combinations.
  • Subpopulation, which is to identify the specific part of dataset on which the target model performs poorly.
  • Adversarial attack aims to find a perturbation of an input text that is able to fool the given model.
  • Complete analytical report to accurately explain where your model's shortcomings are, such as the problems in lexical rules or syntactic rules.

Online Demo

You can test most of transformations directly on our online demo.

Table of Contents

Setup

Require python version >= 3.7, recommend install with pip.

pip install textflint

Once TextFlint is installed, you can run it via command-line (textflint ...) or integrate it inside another NLP project.

Usage

Workflow

The general workflow of TextFlint is displayed above. Evaluation of target models could be divided into three steps:

  1. For input preparation, the original dataset for testing, which is to be loaded by Dataset, should be firstly formatted as a series of JSON objects. You can use the built-in Dataset following this instruction. TextFlint configuration is specified by Config. Target model is also loaded as FlintModel.
  2. In adversarial sample generation, multi-perspective transformations (i.e., 80+Transformation, Subpopulation and AttackRecipe), are performed on Dataset to generate transformed samples. Besides, to ensure semantic and grammatical correctness of transformed samples, Validator calculates confidence of each sample to filter out unacceptable samples.
  3. Lastly, Analyzer collects evaluation results and ReportGenerator automatically generates a comprehensive report of model robustness.

For example, on the Sentiment Analysis (SA) task, this is a statistical chart of the performance ofXLNET with different types of Transformation/Subpopulation/AttackRecipe on the IMDB dataset.

We release tutorials of performing the whole pipeline of TextFlint on various tasks, including:

Quick Start

Using TextFlint to verify the robustness of a specific model is as simple as running the following command:

$ textflint --dataset input_file --config config.json

where input_file is the input file of csv or json format, config.json is a configuration file with generation and target model options. Transformed datasets would save to your out dir according to your config.json.

Based on the design of decoupling sample generation and model verification, TextFlint can be used inside another NLP project with just a few lines of code.

from textflint import Engine

data_path = 'input.json'
config = 'config.json'
engine = Engine()
engine.run(data_path, config)

For more input and output instructions of TextFlint, please refer to the IO format document.

Architecture

Input layer: receives textual datasets and models as input, represented as Dataset and FlintModel separately.

  • DataSet: a container, provides efficient and handy operation interfaces for Sample. Dataset supports loading, verification, and saving data in Json or CSV format for various NLP tasks.
  • FlintModel: a target model used in an adversarial attack.

Generation layer: there are mainly four parts in generation layer:

  • Subpopulation: generates a subset of a DataSet.
  • Transformation: transforms each sample of Dataset if it can be transformed.
  • AttackRecipe: attacks the FlintModel and generates a DataSet of adversarial examples.
  • Validator: verifies the quality of samples generated by Transformation and AttackRecipe.

textflint provides an interface to integrate the easy-to-use adversarial attack recipes implemented based on textattack. Users can refer to textattack for more information about the supported AttackRecipe.

Report layer: analyzes model testing results and provides robustness report for users.

Learn More

Section Description
Documentation Full API documentation and tutorials
Tutorial The tutorial of textflint components and pipeline
Website Provides evaluation results of SOTA models and transformed data download
Online Demo Interactive demo to try single text transformations
Paper Our system paper which was received by ACL2021

Contributing

We welcome community contributions to TextFlint in the form of bugfixes 🛠️ and new features💡! If you want to contribute, please first read our contribution guideline.

Citation

If you are using TextFlint for your work, please kindly cite our ACL2021 TextFlint demo paper:

@inproceedings{wang-etal-2021-textflint,
    title = {TextFlint: Unified Multilingual Robustness Evaluation Toolkit for Natural Language Processing},
    author = {Wang, Xiao  and Liu, Qin  and Gui, Tao  and Zhang, Qi and others},
    booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations},
    month = {aug},
    year = {2021},
    address = {Online},
    publisher = {Association for Computational Linguistics},
    url = {https://aclanthology.org/2021.acl-demo.41},
    doi = {10.18653/v1/2021.acl-demo.41},
    pages = {347--355}
}

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

textflint-0.1.0.tar.gz (241.4 kB view details)

Uploaded Source

Built Distribution

textflint-0.1.0-py3-none-any.whl (377.0 kB view details)

Uploaded Python 3

File details

Details for the file textflint-0.1.0.tar.gz.

File metadata

  • Download URL: textflint-0.1.0.tar.gz
  • Upload date:
  • Size: 241.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/34.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.63.0 importlib-metadata/4.11.3 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.10.2

File hashes

Hashes for textflint-0.1.0.tar.gz
Algorithm Hash digest
SHA256 48d82bf9a16bb2c95778aa0540b92ec0db8b46f2d98ccd8f2222e72d91781cf1
MD5 6586f73e2b1068823fe3268295d7b48f
BLAKE2b-256 d9f926f9026e18139e2e641a6af9c522afc64a17bea9790be5197f8d314eb124

See more details on using hashes here.

File details

Details for the file textflint-0.1.0-py3-none-any.whl.

File metadata

  • Download URL: textflint-0.1.0-py3-none-any.whl
  • Upload date:
  • Size: 377.0 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/34.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.63.0 importlib-metadata/4.11.3 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.10.2

File hashes

Hashes for textflint-0.1.0-py3-none-any.whl
Algorithm Hash digest
SHA256 a741b5244bdf1e8bca365bc52680d9057a90b61d50b4848226b8f4436dd393c9
MD5 b0ef37b75f4317a1ea2b127e34694d0c
BLAKE2b-256 7e2ca1d3420299b4f33a040088383544d92356b2cfad1508dcff67a764f48931

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page