gec-datasets

This library is to handle datasets of Grammatical Error Correction.

Install

pip install gec-datasets

Usage

API

from gec_datasets import GECDatasets
gec = GECDatasets(
    base_path='gec_datasets_base/'
)
conll14 = gec.load('conll14')

assert conll14.srcs is not None
assert conll14.refs is not None
# The number of sentences is 1312.
assert len(conll14.srcs) == 1312
# CoNLL-2014 contains two official references.
assert len(conll14.refs) == 2
# Each reference also contains 1312 sentences.
assert len(conll14.refs[0]) == 1312
assert len(conll14.refs[1]) == 1312

Available ids can be found by:

import gec_datasets
print(gec_datasets.available())

CLI

You can specify multiple ids of the data you want to download in the --ids field.

gecdatasets-download --base_path "gec_datasets_base/" --ids conll14 bea19-dev

Available ids can be found by:

gecdatasets-available

In both API and CLI, datasets will be stored under base_path=.
The first time it is downloaded automatically, and thereafter it is loaded from the saved files.

When you call gec.load('sample'), gec-datasets simply refers to <base_path>/'sample'/{src.txt|ref0.txt|...}.

gec_datasets_base/
├── conll14
│   ├── ref0.txt
│   ├── ref1.txt
│   └── src.txt
├── bea19-dev
│   ├── ref0.txt
│   ├── src.txt
├── bea19-test
│   └── src.txt
...

Supported datasets

Public datasets

The following is synthetic data.

Non-public datasets

nucle-train

• Request data from HERE.
• You will receive an email with release3.3.tar.bz2 attached.
• mkdir <base_path>/nucle/ and put the data as <base_path>/nucle/release3.3.tar.bz2.
• You can now use the data with .load("nucle-train"). The data will be extracted automatically.

lang8-train

• Request data from HERE.
• You will receive an email titled "[NAIST Lang-8 Corpus of Learner English for the 14th BEA Shared Task]".
• mkdir <base_path>/lang8/ and put the data as <base_path>/lang8/lang8.bea19.tar.gz.
• You can now use the data with .load("lang8-train"). The data will be extracted automatically.