Multi-components system for automatic Arabic diacritics restoration

About

This tool is a command-line application written in Python 3 that automatically add diacritics to raw undiacritized Arabic text. To accomplish this task, it uses several techniques: Deep Learning, rule-based and statistical corrections. The deep learning part was implemented using Tensorflow. It was released as a support for the research paper: "Multi-components system for automatic Arabic diacritization" at ECIR2020.

Installation

This tool is available as a Python 3 package pipeline-diacritizer installable through pip. For installation instructions check the Download and installation wiki page.

Functions

This tool has 4 main functions: preprocessing of the data, training on the processed data, testing, and restoring the diacritics of an undiacritized text. In addition, it can calculates some statistics on a given dataset and the ratio of Out-of-Vocabulary words in a testing set according to a train set.

This is a quick introduction to the most important ones, without mentioning all the possible options for each one. For additional options, consider calling any subcommand with the option --help or -h (ex: pipeline_diacritizer train --help) or check the wiki for more details.

Preprocessing

Before feeding the new data to this application for training or testing, it needs to be converted to the standard format of this application: one sentence per line, where a sentence is delimited by a dot, a comma, or an end of line character.

$ pipeline_diacritizer process <source_file> <destination_file>

If the data is not yet partitioned into training, validation and testing sets, the program can help in this task using the following command:

$ pipeline_diacritizer partition <dataset_file>

Training

To run the training and validation on selected training/validation sets, use the next command:

$ pipeline_diacritizer train --train-data <train_file> --val-data <val_file>

Testing

To evaluate the performances of the application on a testing set, use this command:

$ pipeline_diacritizer test <test_file>

Diacritization

The following command restores the diacritics of the Arabic words from the supplied text file and outputs a diacritized copy:

$ pipeline_diacritizer diacritize <text_file>

Statistics

To get some statistics about the dataset, such as the count of tokens, arabic words, numbers... use the following command:

$ pipeline_diacritizer stat <dataset_file>

OoV Counting

To calculate the ratio of the Out-of-Vocabulary words between the train set and the validation/test set, use the next command:

$ pipeline_diacritizer oov <train_file> <test_file>

License

Pipeline-diacritizer code is licensed under MIT License.