Command-line application to automatically restore the diacritics of an Arabic text.
Multi-components system for automatic Arabic diacritics restoration
This tool is a command-line application written in Python 3 that automatically add diacritics to raw undiacritized Arabic text. To accomplish this task, it uses several techniques: Deep Learning, rule-based and statistical corrections. The deep learning part was implemented using Tensorflow. It was released as a support for the research paper: "Multi-components system for automatic Arabic diacritization" at ECIR2020.
This tool is available as a Python 3 package
pipeline-diacritizer installable through
pip. For installation
instructions check the
Download and installation wiki page.
This tool has 4 main functions: preprocessing of the data, training on the processed data, testing, and restoring the diacritics of an undiacritized text. In addition, it can calculates some statistics on a given dataset and the ratio of Out-of-Vocabulary words in a testing set according to a train set.
This is a quick introduction to the most important ones, without mentioning all the possible options for each one. For
additional options, consider calling any subcommand with the option
python3 pipeline_diacritizer.py train --help) or check the wiki
for more details.
Before feeding the new data to this application for training or testing, it needs to be converted to the standard format of this application: one sentence per line, where a sentence is delimited by a dot, a comma, or an end of line character.
$ python3 pipeline_diacritizer.py process <source_file> <destination_file>
If the data is not yet partitioned into training, validation and testing sets, the program can help in this task using the following command:
$ python3 pipeline_diacritizer.py partition <dataset_file>
To run the training and validation on selected training/validation sets, use the next command:
$ python3 pipeline_diacritizer.py train --train-data <train_file> --val-data <val_file>
To evaluate the performances of the application on a testing set, use this command:
$ python3 pipeline_diacritizer.py test <test_file>
The following command restores the diacritics of the Arabic words from the supplied text file and outputs a diacritized copy:
$ python3 pipeline_diacritizer.py diacritize <text_file>
To get some statistics about the dataset, such as the count of tokens, arabic words, numbers... use the following command:
$ python3 pipeline_diacritizer.py stat <dataset_file>
To calculate the ratio of the Out-of-Vocabulary words between the train set and the validation/test set, use the next command:
$ python3 pipeline_diacritizer.py oov <train_file> <test_file>
Pipeline-diacritizer code is licensed under MIT License.
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|Filename, size||File type||Python version||Upload date||Hashes|
|Filename, size pipeline_diacritizer-1.0.1-py3-none-any.whl (18.6 kB)||File type Wheel||Python version py3||Upload date||Hashes View|
|Filename, size pipeline_diacritizer-1.0.1.tar.gz (17.5 kB)||File type Source||Python version None||Upload date||Hashes View|
Hashes for pipeline_diacritizer-1.0.1-py3-none-any.whl
Hashes for pipeline_diacritizer-1.0.1.tar.gz