Skip to main content

Command-line application to automatically restore the diacritics of an Arabic text.

Project description

Multi-components system for automatic Arabic diacritics restoration


This tool is a command-line application written in Python 3 that automatically add diacritics to raw undiacritized Arabic text. To accomplish this task, it uses several techniques: Deep Learning, rule-based and statistical corrections. The deep learning part was implemented using Tensorflow. It was released as a support for the research paper: "Multi-components system for automatic Arabic diacritization" at ECIR2020.


This tool is available as a Python 3 package pipeline-diacritizer installable through pip. For installation instructions check the Download and installation wiki page.


This tool has 4 main functions: preprocessing of the data, training on the processed data, testing, and restoring the diacritics of an undiacritized text. In addition, it can calculates some statistics on a given dataset and the ratio of Out-of-Vocabulary words in a testing set according to a train set.

This is a quick introduction to the most important ones, without mentioning all the possible options for each one. For additional options, consider calling any subcommand with the option --help or -h (ex: pipeline_diacritizer train --help) or check the wiki for more details.


Before feeding the new data to this application for training or testing, it needs to be converted to the standard format of this application: one sentence per line, where a sentence is delimited by a dot, a comma, or an end of line character.

$ pipeline_diacritizer process <source_file> <destination_file>

If the data is not yet partitioned into training, validation and testing sets, the program can help in this task using the following command:

$ pipeline_diacritizer partition <dataset_file>


To run the training and validation on selected training/validation sets, use the next command:

$ pipeline_diacritizer train --train-data <train_file> --val-data <val_file>


To evaluate the performances of the application on a testing set, use this command:

$ pipeline_diacritizer test <test_file>


The following command restores the diacritics of the Arabic words from the supplied text file and outputs a diacritized copy:

$ pipeline_diacritizer diacritize <text_file>


To get some statistics about the dataset, such as the count of tokens, arabic words, numbers... use the following command:

$ pipeline_diacritizer stat <dataset_file>

OoV Counting

To calculate the ratio of the Out-of-Vocabulary words between the train set and the validation/test set, use the next command:

$ pipeline_diacritizer oov <train_file> <test_file>


Pipeline-diacritizer code is licensed under MIT License.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pipeline_diacritizer-1.0.2.tar.gz (17.5 kB view hashes)

Uploaded source

Built Distribution

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page