Skip to main content

Command-line application to automatically restore the diacritics of an Arabic text.

Project description

Multi-components system for automatic Arabic diacritics restoration


This tool is a command-line application written in Python 3 that automatically add diacritics to raw undiacritized Arabic text. To accomplish this task, it uses several techniques: Deep Learning, rule-based and statistical corrections. The deep learning part was implemented using Tensorflow. It was released as a support for the research paper: "Multi-components system for automatic Arabic diacritization" at ECIR2020.


This tool is available as a Python 3 package pipeline-diacritizer installable through pip. For installation instructions check the Download and installation wiki page.


This tool has 4 main functions: preprocessing of the data, training on the processed data, testing, and restoring the diacritics of an undiacritized text. In addition, it can calculates some statistics on a given dataset and the ratio of Out-of-Vocabulary words in a testing set according to a train set.

This is a quick introduction to the most important ones, without mentioning all the possible options for each one. For additional options, consider calling any subcommand with the option --help or -h (ex: python3 train --help) or check the wiki for more details.


Before feeding the new data to this application for training or testing, it needs to be converted to the standard format of this application: one sentence per line, where a sentence is delimited by a dot, a comma, or an end of line character.

$ python3 process <source_file> <destination_file>

If the data is not yet partitioned into training, validation and testing sets, the program can help in this task using the following command:

$ python3 partition <dataset_file>


To run the training and validation on selected training/validation sets, use the next command:

$ python3 train --train-data <train_file> --val-data <val_file>


To evaluate the performances of the application on a testing set, use this command:

$ python3 test <test_file>


The following command restores the diacritics of the Arabic words from the supplied text file and outputs a diacritized copy:

$ python3 diacritize <text_file>


To get some statistics about the dataset, such as the count of tokens, arabic words, numbers... use the following command:

$ python3 stat <dataset_file>

OoV Counting

To calculate the ratio of the Out-of-Vocabulary words between the train set and the validation/test set, use the next command:

$ python3 oov <train_file> <test_file>


Pipeline-diacritizer code is licensed under MIT License.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for pipeline-diacritizer, version 1.0.1
Filename, size File type Python version Upload date Hashes
Filename, size pipeline_diacritizer-1.0.1-py3-none-any.whl (18.6 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size pipeline_diacritizer-1.0.1.tar.gz (17.5 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page