Skip to main content

No project description provided

Project description

tnkeeh (تنقيح) is an Arabic preprocessing library for python. It was designed using re for creating quick replacement expressions for several examples.


pip install tnkeeh


  • Quick cleaning
  • Segmentation
  • Normalization
  • Data splitting


Data Cleaning

import tnkeeh as tn
tn.clean_data(file_path = 'data.txt', save_path = 'cleaned_data.txt',)


  • segment uses farasa for segmentation.
  • remove_diacritics removes all diacritics.
  • remove_special_chars removes all sepcial chars.
  • remove_english removes english alphabets and digits.
  • normalize match digits that have the same writing but different encodings.
  • remove_tatweel tatweel character ـ is used a lot in arabic writing.
  • remove_repeated_chars remove characters that appear three times in sequence.
  • remove_html_elements remove html elements in the form with their attirbutes.
  • remove_links remove links.
  • remove_twitter_meta remove twitter mentions, links and hashtags.
  • remove_long_words remove words longer than 15 chars.
  • by_chunk read files by chunks with size chunk_size.

HuggingFace datasets

import tnkeeh as tn 
from datasets import load_dataset

dataset = load_dataset('metrec')

cleander = tn.Tnkeeh(remove_diacritics = True)
cleaned_dataset = cleander.clean_hf_dataset(dataset, 'text')

Data Splitting

Splits raw data into training and testing using the split_ratio

import tnkeeh as tn
tn.split_raw_data(data_path, split_ratio = 0.8)

Splits data and labels into training and testing using the split_ratio

import tnkeeh as tn
tn.split_classification_data(data_path, lbls_path, split_ratio = 0.8)

Splits input and target data with ration split_ratio. Commonly used for translation


Data Reading

Read split data, depending if it was raw or classification

import tnkeeh as tn
train_data, test_data = tn.read_data(mode = 0)


  • mode = 0 read raw data.
  • mode = 1 read labeled data.
  • mode = 2 read parallel data.


This is an open source project where we encourage contributions from the community.


MIT license.


  author = {Zaid Alyafeai and Maged Saeed},
  title = {tkseem: A Preprocessing Library for Arabic.},
  year = {2020},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{}}

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

tnkeeh-0.0.9.tar.gz (7.8 kB view hashes)

Uploaded source

Built Distribution

tnkeeh-0.0.9-py3-none-any.whl (8.6 kB view hashes)

Uploaded py3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page