cadar 0.1.13.dev0

Canonicalization and Darija Representation - Bidirectional transliteration for Darija

CaDaR: Canonicalization and Darija Representation

High-performance bidirectional transliteration for Darija (Moroccan Arabic)

Overview

CaDaR is a robust, FST-style (Finite State Transducer) transliteration library designed specifically for Darija (Moroccan Arabic). It provides bidirectional conversion between Arabic script and Latin (Romanized/Bizi) script, along with standardization capabilities for both scripts.

Key Features

• Bidirectional Transliteration: Convert seamlessly between Arabic and Latin scripts
• Intelligent Normalization: Handles noise, diacritics, and common variations
• Darija-Aware Processing: Respects Darija-specific linguistic patterns
• Intermediate Canonical Representation (ICR): Unified internal representation for accurate conversion
• High Performance: Written in Rust with Python bindings for optimal speed
• Extensible: Designed to support multiple Darija dialects (currently Moroccan)

Architecture

CaDaR uses a 6-stage pipeline with an Intermediate Canonical Representation (ICR):

Raw Input
   ↓
Stage 1: Script Detection
   ↓
Stage 2: Noise Cleaning & Normalization
   ↓
Stage 3: Tokenization (Darija-aware)
   ↓
Stage 4: Canonical Darija Representation (ICR)
   ↓
Stage 5: Target Script Generation
   ↓
Stage 6: Post-validation & Fixes
   ↓
Clean Standard Output

What is ICR?

The Intermediate Canonical Representation (ICR) is the core innovation of CaDaR. It's a script-independent, phonologically-grounded representation that:

• Abstracts away script-specific quirks
• Preserves Darija phonological distinctions
• Enables lossless round-trip conversions
• Allows for consistent normalization across scripts

Installation

From PyPI

pip install cadar

From Source

Prerequisites

• Python 3.8 or higher
• Rust toolchain (for building from source)
• Maturin (for Python packaging)

Build and Install

# Clone repository
git clone https://github.com/Oit-Technologies/CaDaR.git
cd CaDaR

# Install Maturin
pip install maturin

# Build and install in development mode
maturin develop

# Or build a wheel for distribution
maturin build --release

Usage

Python API

CaDaR provides four main functions that match the requested API:

1. ara2bizi() - Arabic to Latin/Bizi

import cadar

result = cadar.ara2bizi("سلام", darija="Ma")
print(result)
# Output: "slam"

2. bizi2ara() - Latin/Bizi to Arabic

import cadar

result = cadar.bizi2ara("salam", darija="Ma")
print(result)
# Output: "سلام"

3. ara2ara() - Arabic Standardization

import cadar

result = cadar.ara2ara("أنَا", darija="Ma")
print(result)
# Output: "انا"

4. bizi2bizi() - Latin/Bizi Standardization

import cadar

result = cadar.bizi2bizi("salaaaam", darija="Ma")
print(result)
# Output: "salam"

Convenience Functions

CaDaR also provides convenience functions with auto-detection:

import cadar

# Auto-detect and transliterate
text = "سلام"  # Arabic
result = cadar.transliterate(text, target="latin", darija="Ma")
print(result)  # "slam"

# Auto-detect and standardize
text = "salaaaam"
result = cadar.standardize(text, script="auto", darija="Ma")
print(result)  # "salam"

CaDaR Class

For multiple operations, use the CaDaR class:

import cadar

# Create processor
processor = cadar.CaDaR(darija="Ma")

# Use for multiple operations
texts = ["سلام", "كيفاش", "بزاف"]
for text in texts:
    result = processor.ara2bizi(text)
    print(f"{text} → {result}")

Version History

See CHANGELOG.md for details.

License

CaDaR is licensed under the Apache License 2.0.

Contributing

Contributions are welcome! Please see CONTRIBUTING.md for guidelines.

Links

• Documentation
• GitHub Repository
• PyPI Package
• Issue Tracker