candle-deduplicator

ONNX-based Arabic consecutive-character deduplication powered by the CANDLE model from Abjad AI.

The model removes unintentional character elongation that appears in user-generated Arabic text — for example:

Installation

pip install candle-deduplicator

For GPU inference, uninstall onnxruntime and install the GPU variant instead:

pip uninstall onnxruntime
pip install candle-deduplicator onnxruntime-gpu

ONNX model weights are downloaded automatically on first use and cached inside the package directory.

Quick start

Single string

from candle_deduplicator import CandleDeduplicator

model = CandleDeduplicator()
print(model.deduplicate('الممملكككة الأررررردنييية الللهههااااششميية'))
# المملكة الأردنية الهاشمية

Batch inference

texts = [
    'الممملكككة الأررررردنييية',
    'مررررحبببببا بببالعررربييية',
]
results = model.deduplicate_batch(texts, batch_size=32)
print(results)

Distilled (faster) variant

from candle_deduplicator import CandleDeduplicatorDistilled

model = CandleDeduplicatorDistilled()
print(model.deduplicate('مررررحبببببا'))
# مرحبا

Load a local ONNX file

Pass model_path to skip the auto-download and load directly from disk:

model = CandleDeduplicator(model_path="full_model.onnx")
model = CandleDeduplicatorDistilled(model_path="distilled_model.onnx")

GPU inference

model = CandleDeduplicator(model_path="full_model.onnx", device="cuda")
# ✅ Loaded model successfully! (provider: CUDAExecutionProvider)

If onnxruntime-gpu is not installed or CUDA is unavailable, ONNX Runtime falls back to CPU automatically and the printed provider will reflect that.

API reference

CandleDeduplicator(model_path=None, auto_preprocess=True, device="cpu")

Full 6-layer encoder model. Higher accuracy, recommended for offline batch processing.

CandleDeduplicatorDistilled(model_path=None, auto_preprocess=True, device="cpu")

Distilled 2-layer encoder model. Faster inference, recommended for latency-sensitive applications.

Both classes share the same interface:

Constructor parameters

How it works

The CANDLE model uses an encoder-only Transformer trained with CTC loss. During inference:

1. Each input string is checked for consecutive duplicate characters. Strings that are already clean skip the model entirely.
2. Strings with duplicates are normalised to at most two consecutive occurrences of each character and tokenised at the character level.
3. The encoder produces per-token logits; greedy argmax selects the most likely token at each position.
4. CTC greedy decoding collapses repeated tokens and removes blank symbols to recover the deduplicated text.
5. A word-level guard ensures that words without duplicates in the original text are never altered by the model.

License

Apache 2.0 — see LICENSE.