candle-deduplicator 1.0.0

ONNX-based Arabic Consecutive-Character Deduplicator (Candle model by Abjad AI)

candle-deduplicator

ONNX-based Arabic consecutive-character deduplication powered by the CANDLE model from Abjad AI.

The model removes unintentional character elongation that appears in user-generated Arabic text — for example:

Input	Output
الممملكككة الأررررردنييية الللهههااااششميية	المملكة الأردنية الهاشمية
الممملكككة الأررررردنييية ررراااننيييااا	الملكة الأردنية رانيا
مررررحبببببا	مرحبا

---

Installation

pip install candle-deduplicator

For GPU inference, uninstall onnxruntime and install the GPU variant instead:

pip uninstall onnxruntime
pip install candle-deduplicator onnxruntime-gpu

ONNX model weights are downloaded automatically on first use and cached inside the package directory.

---

Quick start

Single string

from candle_deduplicator import CandleDeduplicator

model = CandleDeduplicator()
print(model.deduplicate('الممملكككة الأررررردنييية الللهههااااششميية'))
# المملكة الأردنية الهاشمية

Batch inference

texts = [
    'الممملكككة الأررررردنييية',
    'مررررحبببببا بببالعررربييية',
]
results = model.deduplicate_batch(texts, batch_size=32)
print(results)

Distilled (faster) variant

from candle_deduplicator import CandleDeduplicatorDistilled

model = CandleDeduplicatorDistilled()
print(model.deduplicate('مررررحبببببا'))
# مرحبا

Load a local ONNX file

Pass model_path to skip the auto-download and load directly from disk:

model = CandleDeduplicator(model_path="full_model.onnx")
model = CandleDeduplicatorDistilled(model_path="distilled_model.onnx")

GPU inference

model = CandleDeduplicator(model_path="full_model.onnx", device="cuda")
# ✅ Loaded model successfully! (provider: CUDAExecutionProvider)

If onnxruntime-gpu is not installed or CUDA is unavailable, ONNX Runtime falls back to CPU automatically and the printed provider will reflect that.

---

API reference

CandleDeduplicator(model_path=None, auto_preprocess=True, device="cpu")

Full 6-layer encoder model. Higher accuracy, recommended for offline batch processing.

CandleDeduplicatorDistilled(model_path=None, auto_preprocess=True, device="cpu")

Distilled 2-layer encoder model. Faster inference, recommended for latency-sensitive applications.

Both classes share the same interface:

Method	Description
deduplicate(text)	Deduplicate a single string.
deduplicate_batch(texts, batch_size=16, verbose=True)	Deduplicate a list of strings.

Constructor parameters

Parameter	Type	Default	Description
model_path	str or None	None	Path to a local .onnx file. When None the model is downloaded automatically from GitHub Releases.
auto_preprocess	bool	True	Strip non-Arabic characters before running the model.
device	str	"cpu"	Execution device: "cpu" or "cuda". GPU requires onnxruntime-gpu.

---

How it works

The CANDLE model uses an encoder-only Transformer trained with CTC loss. During inference:

1. Each input string is checked for consecutive duplicate characters. Strings that are already clean skip the model entirely.
2. Strings with duplicates are normalised to at most two consecutive occurrences of each character and tokenised at the character level.
3. The encoder produces per-token logits; greedy argmax selects the most likely token at each position.
4. CTC greedy decoding collapses repeated tokens and removes blank symbols to recover the deduplicated text.
5. A word-level guard ensures that words without duplicates in the original text are never altered by the model.

---

License

Apache 2.0 — see LICENSE.