polyglean 0.1.0

A universal text cleaning and dataset preparation library for any language or script.

polyglean

A universal text cleaning and dataset preparation library for any language or script.

Built for TTS (text-to-speech) voice data pipelines but useful for any NLP corpus work.
Handles Latin, Arabic, Devanagari, CJK, and any Unicode-based script correctly.

---

Installation

pip install polyglean

---

Quick start

import polyglean

config = {
    "base_dir":       "/path/to/your/data/",
    "output_dir":     "/path/to/output/",
    "source_config":  [
        ("bbc_news.xlsx", "sentence", "bbc"),
        ("bible.xlsx",    "text",     "bible"),
    ],
    "separate_files": [],
    "extra_cleaners": [],
    "word_count_min": 5,
    "word_count_max": 20,
    "output_tts":   "my_language_tts.xlsx",
    "output_other": "my_language_other.xlsx",
}

polyglean.run_language("my_language", config)

Two output files are created:

• my_language_tts.xlsx — sentences within the word count window (TTS-ready)
• my_language_other.xlsx — sentences outside the window (for further splitting)

---

CLI usage

Register your language config, then use the command-line tools:

# Run the full cleaning pipeline
polyglean-clean --lang yoruba

# Process multiple languages
polyglean-clean --lang yoruba hausa igbo

# Split long sentences into TTS-ready chunks
polyglean-split --lang yoruba

---

Supported file formats

Type key	Format	Extra options
excel (default)	.xlsx / .xls	col (column name)
csv	.csv	col, sep, encoding
parquet	.parquet	col
json_list	JSON array of objects	col (field to extract)
json_articles_body	JSON with articles[].body	—
txt	Plain text, one sentence/line	—

source_config = [
    # Excel (shorthand tuple)
    ("corpus.xlsx", "sentence", "corpus"),

    # CSV with tab separator
    {"path": "data.tsv", "col": "text", "label": "web", "type": "csv", "sep": "\t"},

    # Parquet
    {"path": "data.parquet", "col": "sentence", "label": "hf", "type": "parquet"},

    # JSON array
    {"path": "stories.json", "col": "story_text", "label": "stories", "type": "json_list"},

    # Plain text
    {"path": "sentences.txt", "label": "misc", "type": "txt"},
]

---

What the cleaning pipeline does

Applied to every sentence, in order:

1. Unicode NFC normalization — prevents false duplicates from diacritic encoding differences
2. Control character removal — strips non-printable characters
3. Newline flattening — collapses multi-line text to a single line
4. Bracket content removal — removes [footnote], [210], etc.
5. Digit removal — removes numerals (TTS-specific; see below to opt out)
6. Whitespace collapsing — normalizes spacing

Then:

• All-caps rows are dropped (headers, acronym-only rows)
• Empty sentences are dropped
• Duplicates are removed (first occurrence kept)

Adding language-specific cleaners

import re

config["extra_cleaners"] = [
    # Strip a prefix specific to your dataset
    lambda text: re.sub(r"^Chapter\s+\d+:\s*", "", text),

    # Remove URLs from web-scraped sources
    from polyglean.cleaners import remove_urls
    remove_urls,
]

Keeping digits (non-TTS use)

from polyglean.cleaners import BASE_CLEANERS, remove_digits

config["extra_cleaners"] = []
# Pass a custom cleaner list that excludes remove_digits:
CLEANERS_NO_DIGITS = [fn for fn in BASE_CLEANERS if fn is not remove_digits]

---

Using the step-by-step API

from polyglean.loaders import load_sources
from polyglean.pipeline import clean_data, split_and_save, split_other_sentences

df = load_sources(config["base_dir"], config["source_config"])
df = clean_data(df, config.get("extra_cleaners", []))
split_and_save(df, config)

# Later — chunk the long sentences
split_other_sentences("my_language", config)

---

License

MIT © Rasheedat Sikiru